bugGNU Octave - Bugs: bug #58368, UTF16 and UTF32 characters in MAT...

 
 

bug #58368: UTF16 and UTF32 characters in MAT files

Submitted by:  Guillaume <gyom>
Submitted on:  Thu 14 May 2020 10:36:09 AM UTC  
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Fixed Assigned to:  None
Originator Name:  Guillaume Open/Closed:  Closed
Release:  dev Operating System:  Any

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 24 May 2020 07:58:36 AM UTC, comment #10: 

I don't have a test case for UTF-32 encoded strings in .mat files. But since the code blocks for reading UTF-32 and UTF-16 are very similar, I suppose they'd work, too.

Closing as fixed.

Markus Mützel <mmuetzel>
Project Member
Sat 23 May 2020 04:12:57 PM UTC, comment #9: 

Ready to close?

Rik <rik5>
Project Administrator
Mon 18 May 2020 02:34:50 PM UTC, comment #8: 

Guillaume, thanks for testing the latest patch.

I pushed the change here:
https://hg.savannah.gnu.org/hgweb/octave/rev/d13ad9dc9348

Marking as ready for test.

Markus Mützel <mmuetzel>
Project Member
Sun 17 May 2020 10:28:19 PM UTC, comment #7: 

Thanks for the update Markus. I could also load and save the MAT-file in Octave and open it in Matlab without trailing characters.

Guillaume <gyom>
Sun 17 May 2020 01:42:23 PM UTC, comment #6: 

The attached patch is just a minor clean-up over v3 and makes the BIST with "-v7" conditional on HAVE_ZLIB.

(file #49103)

Markus Mützel <mmuetzel>
Project Member
Sun 17 May 2020 11:17:07 AM UTC, comment #5: 

Even worse that the behavior of Matlab seems to be undetermined when reading UTF-8 strings with non-ASCII characters.

The attached patch converts character strings to UTF-16 (the encoding Matlab uses for strings) before saving them to .mat files (-v6 or -v7).
It also adds some tests.

Again, the conversion cannot be safely performed in general for character matrices and ND-arrays. So they are stored as UTF-8 (the encoding Octave uses for strings).
But character matrices and ND-arrays should be avoided anyway imho. At least as an exchange format.

With that change, I can successfully load the .mat file saved by Octave in Matlab R2020a. No random trailing characters.

(file #49101)

Markus Mützel <mmuetzel>
Project Member
Sat 16 May 2020 08:02:42 PM UTC, comment #4: 

Thanks, no more warnings in Octave with your latest patch. When loading in Matlab R2020a the file saved by Octave, I get a trailing space:

>> double(A{14})

ans =

    82   233   117   110   105   111   110    32

Guillaume <gyom>
Sat 16 May 2020 04:40:21 PM UTC, comment #3: 

Thanks Guillaume for testing.
The attached patch saves character arrays as UTF-8 to .mat files (-v6 and -v7). With it, a save-load-cycle seems to work fine in Octave.
Also Matlab seems to accept loading .mat files with UTF-8 encoded strings. (Tested with Matlab R2020a). However, they seem to get the string length wrong for strings with multi-byte characters. See the trailing 0 for "Réunion" in Matlab when loading the file saved by Octave:

>> double(A{14})

ans =

    82   233   117   110   105   111   110     0

I'd say this is a Matlab bug. But we could work around it by converting our strings to UTF-16 before writing to the .mat file.

(file #49093)

Markus Mützel <mmuetzel>
Project Member
Sat 16 May 2020 03:33:55 PM UTC, comment #2: 

Thanks Markus, with your patch I could load the file without problem. I then tried to save and load it again and got another warning:

octave:7> save A.mat A -v6
octave:8> load A.mat
warning: load: failed to convert from input to UTF-8; replacing non-ASCII characters with '?'

Guillaume <gyom>
Fri 15 May 2020 09:39:37 PM UTC, comment #1: 

Thanks for the test case.

The attached patch adds the conversion from UTF-16 or UTF-32 to Octave's internal UTF-8 encoding when reading from v5 .mat files. It also no longer strips non-ASCII characters from UTF-8 strings.

This is only save to do for character vectors. For character matrices or ND arrays (in UTF-16 or UTF-32), it falls back to the previous behavior of replacing non-ASCII characters with "?".
Maybe we should fall back to this behavior for column vectors, too?

Is there a good way of constructing a charMatrix from a non-zero terminated C string buffer? In the patch, I constructed a C++ string with the known length constructor and passed that to the charMatrix constructor. Can the construction of the intermediate object be avoided?

I only tested with the file from comment #0 which contains UTF-16 encoded strings. But I hope that UTF-32 and UTF-8 should be working, too.

(file #49082)

Markus Mützel <mmuetzel>
Project Member
Thu 14 May 2020 10:36:09 AM UTC, original submission:  

Following this:
https://lists.gnu.org/archive/html/octave-maintainers/2020-05/msg00068.html
I attach a MAT-file created by Matlab where the following warning is issued when loading in Octave:

warning: load: can not read non-ASCII portions of UTF characters; replacing unreadable characters with '?'

Guillaume <gyom>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49103:  bug58368_utf_mat_v4.patch added by mmuetzel (14KiB - application/octet-stream)
file #49101:  bug58368_utf_mat_v3.patch added by mmuetzel (14KiB - application/octet-stream)
file #49093:  bug58368_utf_mat_v2.patch added by mmuetzel (8KiB - application/octet-stream)
file #49082:  bug58368_utf_mat.patch added by mmuetzel (8KiB - application/octet-stream)
file #49064:  A.mat added by gyom (3KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by gyom (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-05-24 mmuetzel StatusReady For Test => Fixed
        Open/ClosedOpen => Closed
    2020-05-18 mmuetzel StatusPatch Submitted => Ready For Test
    2020-05-17 mmuetzel Attached File- => Added bug58368_utf_mat_v4.patch, #49103
    2020-05-17 mmuetzel Attached File- => Added bug58368_utf_mat_v3.patch, #49101
    2020-05-16 mmuetzel Attached File- => Added bug58368_utf_mat_v2.patch, #49093
        StatusNone => Patch Submitted
    2020-05-15 mmuetzel Attached File- => Added bug58368_utf_mat.patch, #49082
    2020-05-14 gyom Attached File- => Added A.mat, #49064

    Back to the top


    Powered by Savane 3.5