bugGNU Octave - Bugs: bug #58368, UTF16 and UTF32 characters in MAT...

 
 

bug #58368: UTF16 and UTF32 characters in MAT files

Submitter:  Guillaume <gyom>
Submitted:  Thu 14 May 2020 10:36:09 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Fixed Assigned to:  None
Originator Name:  Guillaume Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 24 May 2020 07:58:36 AM UTC, comment #10: 

I don't have a test case for UTF-32 encoded strings in .mat files. But since the code blocks for reading UTF-32 and UTF-16 are very similar, I suppose they'd work, too.

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Sat 23 May 2020 04:12:57 PM UTC, comment #9: 

Ready to close?

Rik <rik5>
Group administrator
Mon 18 May 2020 02:34:50 PM UTC, comment #8: 

Guillaume, thanks for testing the latest patch.

I pushed the change here:
https://hg.savannah.gnu.org/hgweb/octave/rev/d13ad9dc9348

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Sun 17 May 2020 10:28:19 PM UTC, comment #7: 

Thanks for the update Markus. I could also load and save the MAT-file in Octave and open it in Matlab without trailing characters.

Guillaume <gyom>
Sun 17 May 2020 01:42:23 PM UTC, comment #6: 

The attached patch is just a minor clean-up over v3 and makes the BIST with "-v7" conditional on HAVE_ZLIB.

(file #49103)

Markus Mützel <mmuetzel>
Group administrator
Sun 17 May 2020 11:17:07 AM UTC, comment #5: 

Even worse that the behavior of Matlab seems to be undetermined when reading UTF-8 strings with non-ASCII characters.

The attached patch converts character strings to UTF-16 (the encoding Matlab uses for strings) before saving them to .mat files (-v6 or -v7).
It also adds some tests.

Again, the conversion cannot be safely performed in general for character matrices and ND-arrays. So they are stored as UTF-8 (the encoding Octave uses for strings).
But character matrices and ND-arrays should be avoided anyway imho. At least as an exchange format.

With that change, I can successfully load the .mat file saved by Octave in Matlab R2020a. No random trailing characters.

(file #49101)

Markus Mützel <mmuetzel>
Group administrator
Sat 16 May 2020 08:02:42 PM UTC, comment #4: 

Thanks, no more warnings in Octave with your latest patch. When loading in Matlab R2020a the file saved by Octave, I get a trailing space:


>> double(A{14})

ans =

    82   233   117   110   105   111   110    32


Guillaume <gyom>
Sat 16 May 2020 04:40:21 PM UTC, comment #3: 

Thanks Guillaume for testing.
The attached patch saves character arrays as UTF-8 to .mat files (-v6 and -v7). With it, a save-load-cycle seems to work fine in Octave.
Also Matlab seems to accept loading .mat files with UTF-8 encoded strings. (Tested with Matlab R2020a). However, they seem to get the string length wrong for strings with multi-byte characters. See the trailing 0 for "Réunion" in Matlab when loading the file saved by Octave:

>> double(A{14})

ans =

    82   233   117   110   105   111   110     0


I'd say this is a Matlab bug. But we could work around it by converting our strings to UTF-16 before writing to the .mat file.

(file #49093)

Markus Mützel <mmuetzel>
Group administrator
Sat 16 May 2020 03:33:55 PM UTC, comment #2: 

Thanks Markus, with your patch I could load the file without problem. I then tried to save and load it again and got another warning:


octave:7> save A.mat A -v6
octave:8> load A.mat
warning: load: failed to convert from input to UTF-8; replacing non-ASCII characters with '?'


Guillaume <gyom>
Fri 15 May 2020 09:39:37 PM UTC, comment #1: 

Thanks for the test case.

The attached patch adds the conversion from UTF-16 or UTF-32 to Octave's internal UTF-8 encoding when reading from v5 .mat files. It also no longer strips non-ASCII characters from UTF-8 strings.

This is only save to do for character vectors. For character matrices or ND arrays (in UTF-16 or UTF-32), it falls back to the previous behavior of replacing non-ASCII characters with "?".
Maybe we should fall back to this behavior for column vectors, too?

Is there a good way of constructing a charMatrix from a non-zero terminated C string buffer? In the patch, I constructed a C++ string with the known length constructor and passed that to the charMatrix constructor. Can the construction of the intermediate object be avoided?

I only tested with the file from comment #0 which contains UTF-16 encoded strings. But I hope that UTF-32 and UTF-8 should be working, too.

(file #49082)

Markus Mützel <mmuetzel>
Group administrator
Thu 14 May 2020 10:36:09 AM UTC, original submission:  

Following this:
https://lists.gnu.org/archive/html/octave-maintainers/2020-05/msg00068.html
I attach a MAT-file created by Matlab where the following warning is issued when loading in Octave:

warning: load: can not read non-ASCII portions of UTF characters; replacing unreadable characters with '?'


Guillaume <gyom>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49103:  bug58368_utf_mat_v4.patch added by mmuetzel (14KiB - application/octet-stream)
file #49101:  bug58368_utf_mat_v3.patch added by mmuetzel (14KiB - application/octet-stream)
file #49093:  bug58368_utf_mat_v2.patch added by mmuetzel (8KiB - application/octet-stream)
file #49082:  bug58368_utf_mat.patch added by mmuetzel (8KiB - application/octet-stream)
file #49064:  A.mat added by gyom (3KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by gyom (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-05-24 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2020-05-18 mmuetzel StatusPatch Submitted Ready For Test
    2020-05-17 mmuetzel Attached File- Added bug58368_utf_mat_v4.patch, #49103
    2020-05-17 mmuetzel Attached File- Added bug58368_utf_mat_v3.patch, #49101
    2020-05-16 mmuetzel Attached File- Added bug58368_utf_mat_v2.patch, #49093
        StatusNone Patch Submitted
    2020-05-15 mmuetzel Attached File- Added bug58368_utf_mat.patch, #49082
    2020-05-14 gyom Attached File- Added A.mat, #49064

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code