bugGNU Octave - Bugs: bug #57304, Error message for non-UTF-8...

 
 

bug #57304: Error message for non-UTF-8 encoded *.m files

Submitter:  Guillaume <gyom>
Submitted:  Tue 26 Nov 2019 11:10:33 AM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Missed Error or Warning
Status:  Fixed Assigned to:  None
Originator Name:  Guillaume Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 29 Nov 2019 04:14:55 PM UTC, comment #8: 

Thanks again Markus, both approaches work and make sense.

Guillaume <gyom>
Fri 29 Nov 2019 01:10:08 PM UTC, comment #7: 

Sorry, my bad. "fread" doesn't convert the encoding. It just reads the bytes as they are stored in the file. (At least this is what Matlab does.)

You could probably use fscanf or fgets/fgetl (still, I couldn't try):

fid = fopen ("angstrom.m", "r", "n", "iso-8859-1");
regexp (fscanf (fid, "%s")), "Hello")
fclose (fid);


Or combine "fread" with "native2unicode" (also works in Octave 5.1):

fid = fopen ("angstrom.m");
regexp (native2unicode (fread (fid).', "iso-8859-1"), "Hello")
fclose (fid);


Markus Mützel <mmuetzel>
Group administrator
Fri 29 Nov 2019 12:52:29 PM UTC, comment #6: 

Ah, thanks! I was missing the uint8() call and your first option now works in Octave 6.

Using fopen(...,"iso-8859-1") in Octave 6 still results in an error with regexp().

Guillaume <gyom>
Fri 29 Nov 2019 12:39:01 PM UTC, comment #5: 

The following works for me with Octave 5.1:

regexp (native2unicode (uint8 (fileread ("angstrom.m")), "iso-8859-1"), "Hello")


I hope this still works in Octave 6.

The crucial part might be this sentence in the documentation of native2unicode:

> If native_bytes is a string vector, it is returned as is.



With Octave 6, it should also be possible to specify the encoding when opening the file. But I can't try at the moment:

fid = fopen ("angstrom.m", "r", "n", "iso-8859-1");
regexp (char (fread (fid).'), "Hello")
fclose (fid);



Markus Mützel <mmuetzel>
Group administrator
Fri 29 Nov 2019 12:22:40 PM UTC, comment #4: 

Thanks Markus. I still have one question: how would you go at parsing a non UTF-8 encoded file:


regexp (fileread ("angstrom.m"), "Hello")


fileread() does not allow you to specify an encoding but I didn't succeed when trying with fopen()/fread()/fclose() or calling native2unicode() before regexp().

Guillaume <gyom>
Wed 27 Nov 2019 01:03:42 PM UTC, comment #3: 

Thanks for testing.
Closing as fixed.

If you want to use files that are encoded in another encoding (not UTF-8) and expect non-ASCII characters in strings to work (e.g. as labels on axes), you would have to set the mfile encoding first - either via the drop down list in the editor preferences or with a command, e.g.:

__mfile_encoding__ ("iso-8859-1");
clear functions  # If the file was parsed before, this is necessary to parse it again with the new encoding.


That should already be working with Octave 5.1.

If you don't set the correct encoding, the results might be unexpected. That could be gibberish or errors when the strings are used.

The difference for Octave 6 will be that the default encoding will match the system locale independent of the platform (i.e. not only on Windows). This is "UTF-8" for most modern Linux and OS X anyway. Hence, there shouldn't be any noticeable changes for the huge majority of Linux or Mac users.

Files only containing ASCII characters aren't affected in any case.

Markus Mützel <mmuetzel>
Group administrator
Wed 27 Nov 2019 12:36:32 PM UTC, comment #2: 

Thanks Markus. With your change, it indeed now runs without errors on Linux. I'm still a bit confused about how things are meant to work as the file is not valid UTF-8 but if Octave is fine with it then I am too.


isutf8 angstrom.m
angstrom.m: line 1, char 3, byte 2: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.


Guillaume <gyom>
Tue 26 Nov 2019 07:30:16 PM UTC, comment #1: 

There shouldn't be any conversion happening if the input encoding is UTF-8.
I don't see any errors if I set the mfile_encoding to "UTF-8" on Windows.
But I do see the error on Linux. This is because the default mfile_encoding is "system" and the system happens to return upper case "UTF-8" for that. This is compared to lower case "utf-8".
Setting the mfile_encoding to "UTF-8" avoids the issue also on Linux.

I pushed the following change to convert the system encoding to lower case before comparing it:
http://hg.savannah.gnu.org/hgweb/octave/rev/5fc630696dc7

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Tue 26 Nov 2019 11:10:33 AM UTC, original submission:  

With Octave 6 being more specific about file encoding, I foresee that a non-negligible amount of existing *.m files will have issues. Currently the reported error is (see attached example):


error: file_reader::get_input: converting from codepage 'UTF-8' to UTF-8: Invalid or incomplete multibyte or wide character


This takes place even if the problematic character is in a comment and it's not necessarily obvious from the error message that the issue is with the parsing of the *.m file itself and not during its execution. Would it be possible to make the error message clearer to the user, pointing out the first line with an issue and perhaps provide a link or instruction on how to solve the issue?

Guillaume <gyom>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #47933:  angstrom.m added by gyom (34B - text/x-matlab)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by gyom (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-11-27 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2019-11-26 mmuetzel StatusNone Ready For Test
    2019-11-26 gyom Attached File- Added angstrom.m, #47933

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code