bugGNU Octave - Bugs: bug #64139, fileread doesn't support...

 
 

bug #64139: fileread doesn't support 'encoding' argument

Submitter:  Guillaume <gyom>
Submitted:  Wed 03 May 2023 02:11:21 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Guillaume Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  9.1.0 Planned Release:  9.1.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 09 May 2023 07:02:05 AM UTC, comment #13: 

The test suite passes in the CI on the affected platforms now.

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Sat 06 May 2023 04:47:25 PM UTC, comment #12: 

I went ahead and updated the condition for the test here:
https://hg.savannah.gnu.org/hgweb/octave/rev/687ea87ccf5d

Markus Mützel <mmuetzel>
Group administrator
Fri 05 May 2023 04:24:42 PM UTC, comment #11: 

Also add a note similar to this

> # FIXME: This test should only run if OCTAVE_HAVE_STRICT_ENCODING_FACET is defined.


Markus Mützel <mmuetzel>
Group administrator
Fri 05 May 2023 04:20:12 PM UTC, comment #10: 

The test suite seems to be failing with clang compilers:
E.g.: https://github.com/gnu-octave/octave/actions/runs/4895045937/jobs/8740091878#step:13:89949

>>>>> processing /home/runner/work/octave/octave/scripts/io/fileread.m
***** error <conversion from codepage 'unknownvalue' not supported>
 fileread ("filename", "Encoding", "UnknownValue")
!!!!! error failed.
Expected <conversion from codepage 'unknownvalue' not supported>, but got <fileread: cannot open file>


The actual issue might not be clang but libc++ with which conversion of streams is currently turned off. In that case, `fopen` falls back to "UTF-8" (with a warning) independent on which encoding was passed to `fopen` iirc.

We'd probably need to run that test only conditionally on `! _have_feature_ ("LLVM_LIBCXX")`.
Maybe something along the lines of:

%!testif ; ! __have_feature__ ("LLVM_LIBCXX")
%! fail ('fileread ("filename", "Encoding", "UnknownValue")', ...
%!       "conversion from codepage 'unknownvalue' not supported");


(Currently untested.)

Markus Mützel <mmuetzel>
Group administrator
Fri 05 May 2023 03:29:18 PM UTC, comment #9: 

Thank you for your contribution. I pushed it with very minor changes here:
https://hg.savannah.gnu.org/hgweb/octave/rev/7ce8302036c1

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Fri 05 May 2023 09:27:54 AM UTC, comment #8: 

Patch attached. The code allows for multiple param/value pairs even if only one is defined - this is for compatibility with Matlab where the following works:


fileread ("example.txt", "Encoding", "UTF-8", "Encoding", "UTF-8")


(file #54704)

Guillaume <gyom>
Thu 04 May 2023 03:24:29 PM UTC, comment #7: 

Entirely agreeing with comment #6. I don't know if the use of this syntax in fileread's documentation is a slip up but if MW updates its documentation to use this syntax wherever possible, it is quickly going to become pervasive in Matlab code and it will be important for Octave to support it for compatibility.

I will write a patch to add an encoding parameter to fileread.

Guillaume <gyom>
Thu 04 May 2023 12:58:03 PM UTC, comment #6: 

Although Octave does not yet support the


fcn (..., Keyword = value)


syntax in the way that Matlab does now, I believe it is equivalent to


fcn (..., "Keyword", value)


so either should work in Matlab and internally, the function sees "Keyword", value pairs so handling that in the fileread function is all you should need to do.

See also https://octave.discourse.group/t/arguments-block-implementation/4415 and https://octave.discourse.group/t/deprecate-assignment-as-an-expression/1148

John W. Eaton <jwe>
Group administrator
Thu 04 May 2023 11:09:03 AM UTC, comment #5: 


> What I'm looking for is ignoring any BOM, what would be the most robust way to do so?


You'd probably need to strip the BOM manually.
E.g., in Octave:

ex_str = fileread('example.txt');
if (strncmp(a, char([239, 187, 191]), 3))
  ex_str(1:3) = [];
end


Or you could use an external tool to strip the BOM beforehand.
The usage of a BOM in an UTF-8 encoded file is questionable anyway. There is no byte-order if the surrogates are bytes.
Alas, MS chose to add the BOM by default with many of their tools...

Markus Mützel <mmuetzel>
Group administrator
Thu 04 May 2023 10:30:06 AM UTC, comment #4: 


> As for adding the 'encoding' argument for fileread: is it a matter of passing the argument to the fopen call or am I still being confused?


That would probably be it.

I was looking at Matlab's online documentation for that function:
https://www.mathworks.com/help/matlab/ref/fileread.html

text = fileread(filename,Encoding=encoding)

I don't know if it is possible to implement that in a way that would be Matlab compatible. Afaict, Octave doesn't support that kind of syntax (yet).

Markus Mützel <mmuetzel>
Group administrator
Thu 04 May 2023 09:30:06 AM UTC, comment #3: 

Thanks Markus - I find encoding so confusing (perhaps a good reason why I should stay away from the string class). What I'm looking for is ignoring any BOM, what would be the most robust way to do so?

As for adding the 'encoding' argument for fileread: is it a matter of passing the argument to the fopen call or am I still being confused?

Guillaume <gyom>
Wed 03 May 2023 06:52:14 PM UTC, comment #2: 

I broke the conversion to UTF-16 with this changeset on the stable branch:
https://hg.savannah.gnu.org/hgweb/octave/rev/470134b3fc28

I pushed a fix for this regression also to the stable branch here:
https://hg.savannah.gnu.org/hgweb/octave/rev/f7206b6577c2

I'll leave this open as a feature request to implement the 'encoding' argument for `fileread` (even if I'm not sure if this was the intension of the OP).

Markus Mützel <mmuetzel>
Group administrator
Wed 03 May 2023 04:25:20 PM UTC, comment #1: 

The encoding for character arrays in Octave is UTF-8. For Matlab character arrays, it is UTF-16. It cannot be changed in either.

However, you can manually convert to UTF-16 if that is important. But be aware that the result won't work as a character vector in Octave, it is just a vector of integers:

>> u16char = unicode2native(fileread('example.txt'), ['utf-16', nthargout(3, 'computer'), 'e']);
>> double(typecast(u16char(1:floor(numel(u16char)/2)*2), 'uint16'))
ans =

   65279     111      99     116      97     118     101


There seems to be a bug in Octave that leads to an additional zero-byte being returned when converting to UTF-16. That's the reason for the indexing in the last line. That shouldn't be needed.

IIUC, the 'encoding' argument specifies the input encoding for the file to be read. That syntax is currently not supported in Octave.

Markus Mützel <mmuetzel>
Group administrator
Wed 03 May 2023 02:11:21 PM UTC, original submission:  

Using the attached file (UTF-8 Unicode (with BOM) text), I get a different output with fileread. In Matlab:


>> double(fileread('example.txt'))

ans =

   65279   111    99   116   97   118   101


and in Octave:


octave> double(fileread('example.txt'))

ans =

   239   187   191   111    99   116    97   118   101


If I specify the encoding in Matlab:


>> double(fileread('example.txt','encoding','utf-8'))

ans =

   65279   111    99   116   97   118   101

>> double(fileread('example.txt','encoding','ISO-8859-15'))

ans =

   239   187   191   111    99   116    97   118   101


The documentation says that without a specified encoding, a default encoding will be used and it might well be that it differs between Octave and Matlab but how can this be adjusted?

Guillaume <gyom>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #54704:  bug64139.patch added by gyom (3KiB - text/x-patch)
file #54697:  example.txt added by gyom (9B - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by gyom (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-05-09 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2023-05-05 mmuetzel StatusNone Ready For Test
        Fixed ReleaseNone 9.1.0
        Planned ReleaseNone 9.1.0
    2023-05-05 gyom Attached File- Added bug64139.patch, #54704
    2023-05-03 mmuetzel Item GroupMatlab Compatibility Feature Request
        Summarycharacter encoding scheme with fileread fileread doesn't support 'encoding' argument
    2023-05-03 mmuetzel Operating SystemGNU/Linux Any
    2023-05-03 gyom Attached File- Added example.txt, #54697

    Back to the top

    Powered by Savane 3.13-3cd0.
    Corresponding source code