bugGNU Octave - Bugs: bug #57596, Should the "len"...

 
 

bug #57596: Should the "len" argument of "fgetl" and "fgets" mean bytes or characters?

Submitted by:  Markus Mützel <mmuetzel>
Submitted on:  Tue 14 Jan 2020 08:32:26 AM UTC  
 
Category:  Interpreter Severity:  2 - Minor
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Need Info Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Any

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 18 Feb 2020 10:18:43 PM UTC, comment #3: 

TL;DR - the LEN argument in matlab specifies characters, even for multibyte characters. octave should probably try to emulate that for compatibility reasons.

from a compatibility standpoint - Matlab file says fgets(FID, NCHAR).  it does specifically use the word character to describe behavior of that input parameter.  The help says it will read characters using the encoding scheme associated with the file as per fopen.

Using a UTF-8 test file [1], the first multibyte line is:

You should see the Greek word 'kosme':       "κόσμε"

checking in Matlab 2019a:

>> abc=fopen("UTF-8 test file.html",'r','n',"UTF-8");
>> for idx=1:45,disp(fgets(abc)),end

<trimming output to reach multibyte test chars>

>> disp(fgets(abc,47));
You should see the Greek word 'kosme':       "κ
>> disp(fgets(abc,3));
όσμ

without reading file in as UTF-8, reading in that whole line looks like:

You should see the Greek word 'kosme':       "κόσμε"

[1] https://www.w3.org/2001/06/utf-8-wrong/UTF-8-test.html

Nicholas Jankowski <nrjank>
Tue 14 Jan 2020 09:38:51 AM UTC, comment #2: 

At least for "fgets", we shouldn't remove the "len" argument or we would break Matlab compatibility. [1]

The word "character" isn't used consistently throughout the manual. Sometimes it might mean "character entity" (does that include character modifiers?), sometimes "byte" and sometimes "one element of a vector or matrix of type 'char'".

If we changed how "len" is working right now, the following code snippet might stop working:

str(1:len) = fgets(fid, len);

But that already might fail if the source is non-UTF-8 and a non-ASCII character is read.

[1]: https://de.mathworks.com/help/matlab/ref/fgets.html

Markus Mützel <mmuetzel>
Project Member
Tue 14 Jan 2020 09:06:26 AM UTC, comment #1: 

In the manual https://octave.org/doc/v5.1.0/XREFfgetl.html there is

> len characters


but you are right.  I think this was written in the common believe 1 char = 1 byte.

As user I would expect "len" characters to be read.  What is the point of reading a damaged UTF-8 encoded character at the limit of "len" bytes?  But as it is a pure Octave extension, we are free to change/specify that interface, or to deprecate this parameter.  grep-ing the Octave sources shows, that it is unused.

Kai Torben Ohlhus <siko1056>
Project Member
Tue 14 Jan 2020 08:32:26 AM UTC, original submission:  

While implementing the encoding argument for fopen, the question arose of whether the "len" argument of "fgetl" and "fgets" should correspond to the number of bytes or the number of character entities that are read from an input stream. This is different for encodings that use multiple bytes to encode on single character (e.g. non-ASCII characters in UTF-8).

Markus Mützel <mmuetzel>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-01-14 rik5 Carbon-CopyRemoved 72865 => -
    2020-01-14 rik5 SummaryShould the "len" argument of "fgetl" and "fgets" mean bytes or characters => Should the "len" argument of "fgetl" and "fgets" mean bytes or characters?
    2020-01-14 siko1056 StatusNone => Need Info

    Back to the top


    Powered by Savane 3.5