bugGNU Octave - Bugs: bug #57596, Should the "len"...

 
 

bug #57596: Should the "len" argument of "fgetl" and "fgets" mean bytes or characters?

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Tue 14 Jan 2020 08:32:26 AM UTC
   
 
Category:  Documentation Severity:  2 - Minor
Priority:  5 - Normal Item Group:  Documentation
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 10 Jun 2020 05:42:45 PM UTC, comment #10: 


> If you are arguing that for UTF-16, "one character" should be one two-byte "UTF-16 code unit" instead of one Unicode code point, why don't you also argue that, for UTF-8, "one character" should be one one-byte "UTF-8 code unit"?


I am arguing that. (We're talking about the question of how to handle UTF-16 surrogate pairs here, right?) Largely because this is how I've seen it done in other languages, it tends to work all right, it removes the need to strongly distinguish between UTF-16 and UCS-2 input modes (which is an advanced topic even for programmers with a decent grasp of Unicode), and it would provide for Matlab compatibility (because Matlab does UCS-2, but most Matlab programmers I've talked to pretend it's UTF-16, and in that sense it passes through surrogate pair code units unmolested, but treats them as one character each). But this point is the part I'm least confident of among the points I made, and I could probably be convinced otherwise.

> If you are hinting on whether the "char" type in Octave should have a size of one or two bytes: That is another issue and should be kept separate from this bug.


I am not hinting. I recognize that's a whole nother discussion, and not trying to drag it in here.

Andrew Janke <apjanke>
Wed 10 Jun 2020 05:01:39 PM UTC, comment #9: 

@Andrew: Yeah. If we were to make some changes here, I think we should start with "traditional single byte encodings" (quite easy) and UTF-8 (not quite as easy but manageable) first.
If I recall correctly, the first should be working already.

If I remember correctly, it's not as easy as it may sound to read from files that are not byte-based with the current implementation of reading files in Octave. (But I haven't looked at it in a while and might be mis-remembering.)

What I don't understand: If you are arguing that for UTF-16, "one character" should be one two-byte "UTF-16 code unit" instead of one Unicode code point, why don't you also argue that, for UTF-8, "one character" should be one one-byte "UTF-8 code unit"?

Or did I misunderstand?

If you are hinting on whether the "char" type in Octave should have a size of one or two bytes: That is another issue and should be kept separate from this bug.
Please, open a new report or write to the maintainers list if you think this should change.

Markus Mützel <mmuetzel>
Group administrator
Wed 10 Jun 2020 01:58:44 PM UTC, comment #8: 

Pretty sure we need to sort out what's going to happen with UTF-8 and char semantics before doing this one. But otherwise, it's not that bad:

The character-wise behavior for composite characters and the like is pretty well established by other languages and the Unicode standard, though:
- If you're doing characterwise UTF-8, then "one character" is one Unicode code point, however many bytes that's encoded as.
- If you want to be Matlab-compatible and are doing UCS-2, then "one character" is always one two-byte UCS-2 code unit/code point
- If you're doing UTF-16, "one character" should probably be one two-byte UTF-16 code unit, not one Unicode code point.
- A Unicode combining character is still technically just one character and one Unicode code point; you don't have to treat them specially at the I/O level. It's up to the application code to determine the semantics of sequences of characters that involve combining characters.

And if you encounter an invalid byte sequence, then I think you should, and pretty much have to, either throw an error, or convert to the Unicode "replacement character", and this behavior should probably be caller-configurable on a per-filehandle basis, and throwing an error should probably be the default.

UCS-2 has no invalid byte sequences.

Andrew Janke <apjanke>
Wed 10 Jun 2020 01:23:51 PM UTC, comment #7: 

Was the change of the category to "documentation" intentional?

This would require some non-trivial changes to the functions that read text from files (keep counting the number of characters that have been read so far, read ahead until a byte sequence is complete).
One of the complications is that there are "invalid byte sequences" in some multibyte encodings (e.g. UTF-8 or UTF-16). There are several possible ways how those could be treated. All with their own implications.

Next question is: What about composite characters? Those consist of one regular character followed by one or several combining characters. Should those be treated as one character as well? After all, the combining characters cannot be used on their own...

This gets quite complicated very quickly...

Markus Mützel <mmuetzel>
Group administrator
Tue 09 Jun 2020 09:37:25 PM UTC, comment #6: 

I agree that the interpretation should mean "characters", which may mean multiple bytes.

Rik <rik5>
Group administrator
Wed 06 May 2020 09:05:22 AM UTC, comment #5: 

Also, maybe a minor revision to the documentation's wording is in order: I think maybe "character entity" is a misnomer here? When dealing with character encoding, usually people talk about "code points", "code units", "characters", and character values. Never "entities". The only place I've heard the specific phrase "character entity" is in the context of HTML and XML escaping (e.g. https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references), which has nothing to do with what's going on here.

Andrew Janke <apjanke>
Wed 06 May 2020 09:02:19 AM UTC, comment #4: 

I find Nicholas' Matlab-compatibility argument pretty compelling.

Also, I can't think of an actual use case where one might be doing character/text-oriented reading from a stream on a file handle, but would want to specify a number of bytes read instead of number of characters returned. Maybe if you're trying to optimize the buffering and paging behavior of reading from a file? But that's crazy low-level C stuff; surely not something that Octave should be concerned about, right? If that was the case, you'd provided a lower-level non-buffered I/O API, right?

Andrew Janke <apjanke>
Tue 18 Feb 2020 10:18:43 PM UTC, comment #3: 

TL;DR - the LEN argument in matlab specifies characters, even for multibyte characters. octave should probably try to emulate that for compatibility reasons.


from a compatibility standpoint - Matlab file says fgets(FID, NCHAR).  it does specifically use the word character to describe behavior of that input parameter.  The help says it will read characters using the encoding scheme associated with the file as per fopen.

Using a UTF-8 test file [1], the first multibyte line is:


You should see the Greek word 'kosme':       "κόσμε"


checking in Matlab 2019a:

>> abc=fopen("UTF-8 test file.html",'r','n',"UTF-8");
>> for idx=1:45,disp(fgets(abc)),end

<trimming output to reach multibyte test chars>

>> disp(fgets(abc,47));
You should see the Greek word 'kosme':       "κ
>> disp(fgets(abc,3));
όσμ


without reading file in as UTF-8, reading in that whole line looks like:


You should see the Greek word 'kosme':       "κόσμε"


[1] https://www.w3.org/2001/06/utf-8-wrong/UTF-8-test.html


Nicholas Jankowski <nrjank>
Group Member
Tue 14 Jan 2020 09:38:51 AM UTC, comment #2: 

At least for "fgets", we shouldn't remove the "len" argument or we would break Matlab compatibility. [1]

The word "character" isn't used consistently throughout the manual. Sometimes it might mean "character entity" (does that include character modifiers?), sometimes "byte" and sometimes "one element of a vector or matrix of type 'char'".

If we changed how "len" is working right now, the following code snippet might stop working:

str(1:len) = fgets(fid, len);


But that already might fail if the source is non-UTF-8 and a non-ASCII character is read.

[1]: https://de.mathworks.com/help/matlab/ref/fgets.html

Markus Mützel <mmuetzel>
Group administrator
Tue 14 Jan 2020 09:06:26 AM UTC, comment #1: 

In the manual https://octave.org/doc/v5.1.0/XREFfgetl.html there is

> len characters


but you are right.  I think this was written in the common believe 1 char = 1 byte.

As user I would expect "len" characters to be read.  What is the point of reading a damaged UTF-8 encoded character at the limit of "len" bytes?  But as it is a pure Octave extension, we are free to change/specify that interface, or to deprecate this parameter.  grep-ing the Octave sources shows, that it is unused.

Kai Torben Ohlhus <siko1056>
Group Member
Tue 14 Jan 2020 08:32:26 AM UTC, original submission:  

While implementing the encoding argument for fopen, the question arose of whether the "len" argument of "fgetl" and "fgets" should correspond to the number of bytes or the number of character entities that are read from an input stream. This is different for encodings that use multiple bytes to encode on single character (e.g. non-ASCII characters in UTF-8).

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-06-09 rik5 CategoryInterpreter Documentation
        Item GroupIncorrect Result Documentation
        StatusNeed Info Confirmed
    2020-01-14 rik5 Carbon-CopyRemoved 72865 -
    2020-01-14 rik5 SummaryShould the &quot;len&quot; argument of &quot;fgetl&quot; and &quot;fgets&quot; mean bytes or characters Should the "len" argument of "fgetl" and "fgets" mean bytes or characters?
    2020-01-14 siko1056 StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code