> If you are arguing that for UTF-16, "one character" should be one two-byte "UTF-16 code unit" instead of one Unicode code point, why don't you also argue that, for UTF-8, "one character" should be one one-byte "UTF-8 code unit"?

I am arguing that. (We're talking about the question of how to handle UTF-16 surrogate pairs here, right?) Largely because this is how I've seen it done in other languages, it tends to work all right, it removes the need to strongly distinguish between UTF-16 and UCS-2 input modes (which is an advanced topic even for programmers with a decent grasp of Unicode), and it would provide for Matlab compatibility (because Matlab does UCS-2, but most Matlab programmers I've talked to pretend it's UTF-16, and in that sense it passes through surrogate pair code units unmolested, but treats them as one character each). But this point is the part I'm least confident of among the points I made, and I could probably be convinced otherwise.

> If you are hinting on whether the "char" type in Octave should have a size of one or two bytes: That is another issue and should be kept separate from this bug.

I am not hinting. I recognize that's a whole nother discussion, and not trying to drag it in here.

Pretty sure we need to sort out what's going to happen with UTF-8 and char semantics before doing this one. But otherwise, it's not that bad:

The character-wise behavior for composite characters and the like is pretty well established by other languages and the Unicode standard, though:
- If you're doing characterwise UTF-8, then "one character" is one Unicode code point, however many bytes that's encoded as.
- If you want to be Matlab-compatible and are doing UCS-2, then "one character" is always one two-byte UCS-2 code unit/code point
- If you're doing UTF-16, "one character" should probably be one two-byte UTF-16 code unit, not one Unicode code point.
- A Unicode combining character is still technically just one character and one Unicode code point; you don't have to treat them specially at the I/O level. It's up to the application code to determine the semantics of sequences of characters that involve combining characters.

And if you encounter an invalid byte sequence, then I think you should, and pretty much have to, either throw an error, or convert to the Unicode "replacement character", and this behavior should probably be caller-configurable on a per-filehandle basis, and throwing an error should probably be the default.

UCS-2 has no invalid byte sequences.

I agree that the interpretation should mean "characters", which may mean multiple bytes.

I find Nicholas' Matlab-compatibility argument pretty compelling.

Also, I can't think of an actual use case where one might be doing character/text-oriented reading from a stream on a file handle, but would want to specify a number of bytes read instead of number of characters returned. Maybe if you're trying to optimize the buffering and paging behavior of reading from a file? But that's crazy low-level C stuff; surely not something that Octave should be concerned about, right? If that was the case, you'd provided a lower-level non-buffered I/O API, right?

At least for "fgets", we shouldn't remove the "len" argument or we would break Matlab compatibility. [1]

The word "character" isn't used consistently throughout the manual. Sometimes it might mean "character entity" (does that include character modifiers?), sometimes "byte" and sometimes "one element of a vector or matrix of type 'char'".

If we changed how "len" is working right now, the following code snippet might stop working:

str(1:len) = fgets(fid, len);

But that already might fail if the source is non-UTF-8 and a non-ASCII character is read.

[1]: https://de.mathworks.com/help/matlab/ref/fgets.html

While implementing the encoding argument for fopen, the question arose of whether the "len" argument of "fgetl" and "fgets" should correspond to the number of bytes or the number of character entities that are read from an input stream. This is different for encodings that use multiple bytes to encode on single character (e.g. non-ASCII characters in UTF-8).