bugGNU Octave - Bugs: bug #49348, Treat multi-byte characters as one...

 
 

bug #49348: Treat multi-byte characters as one character for char array

Submitted by:  Markus Mützel <mmuetzel>
Submitted on:  Sat 15 Oct 2016 07:30:58 AM UTC  
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Any

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Thu 29 Oct 2020 03:00:39 PM UTC, comment #17: 

Glad, we agree. Just a minor addition:
32 bits are enough to encode all Unicode characters. The encoding that stores the Unicode code point in 32bit units (4 bytes) is called UTF-32. (Plus endianness, …)
Also note that one code point does not necessarily equal one glyph. One glyph could be made up of one or multiple Unicode code points.

Markus Mützel <mmuetzel>
Project Member
Thu 29 Oct 2020 02:59:32 PM UTC, comment #16: 

I'd like to point out how how nice the tenor of the conversation was between HJW and Markus.  Unfortunately, discussion boards often devolve into ranting from either side which helps no one.

Rik <rik5>
Project Administrator
Thu 29 Oct 2020 02:38:22 PM UTC, comment #15: 

Thank you for your comments (and your time, obviously).

I had forgotten you actually need uint64 to properly store all Unicode code points (although you could consider that an encoding as well). Most information sources only talk about 0x0000-0xFFFD when talking about Unicode.

Instead of trusting the Matlab documentation (which claims the default encoding is UTF-8 (going so far as to imply chars are stored internally as UTF-8)), I should have tested it. It turns out Matlab actually uses UTF-16. That explains why I thought it uses uint16 internally: up to 0xFFFF they are equivalent. Probably because of this I had assumed char was just a wrapper for the numeric Unicode code point, which made intuitive sense to me. That is why I expected a 1-to-1 mapping of code point entities and elements of char arrays.

So it seems I need to change gears for Matlab as well. Thank you for taking the time to educate me.

HJW <thrynae>
Thu 29 Oct 2020 01:06:54 PM UTC, comment #14: 

As a rule of thumb when working with strings:
- Casting from a numeric stream to char is problematic. It is important to consider the input encoding (equally in Matlab and in Octave). Use `native2unicode` instead of casting to char.
- It it not save to expect a complete code point entity when indexing into a character array (equally in Matlab and in Octave). Octave has the function `unicode_idx` to somewhat mitigate this issue.

Markus Mützel <mmuetzel>
Project Member
Thu 29 Oct 2020 10:16:40 AM UTC, comment #13: 

You need to specify an encoding. What do you mean by writing "I don't mean UTF-16, I mean uint16"? "uint16" is not an encoding. That is a type.

I'm really trying to help you make your argument.

What does Matlab do for characters beyond the BMP (like most emoji characters that you mentioned before, e.g. 😀)? Does that correspond to one single element when indexing into a character vector?

Markus Mützel <mmuetzel>
Project Member
Thu 29 Oct 2020 09:03:13 AM UTC, comment #12: 

I don't mean UTF-16, I mean uint16, as most chars seem to be stored internally in Matlab. When using functions like strrep, multibyte characters either result in an error (if you keep the class uint8), or lead to a different result from that in Matlab (if you cast to char).

An alternative to completely overhauling Octave might be to overload the size/numel/subsref(/more?) functions for char to hide from the user that their char isn't stored as uint16. That way the interface to the user will be the same as in Matlab, which is what I'm after anyway. I personally don't care if a scalar char is one or more bytes, I just care that it is a single character. I suspect reasonable minds can disagree on that.

As far as the FileExchange is concerned: as the code for most of my submissions is also on GitHub, I don't see why Mathworks would get any say over who is using the code. They even have a category they auto-populate: https://www.mathworks.com/matlabcentral/fileexchange/?category[=1459] (although some of these are about sound). Even if I agreed to that kind of exclusivity, others can still get my code, as it is CC-by-nc-sa (which they (only) allow if you host the code on GitHub).

HJW <thrynae>
Thu 29 Oct 2020 07:50:47 AM UTC, comment #11: 

The emoticon being displayed twice in the command window is probably bug #58637 (which imho deserves a higher priority than this bug here).

Could you give a specific example of something that doesn't work with UTF-8 encoded strings but that would work if the encoding were UTF-16? That would definitely help to support your argument.
Until now I didn't read something actionable.

I am not exactly sure what you mean by "FileExchange". But in case this refers to Matlab's file exchange, please note that it is not clear whether TheMathworks allows to use submissions to that platform in any software other than their own.

Markus Mützel <mmuetzel>
Project Member
Wed 28 Oct 2020 09:19:53 PM UTC, comment #10: 

> I guess volunteers tend to work on the features that interest them.  If you want to bump the priority, would you like to work on this feature or perhaps pay someone to work on it?


Fair point. My problem is that I don't have enough passion for this to go and learn C(++). With my Matlab background, I try to keep all my code compatible, but I think that is where my prowess ends.

I'm afraid my financial means are restricted to the buymeacoffee-level. As I understand it, rewriting char to be 16bit exceeds that by quite a lot.

> It is possible to use UTF-8 characters in Octave.

I was not aware this would work: fprintf('%s\n',uint8([226 140 154]));  (although on my Windows machine that prints the emoji twice). I stand corrected, I should have said 'chars with values above 255'. I will look into the uint8 work-around for my use cases, thank you for the suggestion.

> What do you mean by "text reader"?

Out of what I described in bug #51707 grew my FileExchange submission 68780: readfile() - a general purpose text file reader.

> Which ui elements don't work correctly with UTF-8 characters?

I haven't noticed something missing, hence the distinction between files and UI elements.

HJW <thrynae>
Wed 28 Oct 2020 08:26:42 PM UTC, comment #9: 

> I understand this would take a lot of work, but I don't see why
> this isn't something that has already happened. There are many
> more herculean tasks that have been done, so I don't see why
> this one seems to be on the backburner.


I guess volunteers tend to work on the features that interest them.  If you want to bump the priority, would you like to work on this feature or perhaps pay someone to work on it?

John W. Eaton <jwe>
Project Administrator
Wed 28 Oct 2020 08:21:28 PM UTC, comment #8: 

It is possible to use UTF-8 characters in Octave.

What do you mean by "text reader"?

Which ui elements don't work correctly with UTF-8 characters?

Markus Mützel <mmuetzel>
Project Member
Wed 28 Oct 2020 08:17:33 PM UTC, comment #7: 

Characters beyond 255 are (surprisingly?) common. It isn't just accented characters, it is also Greek symbols. The vast majority of the internet is UTF-8, and while most files will not make use of the full gamut, some will. (especially if you consider emoji, it would be nice if those wouldn't break a text reader)

When dealing with UI elements you can generally make those with the (La)TeX controls, but for file interactions you are forced to hack your way there by writing raw bytes.

I understand this would take a lot of work, but I don't see why this isn't something that has already happened. There are many more herculean tasks that have been done, so I don't see why this one seems to be on the backburner.

My fear is that this is causing stagnation: the case has already been open for so long people have either moved on, or have written their own int16-derived class.

HJW <thrynae>
Wed 28 Oct 2020 07:54:35 PM UTC, comment #6: 

Which severe limitations do you see?

Markus Mützel <mmuetzel>
Project Member
Wed 28 Oct 2020 07:51:00 PM UTC, comment #5: 

I was really hoping to see 16bit chars implemented in Octave 6, but it seems I'm out of luck. Besides this spam-post claiming this is still an issue, is there something else I can do to stimulate work in this regard? Any code working with text is severely limited because of this.

HJW <thrynae>
Sun 23 Oct 2016 07:16:12 PM UTC, comment #4: 

See also bug #43099.
Many of these issues will be solved when MXE builds are made with Qt5 rather than Qt4 together with the patch in bug #49386.

I'll cc John Donoghue as he is the most active mxe-octave dev.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 15 Oct 2016 12:30:49 PM UTC, comment #3: 

You are right, some problems occur in Matlab with Unicode characters that are higher than 65535. Those are clipped to 65535.
So they seem to use UCS-2.
Are there any plans on how to cope with codepage issues in Octave? Should UTF-8 be used everywhere?

Markus Mützel <mmuetzel>
Project Member
Sat 15 Oct 2016 10:17:17 AM UTC, comment #2: 

See also bug #46855 & bug #49222 that undoubtedly motivated this bug report.

I'm sure this has come up in the past several times, though I can't find the bug reports. IIRC it has been mentioned with the symbolic package as well.

There's a relevant wiki page here:
http://wiki.octave.org/International_Characters_Support
See esp. the paragraphs "The state of Octave" and further.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 15 Oct 2016 09:57:16 AM UTC, comment #1: 

Octave uses one byte for char arrays.  Matlab uses wchar_t or 2 bytes for each character.  I believe you can generate the same problems with Matlab if you have a UTF-32 character.

I've marked this as postponed.  This would be a very deep change to the internals of Octave.

Rik <rik5>
Project Administrator
Sat 15 Oct 2016 07:30:58 AM UTC, original submission:  

In Octave, one char seems to be one byte long always. This leads to the following issues when reversing or indexing into a character array with multi-byte characters:

t='abcäöü'
t(end:-1:1)
size(t)
t(4:6)

In Octave:

>> t='abc\303\244\303\266\303\274'
t = abcäöü
>> t(end:-1:1)
ans = �öä�cba
>> size(t)
ans =
   1   9
>> t(4:6)
ans = ä�

In Matlab:

>> t='abcäöü'
t =
abcäöü
>> t(end:-1:1)
ans =
üöäcba
>> size(t)
ans =
     1     6
>> t(4:6)
ans =
äöü

I also noticed that char() doesn't support values >255 and messes up Unicode characters >127:
In Octave:

>> char (269)
warning: range error for conversion to character value
ans =
>> char(228)
ans = �

In Matlab:

>> char(269)
ans =
č
>> char(228)
ans =
ä

Is this a design choice in Octave? Would it be possible that Octave's char class treated Unicode characters as such no matter how many bytes they use in any encoding (UTF-8 seems to be a good choice)?

Markus Mützel <mmuetzel>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by thrynae (Posted a comment)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-10-23 philipnienhuis Carbon-Copy- => Added lostbard
    2016-10-15 philipnienhuis StatusPostponed => None
    2016-10-15 rik5 StatusNone => Postponed

    Back to the top


    Powered by Savane 3.5