bugGNU Octave - Bugs: bug #49348, Treat multi-byte characters as one...

 
 

bug #49348: Treat multi-byte characters as one character for char array

Submitted by:  Markus Mützel <mmuetzel>
Submitted on:  Sat 15 Oct 2016 07:30:58 AM UTC  
 
Category: InterpreterSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Matlab Compatibility
Status: NoneAssigned to: None
Originator Name: Open/Closed: Open
Release: devOperating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 23 Oct 2016 07:16:12 PM UTC, comment #4:

See also bug #43099.
Many of these issues will be solved when MXE builds are made with Qt5 rather than Qt4 together with the patch in bug #49386.

I'll cc John Donoghue as he is the most active mxe-octave dev.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 15 Oct 2016 12:30:49 PM UTC, comment #3:

You are right, some problems occur in Matlab with Unicode characters that are higher than 65535. Those are clipped to 65535.
So they seem to use UCS-2.
Are there any plans on how to cope with codepage issues in Octave? Should UTF-8 be used everywhere?

Markus Mützel <mmuetzel>
Project Member
Sat 15 Oct 2016 10:17:17 AM UTC, comment #2:

See also bug #46855 & bug #49222 that undoubtedly motivated this bug report.

I'm sure this has come up in the past several times, though I can't find the bug reports. IIRC it has been mentioned with the symbolic package as well.

There's a relevant wiki page here:
http://wiki.octave.org/International_Characters_Support
See esp. the paragraphs "The state of Octave" and further.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 15 Oct 2016 09:57:16 AM UTC, comment #1:

Octave uses one byte for char arrays. Matlab uses wchar_t or 2 bytes for each character. I believe you can generate the same problems with Matlab if you have a UTF-32 character.

I've marked this as postponed. This would be a very deep change to the internals of Octave.

Rik <rik5>
Project Administrator
Sat 15 Oct 2016 07:30:58 AM UTC, original submission:

In Octave, one char seems to be one byte long always. This leads to the following issues when reversing or indexing into a character array with multi-byte characters:

In Octave:

In Matlab:

I also noticed that char() doesn't support values >255 and messes up Unicode characters >127:
In Octave:

In Matlab:

Is this a design choice in Octave? Would it be possible that Octave's char class treated Unicode characters as such no matter how many bytes they use in any encoding (UTF-8 seems to be a good choice)?

Markus Mützel <mmuetzel>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by philipnienhuis
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by mmuetzel (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 23 Oct 2016 07:16:12 PM UTCphilipnienhuisCarbon-Copy-=>Added lostbard
    Sat 15 Oct 2016 10:17:17 AM UTCphilipnienhuisStatusPostponed=>None
    Sat 15 Oct 2016 09:57:16 AM UTCrik5StatusNone=>Postponed

    Back to the top


    Powered by Savane 3.1-cleanup1