bugGNU Octave - Bugs: bug #51210, Non-ASCII text displays...

 
 

bug #51210: Non-ASCII text displays incorrectly in plots

Submitted by:  Markus Mützel <mmuetzel>
Submitted on:  Fri 09 Jun 2017 11:58:28 AM UTC  
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Duplicate Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  dev Operating System:  Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sat 30 Jun 2018 02:25:16 PM UTC, comment #6:

I forgot that this bug report existed. This was fixed in bug #53646.

Closing as duplicate.

Markus Mützel <mmuetzel>
Project Member
Tue 20 Jun 2017 02:25:22 PM UTC, comment #5:

That is right, Matlab uses 16-bit characters. It seems to be using UCS-2:

All code points outside the BMP seem to be mapped to 65533 ("replacement character").

Any UTF encoding could be used to map the entire Unicode range. Indexing characters generally is difficult in Unicode anyway (e.g. given the character modifiers). Thus, I reasoned there would be no harm in using Linux' "default encoding" UTF-8...

Personally, I rarely (never) use characters outside the BMP. Thus, I am not opposing to limiting ourselves to UCS-2 for compatibility reasons. That would be a big improvement to the current situation nevertheless. However, that might make it more difficult for people using less common writing systems and symbols outside the BMP.

Markus Mützel <mmuetzel>
Project Member
Tue 20 Jun 2017 01:31:45 PM UTC, comment #4:

Do we care about Matlab compatibility for character strings? As I recall, Matlab uses 16-bit characters.

John W. Eaton <jwe>
Project Administrator
Tue 20 Jun 2017 10:28:03 AM UTC, comment #3:

It might be reasonable to convert literal strings from m-files to Unicode (possibly UTF-8) if necessary and use that encoding everywhere internally. If this were the case, Octave (GUI) could set the locale to "en_us.UTF-8" (?) on startup (also on Windows).
This would probably solve other issues as well like strings being displayed in-correctly in the command window and the workspace view...

If adopted, would the lexer be the right place to do that conversion? I could have a look at it if someone could give me a pointer on where (which files) to look.

Markus Mützel <mmuetzel>
Project Member
Tue 13 Jun 2017 12:16:09 PM UTC, comment #2:

The behavior on Windows is different. Since the single bytes of a multi byte character (UTF-8) are valid code points (e.g. in ISO-8859-1), no decoding errors occur. Instead a character is displayed for each byte (i.e. two characters for one double byte character).
E.g. the small latin umlaut a ("ä" in UTF-8: 195 164) displays as "ä" corresponding to code points 195 and 164 in ISO-8859-1.

Markus Mützel <mmuetzel>
Project Member
Mon 12 Jun 2017 06:43:17 PM UTC, comment #1:

On Debian, if I save a non-ASCII character in a ISO-8859-1 encoded script and run it in Octave, I get the following:

and the text is not displayed on the figure at all, or is truncated at the point of the decoding failure.

Does that match your observation (text is messed up) on Windows?

Mike Miller <mtmiller>
Project Administrator
Fri 09 Jun 2017 11:58:28 AM UTC, original submission:

As described in bug #51203, text is messed up in plots if the encoding selected for the text editor does not match the system codepage.

Steps to reproduce:
1. Select an encoding for the editor that does not match the system codepage (e.g. UTF-8 on Windows).
2. Add text that contains characters that cannot be encoded in the system codepage (e.g. any non-ASCII character for UTF-8 on Windows).

Even if this mainly affects Windows. This could also occur on other platform when working with files that are not UTF-8 encoded. (Haven't checked.)

I think the locale for C-strings should match the encoding selected in Octave. Otherwise std::mbrtowc fails to correctly convert the multi byte string to a wide character string in ft-text-renderer.cc. Something like this should be set:

However, I do not know neither where this should be done (the interpreter?) nor how to query Octave's current encoding.

Markus Mützel <mmuetzel>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-06-30 mmuetzel StatusConfirmed => Duplicate
        Open/ClosedOpen => Closed

    Back to the top


    Powered by Savane 3.3