bugGNU Octave - Bugs: bug #51210, Non-ASCII text displays...

 
 

bug #51210: Non-ASCII text displays incorrectly in plots

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Fri 09 Jun 2017 11:58:28 AM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Duplicate Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 30 Jun 2018 02:25:16 PM UTC, comment #6: 

I forgot that this bug report existed. This was fixed in bug #53646.

Closing as duplicate.

Markus Mützel <mmuetzel>
Group administrator
Tue 20 Jun 2017 02:25:22 PM UTC, comment #5: 

That is right, Matlab uses 16-bit characters. It seems to be using UCS-2:

>> double(char(129296))
ans =
       65535
>> double(native2unicode(129296, 'UTF-32'))
ans =
       65533


All code points outside the BMP seem to be mapped to 65533 ("replacement character").

Any UTF encoding could be used to map the entire Unicode range. Indexing characters generally is difficult in Unicode anyway (e.g. given the character modifiers). Thus, I reasoned there would be no harm in using Linux' "default encoding" UTF-8...

Personally, I rarely (never) use characters outside the BMP. Thus, I am not opposing to limiting ourselves to UCS-2 for compatibility reasons. That would be a big improvement to the current situation nevertheless. However, that might make it more difficult for people using less common writing systems and symbols outside the BMP.

Markus Mützel <mmuetzel>
Group administrator
Tue 20 Jun 2017 01:31:45 PM UTC, comment #4: 

Do we care about Matlab compatibility for character strings?  As I recall, Matlab uses 16-bit characters.

John W. Eaton <jwe>
Group administrator
Tue 20 Jun 2017 10:28:03 AM UTC, comment #3: 

It might be reasonable to convert literal strings from m-files to Unicode (possibly UTF-8) if necessary and use that encoding everywhere internally. If this were the case, Octave (GUI) could set the locale to "en_us.UTF-8" (?) on startup (also on Windows).
This would probably solve other issues as well like strings being displayed in-correctly in the command window and the workspace view...

If adopted, would the lexer be the right place to do that conversion? I could have a look at it if someone could give me a pointer on where (which files) to look.

Markus Mützel <mmuetzel>
Group administrator
Tue 13 Jun 2017 12:16:09 PM UTC, comment #2: 

The behavior on Windows is different. Since the single bytes of a multi byte character (UTF-8) are valid code points (e.g. in ISO-8859-1), no decoding errors occur. Instead a character is displayed for each byte (i.e. two characters for one double byte character).
E.g. the small latin umlaut a ("ä" in UTF-8: 195 164) displays as "ä" corresponding to code points 195 and 164 in ISO-8859-1.

Markus Mützel <mmuetzel>
Group administrator
Mon 12 Jun 2017 06:43:17 PM UTC, comment #1: 

On Debian, if I save a non-ASCII character in a ISO-8859-1 encoded script and run it in Octave, I get the following:


warning: ft_text_renderer: failed to decode string `�' with locale `en_US.UTF-8'
warning: called from
    text at line 156 column 10
    foo at line 1 column 1


and the text is not displayed on the figure at all, or is truncated at the point of the decoding failure.

Does that match your observation (text is messed up) on Windows?

Mike Miller <mtmiller>
Group Member
Fri 09 Jun 2017 11:58:28 AM UTC, original submission:  

As described in bug #51203, text is messed up in plots if the encoding selected for the text editor does not match the system codepage.

Steps to reproduce:
1. Select an encoding for the editor that does not match the system codepage (e.g. UTF-8 on Windows).
2. Add text that contains characters that cannot be encoded in the system codepage (e.g. any non-ASCII character for UTF-8 on Windows).

Even if this mainly affects Windows. This could also occur on other platform when working with files that are not UTF-8 encoded. (Haven't checked.)

I think the locale for C-strings should match the encoding selected in Octave. Otherwise std::mbrtowc fails to correctly convert the multi byte string to a wide character string in ft-text-renderer.cc. Something like this should be set:

std::setlocale (LC_CTYPE, octave_encoding);


However, I do not know neither where this should be done (the interpreter?) nor how to query Octave's current encoding.

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-06-30 mmuetzel StatusConfirmed Duplicate
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code