bugGNU Octave - Bugs: bug #53646, Wrong character encoding in axes...

 
 

bug #53646: Wrong character encoding in axes labels, titles etc.

Submitted by:  Charles Praplan <charprap>
Submitted on:  Sat 14 Apr 2018 07:17:08 AM UTC  
 
Category:  Plotting with OpenGL Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Fixed Assigned to:  None
Originator Name:  Charles Praplan Open/Closed:  Closed
Release:  dev Operating System:  Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 27 May 2018 06:08:16 PM UTC, comment #10:

This is fixed on the current development branch which will be released as Octave 5.0 in probably about half a year.

Markus Mützel <mmuetzel>
Project Member
Wed 09 May 2018 07:31:11 PM UTC, comment #9:

I pushed the patch here:
http://hg.savannah.gnu.org/hgweb/octave/rev/def1b446ba64

It is necessary to run the bootstrap script to pick up the new gnulib function.

Marking as ready for test.

Markus Mützel <mmuetzel>
Project Member
Wed 09 May 2018 06:18:14 PM UTC, comment #8:

The patch in comment #7 ended up with the wrong commit message.
This patch should be better.

(file #44133)

Markus Mützel <mmuetzel>
Project Member
Sun 06 May 2018 08:05:19 PM UTC, comment #7:

The attached patch converts strings to UTF-32 before passing them to FreeType.
It assumes that the source encoding is UTF-8. So bug #53842 should be solved before applying this patch.

(file #44105)

Markus Mützel <mmuetzel>
Project Member
Sun 06 May 2018 11:37:41 AM UTC, comment #6:

The problems with non-ASCII characters and the Windows file system is tracked in bug #49118.

I recently opened a new bug that tracks reading .m files with arbitrary character encodings: bug #53842.

We should keep these issues separate.

The remaining issue here that isn't covered by the other bugs are the changes necessary to get consistent results for plots (axes labels, titles, ...), i.e. converting the strings to UTF-32 encoding before passing them to FreeType (I think).
Re-naming accordingly.

This still leaves the interface to Qt. I haven't tried yet what happens to labels on pushbuttons etc. But if there are also problems, they should be tracked in yet another bug report.

Markus Mützel <mmuetzel>
Project Member
Sun 06 May 2018 11:18:25 AM UTC, comment #5:

One more consequence of this "problem":

If a path or a filename includes such a "special" character, this is again a problem:

An m-file located for example in c:\tmp\Dédé\ cannot be run.

Saving a result in such a folder is also not possible.

Of course it is not the best idea to define such paths, but if the operating system allows them, the softwares running on those systems should also accept them.

Charles Praplan <charprap>
Sun 15 Apr 2018 10:51:21 AM UTC, comment #4:

While I can reproduce your observation with Octave 4.2, the console output no longer differs for the different graphics toolkits with the Octave 4.3.90 (release candidate for Octave 4.4).
In a way that is good. Previously, gnuplot changed the codepage of the command window from 65001 (utf-8) to 1252 (my system codepage "Latin-1"). That should not have happened.
Unfortunately, that still leaves the difference in handling the string encoding in different parts of the program (editor, console and plots).

Markus Mützel <mmuetzel>
Project Member
Sun 15 Apr 2018 09:56:15 AM UTC, comment #3:

Thank you all for answering.

I don't know why the console and the editor are related to graphics, BUT the way how the text is displayed, depends on the selected graphics_toolkit !!

There was a kind of table in my original subscription, showing the results for the different graphic toolkits.
Unfortunately it was not readable, therefore the result in another form:

qt graphics_toolkit:

output in editor : correct
output in console : NOT correct
output on plot : correct

gnuplot graphics_toolkit:

output in editor : correct
output in console : correct
output on plot : correct

fltk graphics_toolkit:

output in editor : correct
output in console : correct NOT correct
output on plot : correct

When changing the graphic toolkit, run the m-file twice, else the result is as for the "old" toolkit.

To illustrate what Markus said, I joint the file encoded as UTF-8. In this case, the situation is different, the text is displayed as desired in the console but not in the editor and in the plots.
In this case the graphic toolkit doesn't not play a role.

(file #43932)

Charles Praplan <charprap>
Sat 14 Apr 2018 01:18:31 PM UTC, comment #2:

That is a known problem on systems where the default editor/system encoding is not UTF-8 (can't find the most relevant bug report right now):
"Windows ANSI" encoding is basically the same as "ISO 8859-1" or "Latin-1". The Command Window always uses UTF-8 encoding to be able to display all Unicode characters.
The Command Window has its own problems in dealing with non-ASCII characters (bug #47571). But if you mark the whole file content and hit F9 the é is "correctly" encoded for the Command Window. But the strings in the plot will show odd.

What Octave probably should do, is use UTF-8 internally as its string encoding everywhere (and convert from whatever encoding is used in the editor/.m files).

While this bug mainly affects Windows (which unfortunately doesn't support UTF-8 as its system codepage), this problem arises on every system that has a system codepage different from UTF-8.

Still present in dev.

Markus Mützel <mmuetzel>
Project Member
Sat 14 Apr 2018 08:59:57 AM UTC, comment #1:

I don't understand what you find is wrong (and thus what you would expect). How are the console and the editor related to graphics toolkits? Why do you call your file "ANSI" (it makes me think of "ANSI C" encoding which doesn't support non ascii "é" character)?

Can you explain a bit more?

Pantxo Diribarne <pantxo>
Project Member
Sat 14 Apr 2018 07:17:08 AM UTC, original submission:

Octave for windows uses different character set in plot, in console and in the editor. Not tested for other OS.

This is graphics_toolkit dependent.

Summary for an ANSI m-file :

qt gnuplot fltk

output in editor correct correct correct
output in console NOT correct correct NOT correct
output on plot correct correct correct

See appended example: Octave_ANSI_files_test_case.m

Charles Praplan <charprap>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44133:  bug53646_freetype_utf32_v2.patch added by mmuetzel (8KiB - application/octet-stream)
file #44105:  bug53646_freetype_utf32.patch added by mmuetzel (9KiB - application/octet-stream)
file #43932:  Octave_UTF8_files_test_case.m added by charprap (140B - application/octet-stream)
file #43925:  Octave_ANSI_files_test_case.m added by charprap (136B - application/octet-stream)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by pantxo (Posted a comment)
  • -email is unavailable- added by charprap (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 15 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-05-27 mmuetzel StatusReady For Test => Fixed
        Open/ClosedOpen => Closed
    2018-05-09 mmuetzel StatusPatch Submitted => Ready For Test
    2018-05-09 mmuetzel Attached File- => Added bug53646_freetype_utf32_v2.patch, #44133
    2018-05-06 mmuetzel StatusConfirmed => Patch Submitted
    2018-05-06 mmuetzel Attached File- => Added bug53646_freetype_utf32.patch, #44105
        CategoryNone => Plotting with OpenGL
        Dependencies- => Depends on bugs #53842
    2018-05-06 mmuetzel Operating SystemMicrosoft Windows => Any
        SummaryDifferent character set in plot, in console and in the editor => Wrong character encoding in axes labels, titles etc.
    2018-04-15 charprap Attached File- => Added Octave_UTF8_files_test_case.m, #43932
    2018-04-14 mmuetzel StatusNeed Info => Confirmed
        Release4.2.2 => dev
    2018-04-14 pantxo StatusNone => Need Info
    2018-04-14 charprap Attached File- => Added Octave_ANSI_files_test_case.m, #43925

    Back to the top


    Powered by Savane 3.3