bugGNU Octave - Bugs: bug #53646, Wrong character encoding in axes...

 
 

bug #53646: Wrong character encoding in axes labels, titles etc.

Submitter:  Charles Praplan <charprap>
Submitted:  Sat 14 Apr 2018 07:17:08 AM UTC
   
 
Category:  Plotting with OpenGL Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Fixed Assigned to:  None
Originator Name:  Charles Praplan Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 10 Feb 2019 09:39:50 AM UTC, comment #26: 

The last part is probably because the script is stored in the symbol table. Executing "clear functions" should reset the symbol table and give different results after changing the m-file encoding.
The file editor sometimes lacks a little bit behind when the encoding is changed. Usually, closing the file and reopening it should display it correctly.

As to your problem with the command window not displaying utf-8 encoded characters correctly: I have no clue what is going on there. The fact that you see 4 diamonds when executing the utf-8 encoded file with your system encoding (probably code page 1251 for an English Windows) suggests that the correct utf-8 encoded characters are sent to the console.
I suspect this shows that the Windows console in Windows 10 still only half-heartedly supports utf-8. I hope that jwe's plan to switch to a cross-platform text widget for the command window will solve this in the future.

In the meantime, could you please open a new bug report for the issue you are seeing? The original issue of this bug report (character encoding wrong when plotting) seems to be resolved.

Markus Mützel <mmuetzel>
Group administrator
Sun 10 Feb 2019 09:12:31 AM UTC, comment #25: 

When executing
_mfile_encoding_ utf-8
Octave_UTF8_files_test_case

I get:

- console : same as before (2 diamonds)
- plots   : ok
- editor  : NOT OK : disp('Résistances');


If I execute once Octave_UTF8_files_test_case
before
_mfile_encoding_ utf-8
Octave_UTF8_files_test_case

it is likely the command _mfile_encoding_ utf-8 has no effect:

- console : 4 diamonds
- plots   : NOT OK : Résistances
- editor  : NOT OK : disp('Résistances');

Charles Praplan <charprap>
Sat 09 Feb 2019 05:37:35 PM UTC, comment #24: 

If you execute the following with file #43932, do you still see the replacement characters in the command window?

__mfile_encoding__ utf-8
Octave_UTF8_files_test_case


Markus Mützel <mmuetzel>
Group administrator
Sat 09 Feb 2019 05:01:32 PM UTC, comment #23: 

1)
I have an English Windows 10 1803 [Version 10.0.17134.523]


2)
C:\>chcp
Active code page: 437


3)
Yes I really see the the question mark inside a diamond!

4)
I think I did not change my readline settings (at least not intentionally) and I do not have a ".inputrc" file in my user folder.
I have a .octaverc file in which I only extend the path and load a package.

Charles Praplan <charprap>
Sat 09 Feb 2019 01:05:58 PM UTC, comment #22: 

This looks all perfectly fine. I am currently at a loss as to why it would work for me (German Windows 10 1810) but doesn't for you.

Which locale is your Windows? What do you see if you open a Windows command prompt ("cmd.exe") and execute "chcp"?

Do you really see two replacement characters (the question mark inside a diamond)? Or is this something that changed when pasting in savannah? Can you upload a screen shot?

Did you change your readline settings? Do you have a ".inputrc" in your user folder (typically at "C:\Users\YOURUSERNAME")?

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Feb 2019 12:26:34 PM UTC, comment #21: 


>> _mfile_encoding_

ans = system

>>

-------------------------------
Courier New   10

IBeam Cursor

Use foreground color checked

history buffer Size 1000

Two other points not checked

Charles Praplan <charprap>
Sat 09 Feb 2019 12:10:45 PM UTC, comment #20: 

That looks good so far.
What do you see for "__mfile_encoding__"?
Which font do you use for the command window ("Edit" menu -> "Preferences..." -> "Command" tab -> "Font" drop down list)?

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Feb 2019 12:00:56 PM UTC, comment #19: 


>> system chcp

Active code page: 65001
ans = 0

>>

Charles Praplan <charprap>
Sat 09 Feb 2019 11:52:34 AM UTC, comment #18: 

What do you see if you type "system chcp" in the command window?

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Feb 2019 11:50:37 AM UTC, comment #17: 

I tried both. The figure are ok in both cases.
The text in the console is not ok in both cases (but looks different).

Charles Praplan <charprap>
Sat 09 Feb 2019 11:40:04 AM UTC, comment #16: 

Do you use the GUI or CLI?

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Feb 2019 11:37:59 AM UTC, comment #15: 

Yes the text in the figure appears correctly.

Charles Praplan <charprap>
Sat 09 Feb 2019 10:00:44 AM UTC, comment #14: 

This works for me with the GUI. It doesn't work with the CLI (see bug #50409).

Does the text in the figure appear correctly?

Markus Mützel <mmuetzel>
Group administrator
Sat 09 Feb 2019 09:48:36 AM UTC, comment #13: 

I just tried version 5.0.91 and the problem is still there.

At least using test file 43925, the following appears in the console: R��sistances

for previous version it was: R�sistances

I use windows 10.

Charles Praplan <charprap>
Sat 04 Aug 2018 07:55:32 AM UTC, comment #12: 

Please see comment #10.

Markus Mützel <mmuetzel>
Group administrator
Sat 04 Aug 2018 07:20:45 AM UTC, comment #11: 

Should this be corrected in 4.4.1-rc2 ?

This is not the case!

Charles Praplan <charprap>
Sun 27 May 2018 06:08:16 PM UTC, comment #10: 

This is fixed on the current development branch which will be released as Octave 5.0 in probably about half a year.

Markus Mützel <mmuetzel>
Group administrator
Wed 09 May 2018 07:31:11 PM UTC, comment #9: 

I pushed the patch here:
http://hg.savannah.gnu.org/hgweb/octave/rev/def1b446ba64

It is necessary to run the bootstrap script to pick up the new gnulib function.

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Wed 09 May 2018 06:18:14 PM UTC, comment #8: 

The patch in comment #7 ended up with the wrong commit message.
This patch should be better.

(file #44133)

Markus Mützel <mmuetzel>
Group administrator
Sun 06 May 2018 08:05:19 PM UTC, comment #7: 

The attached patch converts strings to UTF-32 before passing them to FreeType.
It assumes that the source encoding is UTF-8. So bug #53842 should be solved before applying this patch.

(file #44105)

Markus Mützel <mmuetzel>
Group administrator
Sun 06 May 2018 11:37:41 AM UTC, comment #6: 

The problems with non-ASCII characters and the Windows file system is tracked in bug #49118.

I recently opened a new bug that tracks reading .m files with arbitrary character encodings: bug #53842.

We should keep these issues separate.

The remaining issue here that isn't covered by the other bugs are the changes necessary to get consistent results for plots (axes labels, titles, ...), i.e. converting the strings to UTF-32 encoding before passing them to FreeType (I think).
Re-naming accordingly.

This still leaves the interface to Qt. I haven't tried yet what happens to labels on pushbuttons etc. But if there are also problems, they should be tracked in yet another bug report.

Markus Mützel <mmuetzel>
Group administrator
Sun 06 May 2018 11:18:25 AM UTC, comment #5: 

One more consequence of this "problem":

If a path or a filename includes such a "special" character, this  is again a problem:

An m-file located for example in c:\tmp\Dédé\ cannot be run.

Saving a result in such a folder is also not possible.

Of course it is not the best idea to define such paths, but if the operating system allows them, the softwares running on those systems should also accept them.

Charles Praplan <charprap>
Sun 15 Apr 2018 10:51:21 AM UTC, comment #4: 

While I can reproduce your observation with Octave 4.2, the console output no longer differs for the different graphics toolkits with the Octave 4.3.90 (release candidate for Octave 4.4).
In a way that is good. Previously, gnuplot changed the codepage of the command window from 65001 (utf-8) to 1252 (my system codepage "Latin-1"). That should not have happened.
Unfortunately, that still leaves the difference in handling the string encoding in different parts of the program (editor, console and plots).

Markus Mützel <mmuetzel>
Group administrator
Sun 15 Apr 2018 09:56:15 AM UTC, comment #3: 

Thank you all for answering.

I don't know why the console and the editor are related to graphics, BUT the way how the text is displayed, depends on the selected graphics_toolkit !!

There was a kind of table in my original subscription, showing the results for the different graphic toolkits.
Unfortunately it was not readable, therefore the result in another form:


qt graphics_toolkit:

  output in editor  : correct    
  output in console : NOT correct
  output on plot    : correct    
 
 
gnuplot graphics_toolkit:

  output in editor  : correct
  output in console : correct
  output on plot    : correct
    
    
fltk graphics_toolkit:  

  output in editor  : correct
  output in console : correct NOT correct
  output on plot    : correct


When changing the graphic toolkit, run the m-file twice, else the result is as for the "old" toolkit.

To illustrate what Markus said, I joint the file encoded as UTF-8. In this case, the situation is different, the text is displayed as desired in the console but not in the editor and in the plots.
In this case the graphic toolkit doesn't not play a role.


(file #43932)

Charles Praplan <charprap>
Sat 14 Apr 2018 01:18:31 PM UTC, comment #2: 

That is a known problem on systems where the default editor/system encoding is not UTF-8 (can't find the most relevant bug report right now):
"Windows ANSI" encoding is basically the same as "ISO 8859-1" or "Latin-1". The Command Window always uses UTF-8 encoding to be able to display all Unicode characters.
The Command Window has its own problems in dealing with non-ASCII characters (bug #47571). But if you mark the whole file content and hit F9 the é is "correctly" encoded for the Command Window. But the strings in the plot will show odd.

What Octave probably should do, is use UTF-8 internally as its string encoding everywhere (and convert from whatever encoding is used in the editor/.m files).

While this bug mainly affects Windows (which unfortunately doesn't support UTF-8 as its system codepage), this problem arises on every system that has a system codepage different from UTF-8.

Still present in dev.

Markus Mützel <mmuetzel>
Group administrator
Sat 14 Apr 2018 08:59:57 AM UTC, comment #1: 

I don't understand what you find is wrong (and thus what you would expect). How are the console and the editor related to graphics toolkits? Why do you call your file "ANSI" (it makes me think of "ANSI C" encoding which doesn't support non ascii "é" character)?

Can you explain a bit more?

Pantxo Diribarne <pantxo>
Group Member
Sat 14 Apr 2018 07:17:08 AM UTC, original submission:  

Octave for windows uses different character set in plot, in console and in the editor. Not tested for other OS.

This is graphics_toolkit dependent.

Summary for an ANSI m-file :

                    qt            gnuplot   fltk
                                           
output in editor    correct       correct   correct
output in console   NOT correct   correct   NOT correct
output on plot      correct       correct   correct

See appended example: Octave_ANSI_files_test_case.m

Charles Praplan <charprap>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44133:  bug53646_freetype_utf32_v2.patch added by mmuetzel (8KiB - application/octet-stream)
file #44105:  bug53646_freetype_utf32.patch added by mmuetzel (9KiB - application/octet-stream)
file #43932:  Octave_UTF8_files_test_case.m added by charprap (140B - application/octet-stream)
file #43925:  Octave_ANSI_files_test_case.m added by charprap (136B - application/octet-stream)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by pantxo (Posted a comment)
  • -email is unavailable- added by charprap (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 15 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-05-27 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2018-05-09 mmuetzel StatusPatch Submitted Ready For Test
    2018-05-09 mmuetzel Attached File- Added bug53646_freetype_utf32_v2.patch, #44133
    2018-05-06 mmuetzel StatusConfirmed Patch Submitted
    2018-05-06 mmuetzel Attached File- Added bug53646_freetype_utf32.patch, #44105
        CategoryNone Plotting with OpenGL
        Dependencies- Depends on bugs #53842
    2018-05-06 mmuetzel Operating SystemMicrosoft Windows Any
        SummaryDifferent character set in plot, in console and in the editor Wrong character encoding in axes labels, titles etc.
    2018-04-15 charprap Attached File- Added Octave_UTF8_files_test_case.m, #43932
    2018-04-14 mmuetzel StatusNeed Info Confirmed
        Release4.2.2 dev
    2018-04-14 pantxo StatusNone Need Info
    2018-04-14 charprap Attached File- Added Octave_ANSI_files_test_case.m, #43925

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code