bugGNU Octave - Bugs: bug #43099, Missing support for international...

 
 

bug #43099: Missing support for international characters in the GUI console in Windows 7

Submitted by:  Colin Macdonald <cbm>
Submitted on:  Thu 28 Aug 2014 10:00:49 AM UTC  
 
Category:  GUI Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Wed 19 Jul 2017 08:43:54 AM UTC, comment #40: 

@Markus. I understand. Thank you, Markus for answer!

It's not need you, but there is exist - atached screenshot from Octave-utf8-editor in WinXP


Andrey Aleshin Igorevich <iamandreyaleshin>
Wed 19 Jul 2017 08:30:24 AM UTC, comment #39: 

Octave uses the Windows console for its Command Window (on MS Windows). That is the same console that is used by "cmd.exe".
Unfortunately, its support for the "UTF-8 codepage" 65001 is broken in Windows versions prior to Windows 10.
The only way to fix this in Octave would be to completely re-design  how Octave handles input/output at the command line (cross-platform). This major project is on the wish-list. But nobody is working on it at the moment as far as I know.

Markus Mützel <mmuetzel>
Project Member
Wed 19 Jul 2017 06:31:05 AM UTC, comment #38: 

I forgot say: from utf-8 Octave's editor (menu "Edit" -> "Preferences")

Andrey Aleshin Igorevich <iamandreyaleshin>
Wed 19 Jul 2017 06:20:47 AM UTC, comment #37: 

I'm sorry I am connected to this branch. (@Markus: thank you very much you found this branch of bug to me)

I confused because text read good at Win10 and not read at Win7,Win8.

Do I correct understood this bug still is not solved on win7-8?

I attached my pictures:


Andrey Aleshin Igorevich <iamandreyaleshin>
Thu 23 Mar 2017 11:34:15 AM UTC, comment #36: 

Renaming to the title proposed by Hartmut in bug #49386 to better reflect the remaining issue.

Markus Mützel <mmuetzel>
Project Member
Sun 13 Nov 2016 11:03:58 AM UTC, comment #35: 

This bug report could be renamed to something like "Non-ASCII characters not displayed correctly in command window for Windows 7 and earlier" to reflect the remaining issue.
I guess we can leave it open until someone with Windows 7 seems urged to work on it.

Markus Mützel <mmuetzel>
Project Member
Tue 25 Oct 2016 08:54:08 PM UTC, comment #34: 

Hmmm I lost track of where to post comments as regards these character encoding stuff. So many bug reports about encoding & character display ...

With Hartmut's patch (file #38798) in bug #49118, qt4 and Markus' patch (file #38784) here in bug #43099:

- on Win 10 the GUI terminal properly displays UTF8 input from Andreas' .xlsx file in bug #49222 (file #38622)

- on Win7 however the terminal is still broken. As I wrote I think it is a problem with Win7's cmd.exe. When encoding UTF-8 input through Markus utf82unicode.m (see bug #49222) the contents are displayed properly.

My conclusion is that the display of non-ascii in Windows 7 is broken beyond repair. JWE wrote in another bug report that he hopes to have the terminal replaced by a pane/widget independent of bash or cmd.exe.  So it may be better to wait for that to happen.

If you want I can upload the qt4 and qt5 builds (both have the same patches) to my dropbox. Send me a private mail if you want the URLs.

Philip Nienhuis <philipnienhuis>
Project Member
Fri 21 Oct 2016 07:24:31 PM UTC, comment #33: 

With the patch,

  • On Windows 10 I get perfect results, also in the GUI terminal (even w/o the system ("chcp 65001") command)
  • on Windows 7, still no luck as far as the GUI terminal goes.

So there's a difference between Win7 and Win10 :-(
Win7 will be supported until Jan. 14, 2020, so something needs to be done for the Windows builds I suppose.

But at least for the symbolic package it can be tried to get "beauty" like for Linux (see comment #14  :-) )

Philip Nienhuis <philipnienhuis>
Project Member
Fri 21 Oct 2016 03:46:56 PM UTC, comment #32: 

On Win 10, with 4.3.0+ before your path, some results are better.
mice_and_rice look OK, see attached pic.
ellipsize, lquot and rquot look wrong.

I'm waiting for the cross-build to complete & install it.


Philip Nienhuis <philipnienhuis>
Project Member
Fri 21 Oct 2016 03:17:05 PM UTC, comment #31: 

Windows 7.
I can try on Win10 as well, both my desktop & laptop multiboot Win7/Win10/Mageia-5

Philip Nienhuis <philipnienhuis>
Project Member
Fri 21 Oct 2016 10:37:23 AM UTC, comment #30: 

Philip,
Thank you for testing.

It looks like the codepage did not really change for you and the console is still using cp 1252. At least, I get the same output as you with that codepage.

Could you please try with the attached patch? That should set the codepage to UTF-8 (65001) when the console is created.

I am testing on Windwos 10 64bit. Which version of Windows are you using?

(file #38784)

Markus Mützel <mmuetzel>
Project Member
Thu 20 Oct 2016 06:22:48 PM UTC, comment #29: 

Unfortunately I can't confirm that qt5 solves all of it, that is, the terminal is still broken.

I also cross-built Octave-4.3.0+ (--enable-windows-64) with qt5.
Running 'system ("chcp 65001")' and then running testing_utf8.m (using F9 from the editor) gives the attached screenshot.

In the workspace, editor, and history pane the strings come out fine.
The editor merely needs "UTF-8", that also works fine with Qt4.

The terminal and Variable Editor (patch #8656) show garbage.
(Now the Variable Editor is highly experimental, but on Linux it works fine as well as regards encoding.)


Philip Nienhuis <philipnienhuis>
Project Member
Sat 15 Oct 2016 06:50:57 AM UTC, comment #28: 

After some tests (see below), I'd like to propose to change the default codepage for the editor and the command window to UTF-8 on Windows and distribute a version that is compiled against QT5.

I cross-compiled Octave with QT5 and most of the issues seem to be resolved after I change the following manually:
- Change encoding of the editor from "SYSTEM" to "UTF-8" in the properties.
- Execute 'system ("chcp 65001")' in the command window.

After this, when I run Colin's "testing_UTF8.m", I get the following correct output:

>> testing_utf8

timesstr = ×
ellipsize = …
lquot = “
rquot = ”
mice_and_rice = 我爱你 爱着你 就像老鼠爱大米

The characters also display fine in the "Workspace" view. They look weird in 4.2.0-rc2.

Selecting all and pressing F9 yields the following:

>> timesstr = '\303\227'

timesstr = ×
>> ellipsize = '\342\200\246'

ellipsize = …
>> lquot = '\342\200\234'

lquot = “
>> rquot = '\342\200\235'

rquot = ”
>> mice_and_rice = '\346\210\221\347\210\261\344\275\240 \347\210\261\347\235\200\344\275\240 \345\260\261\345\203\217\350\200\201\351\274\240\347\210\261\345\244\2>

mice_and_rice = 我爱你 爱着你 就像老鼠爱大米

This looks like the decoded bytestream in UTF-8.

I still cannot enter umlauts like "äöü" or "ß" in the command window (bug #47571). The keystrokes to these letters are silently ignored. Just a wild guess because I don't know how that actually works: Could this be an issue with readline on Windows which drops non-ASCII characters?

Markus Mützel <mmuetzel>
Project Member
Sun 17 Jul 2016 01:04:46 AM UTC, comment #27: 

I noticed that non ascii characters saved as utf8 are not compatible between on 4.0.x and 4.1.0+ on the octave editor.

I can transfer them using  other editors (e.g notepad++) via clipboard.

Does some other person see this?

Tatsuro MATSUOKA <tmacchant>
Wed 13 Jul 2016 07:58:03 AM UTC, comment #26: 

I propose that

1. bug #44538 and bug #47571 integrated to one
(omit #47571) and title of #44538 is changed to
"command window problem for windows for non-ASCII characters"

2. bug #43099 (this bug) and bug #48306 integrated to one
and remove command windows problem and re-titled.
(I do not find good title at present.
Colin, do you have a good title?)

Tatsuro MATSUOKA <tmacchant>
Wed 13 Jul 2016 03:10:27 AM UTC, comment #25: 

For the bug #44538 and bug #47571 are duplicated to this bug on command window issue.  However, the command windows issue is intrinsically different from others because it is related to windows console but others are not.  The issue for command windows should be treated exasperatedly.

The bug #48306 is related to documentation windows.

The bugs are categorized in two.

1. Issues purely comes from qt-qui.
(editor, workspace pane, and documentation windows)

2. Issue on command windows which dependes both on qt-gui and command console of windows.

Tatsuro MATSUOKA <tmacchant>
Tue 12 Jul 2016 07:12:46 PM UTC, comment #24: 

You didn't mention the documentation browser. Can you test that as well? Some help strings should have some UTF-8 quote characters or bullets.

Can someone look at related bugs and see if there is anything additional there or if they can be closed as duplicates of this report?

Check out bug #44538, bug #47571, and bug #48306.

Mike Miller <mtmiller>
Project Administrator
Sun 03 Jul 2016 06:34:01 AM UTC, comment #23: 

Anyway, I agree the critical thing is the command windows.  +1 for having this on the 4.2 list.

I just tried again on 4.0.3.  What I see is different from the title (but also slightly different from Tatsuro:

Editor: utf-8 looks fine (Chinese and some math chars tested)

Command windows: broken (usual high ascii junk)

Workspace pane: broken (but with different chars).  In my original report, this worked.

I think this on Windows 10. Its a different machine than I originally reported this bug on.

Colin Macdonald <cbm>
Sat 02 Jul 2016 09:15:56 AM UTC, comment #22: 

Is the title correct?

utf-8 files: editor, command windows issues, workspace pane shows correctly

For editor, Japanese characters encoded in utf8 are represented correctly if the font is set properly on "preferences".

I think that the issue is limited to the command window.

Tatsuro MATSUOKA <tmacchant>
Sun 22 Mar 2015 01:10:45 AM UTC, comment #21: 

I had seen that - lots of "in windows you will need to do xxxx and yyyy and ..... "

John Donoghue <lostbard>
Project Member
Sun 22 Mar 2015 12:47:41 AM UTC, comment #20: 

> Windows seems to like utf-16, so maybe all we need to do is
> wrapper any functions to convert between utf-8 and 16.


Is this helpful?  http://utf8everywhere.org

Colin Macdonald <cbm>
Sun 22 Mar 2015 12:41:16 AM UTC, comment #19: 

Kind of hard to tell if its still an issue, but many reports where setting the code page for UTF-8 breaks OS level functions such as ReadFile etc.

Windows seems to like utf-16, so maybe all we need to do is wrapper any functions to convert between utf-8 and 16.

John Donoghue <lostbard>
Project Member
Sat 21 Mar 2015 11:35:54 PM UTC, comment #18: 

@Colin:
No I wasn't offended at all of course, in fact I appreciated the joke. Seriously, there's quite some truth in it....

@JohnD:
Is it really true that "unicode support on Windows is broken at OS level"? I'd rather suspect it's the mingw tools that are at fault. On e.g. stackoverflow there are quite a few reports on unicode issues with mingw.
AFAIK MSVC++ doesn't have that many unicode problems.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 21 Mar 2015 06:18:31 PM UTC, comment #17: 

@Philip: I meant no offense and hope none was taken.  It is definitely important to have Octave working on Windows.

IMHO, the title of this bug is correct:

  * On Windows, workspace pane shows utf-8 strings correctly.

  * On Windows, editor and command terminal show gibberish
    for utf-8 encoded files/strings.

(On GNU/Linux, both are fine.)

@lostbard: understood. I will leave the windows-ascii-default in place in the Symbolic pkg.

Colin Macdonald <cbm>
Sat 21 Mar 2015 11:21:40 AM UTC, comment #16: 

There is now a collection of Windows utf-8 related bugs reported.

Some for display/font issues, some for file/os related functions.

And indications that windows unicode support is broken at the OS level.

So ... its probaly going to take a while to fix the issues.

John Donoghue <lostbard>
Project Member
Sat 21 Mar 2015 10:05:24 AM UTC, comment #15: 

"on GNU/Linux we have beauty. Which is of course true on many levels"

Sure, point taken :-)
[OT] (BTW I'm no windows fan or so. I just happen to work in a "culture" where Windows (and much other proprietary SW) is used for about everything; if they could make coffee with it they'd do that as well. So my primary interest here, apart from fun/hobby, is simply to get/keep Octave running in that environment/culture. No more, no less.

[Back to topic]
So, had this bug better be renamed to "Unicode doesn't work on Windows"? Or just leave it as is, enter a new bug report for unicode-on-windows (if it isn't already there) and add that as a dependency? (that seems to be possible)

What I will do is set "Release" to "dev"

Philip Nienhuis <philipnienhuis>
Project Member
Thu 19 Mar 2015 10:06:03 PM UTC, comment #14: 

> Is it all just a matter of a sympref setting? if so it might be easy to tackle this.


The point of this bug is that unicode doesn't work on Windows.

If I understand correctly, what you are suggesting is already done in symbolic: I detect Windows and default to ugly ascii art there to workaround this bug.  Whereas on GNU/Linux we have beauty. Which is of course true on many levels ;-)

Colin Macdonald <cbm>
Thu 19 Mar 2015 09:54:29 PM UTC, comment #13: 

Sorry for a lte reply (busy week).

With "sympref display unicode" I get the garbled stuff back.
AFAIU unicode isn't supported in the Octave terminal?

But on Windows with default settings it all works fine.

Is it all just a matter of a sympref setting? if so it might be easy to tackle this.

Philip Nienhuis <philipnienhuis>
Project Member
Wed 18 Mar 2015 10:37:33 PM UTC, comment #12: 

My collaborator Andrés says:

> Unfortunately, Octave 4.0.0-rc1 fails to handle unicode
> strings correctly (see the attached screenshot). It has been
> tested using Windows 7 Professional (Service Pack 1).


I will attach a crop of that screenshot.  So no, we're not there yet.


Colin Macdonald <cbm>
Mon 16 Mar 2015 01:46:54 PM UTC, comment #11: 

I noticed the windows binary from ftp://alpha.gnu.org/gnu/octave/ is an ...-installer.exe.  I just tried and I do not have permissions on the terminal server at work to install this.  @Philip: are you able to test this?  Sorry for the hassle.

Colin Macdonald <cbm>
Sun 15 Mar 2015 10:24:28 PM UTC, comment #10: 

Philip, thank you for testing but unfortunately that doesn't really check, because I changed the default to ascii on windows (b/c of this issue).

I'll try to test early this week.  But if easy for you to do so, try redoing your example with "sympref display unicode" (the default on unix).

Colin Macdonald <cbm>
Sun 15 Mar 2015 10:10:26 PM UTC, comment #9: 

With 4.0.0-rc1 & symbolic-2.1.0 I see no more garbled text on the terminal (see attached pic).

Depending on used fonts, several characters used to draw e.g., large parentheses, might not line up very perfectly, but that's a minor issue.

All in all the symbolic package is very usable on Windows now. Thanks!


Philip Nienhuis <philipnienhuis>
Project Member
Sat 14 Mar 2015 07:54:19 AM UTC, comment #8: 

Colin, could you please try the 4.0.0-rc1 build? The changeset http://hg.savannah.gnu.org/hgweb/octave/rev/3db04b75c7c0?revcount=40 might have fixed some of the issues.

Torsten Lilge <ttl>
Project Member
Mon 09 Mar 2015 12:07:42 AM UTC, comment #7: 

Sorry not sure, but almost certainly they are related.

Colin Macdonald <cbm>
Sun 08 Mar 2015 02:44:42 PM UTC, comment #6: 

Is there any difference between this bug and bug #42036?

Mike Miller <mtmiller>
Project Administrator
Wed 18 Feb 2015 09:10:29 PM UTC, comment #5: 

Comment #4 implies that the mxe-build of octave (not the gui) for windows is ansi not utf-8. Is there an easy way to test an unicode build?

Torsten Lilge <ttl>
Project Member
Tue 10 Feb 2015 09:07:49 PM UTC, comment #4: 

I noticed that the command history has the encoding issue, too, but also noticed that this issue disappears if I manually change the encoding of the .octave_hist file in my personal directory to UTF-8 (via Notepad++, ...). So a really minor fix to one the bugs related to the encoding issues would be to make sure the encoding of the generated .octave_hist file is actually UTF-8. (Then the filter works, too). Of course this does not solve the original bug report but since the report is rather general, I posted my observations here.

See screenshots below for a simple demonstration.


Lutz Mager <rocketsound>
Thu 28 Aug 2014 11:38:58 PM UTC, comment #3: 

Yes, that is why I marked it Windows :)

Colin Macdonald <cbm>
Thu 28 Aug 2014 08:31:00 PM UTC, comment #2: 

Works for me on linux in the gui- and cli-version of the gui-release branch.

The font in the editor can be changed in the settings dialog (file menu / preferences) and there in the editor-styles tab. But I guess this won't change anything.

Torsten Lilge <ttl>
Project Member
Thu 28 Aug 2014 10:04:18 AM UTC, comment #1: 

bug #42036 probably related.

Colin Macdonald <cbm>
Thu 28 Aug 2014 10:00:49 AM UTC, original submission:  

See attached screen shot.

Could be a font issue, although I tried changing the command windows from "Courier New" to "DejaVu Sans Mono" (the font I use on GNU/Linux) and it looks essentially the smae (see 2nd screenshot).  I didn't see a way to change the font in the editor.

Its also broken in the "cmd.exe" terminal.

.m file attached.

Colin Macdonald <cbm>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #41241:  wxp.JPG added by iamandreyaleshin (35KiB - image/jpeg)
file #41240:  w7-8.jpg added by iamandreyaleshin (16KiB - image/jpeg)
file #38777:  bug43099.png added by philipnienhuis (43KiB - image/png)
file #33047:  Screenshot_1.png added by rocketsound (12KiB - image/png)
file #33048:  Screenshot_2.png added by rocketsound (6KiB - image/png)
file #31990:  testing_utf8.m added by cbm (124B - text/x-objcsrc)

 

Carbon-Copy List
  • -email is unavailable- added by iamandreyaleshin (Updated the item)
  • -email is unavailable- added by hardy
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by tiagoma
  • -email is unavailable- added by tmacchant (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by rocketsound (Updated the item)
  • -email is unavailable- added by ttl (Posted a comment)
  • -email is unavailable- added by cbm (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 22 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-07-19 iamandreyaleshin Attached File- => Added wxp.JPG, #41241
    2017-07-19 iamandreyaleshin Attached File- => Added w10(w7 winupdated).jpg, #41239
        Attached File- => Added w7-8.jpg, #41240
    2017-03-23 mmuetzel Summaryutf-8 files: editor, command windows issues, workspace pane shows correctly => Missing support for international characters in the GUI console in Windows 7
    2017-03-23 mmuetzel Dependencies- => bugs #49386 is dependent
    2017-03-23 mmuetzel Dependencies- => bugs #44538 is dependent
    2016-10-26 hardy Carbon-Copy- => Added hardy
    2016-10-26 hardy Carbon-CopyRemoved hartmut => -
    2016-10-26 hardy Carbon-Copy- => Added hartmut
    2016-10-21 philipnienhuis Attached File- => Added bug_43099#2_win10.png, #38786
    2016-10-21 mmuetzel Attached File- => Added bug43099_console_cp65001.patch, #38784
    2016-10-20 philipnienhuis Attached File- => Added bug43099.png, #38777
    2016-08-04 tiagoma Carbon-Copy- => Added tiagoma
    2015-09-10 mtmiller Dependencies- => bugs #45909 is dependent
    2015-03-21 philipnienhuis Release3.8.2 => dev
    2015-03-18 cbm Attached File- => Added octave4.0.0-rc1_utf8_fail.png, #33395
    2015-03-15 philipnienhuis Attached File- => Added symbolic-2.1.0_in_4.0.0-rc1.png, #33356
    2015-02-10 rocketsound Attached File- => Added Screenshot_1.png, #33047
        Attached File- => Added Screenshot_2.png, #33048
    2014-08-28 cbm Attached File- => Added Screenshot from 2014-08-28 10:52:32.png, #31989
        Attached File- => Added testing_utf8.m, #31990
        Attached File- => Added Screenshot from 2014-08-28 10:58:03.png, #31991

    Back to the top


    Powered by Savane 3.5