Re-opening because I hope this will still be fixed some day when the cross-platform console widget will be released.

@Markus. I understand. Thank you, Markus for answer!

It's not need you, but there is exist - atached screenshot from Octave-utf8-editor in WinXP

I forgot say: from utf-8 Octave's editor (menu "Edit" -> "Preferences")

Renaming to the title proposed by Hartmut in bug #49386 to better reflect the remaining issue.

Hmmm I lost track of where to post comments as regards these character encoding stuff. So many bug reports about encoding & character display ...

With Hartmut's patch (file #38798) in bug #49118, qt4 and Markus' patch (file #38784) here in bug #43099:

- on Win 10 the GUI terminal properly displays UTF8 input from Andreas' .xlsx file in bug #49222 (file #38622)

- on Win7 however the terminal is still broken. As I wrote I think it is a problem with Win7's cmd.exe. When encoding UTF-8 input through Markus utf82unicode.m (see bug #49222) the contents are displayed properly.

My conclusion is that the display of non-ascii in Windows 7 is broken beyond repair. JWE wrote in another bug report that he hopes to have the terminal replaced by a pane/widget independent of bash or cmd.exe.  So it may be better to wait for that to happen.

If you want I can upload the qt4 and qt5 builds (both have the same patches) to my dropbox. Send me a private mail if you want the URLs.

On Win 10, with 4.3.0+ before your path, some results are better.
mice_and_rice look OK, see attached pic.
ellipsize, lquot and rquot look wrong.

I'm waiting for the cross-build to complete & install it.

Philip,
Thank you for testing.

It looks like the codepage did not really change for you and the console is still using cp 1252. At least, I get the same output as you with that codepage.

Could you please try with the attached patch? That should set the codepage to UTF-8 (65001) when the console is created.

I am testing on Windwos 10 64bit. Which version of Windows are you using?

(file #38784)

After some tests (see below), I'd like to propose to change the default codepage for the editor and the command window to UTF-8 on Windows and distribute a version that is compiled against QT5.

I cross-compiled Octave with QT5 and most of the issues seem to be resolved after I change the following manually:
- Change encoding of the editor from "SYSTEM" to "UTF-8" in the properties.
- Execute 'system ("chcp 65001")' in the command window.

After this, when I run Colin's "testing_UTF8.m", I get the following correct output:

>> testing_utf8

timesstr = ×
ellipsize = …
lquot = “
rquot = ”
mice_and_rice = 我爱你 爱着你 就像老鼠爱大米

The characters also display fine in the "Workspace" view. They look weird in 4.2.0-rc2.

Selecting all and pressing F9 yields the following:

>> timesstr = '\303\227'

timesstr = ×
>> ellipsize = '\342\200\246'

ellipsize = …
>> lquot = '\342\200\234'

lquot = “
>> rquot = '\342\200\235'

rquot = ”
>> mice_and_rice = '\346\210\221\347\210\261\344\275\240 \347\210\261\347\235\200\344\275\240 \345\260\261\345\203\217\350\200\201\351\274\240\347\210\261\345\244\2>

mice_and_rice = 我爱你 爱着你 就像老鼠爱大米

This looks like the decoded bytestream in UTF-8.

I still cannot enter umlauts like "äöü" or "ß" in the command window (bug #47571). The keystrokes to these letters are silently ignored. Just a wild guess because I don't know how that actually works: Could this be an issue with readline on Windows which drops non-ASCII characters?

I propose that

1. bug #44538 and bug #47571 integrated to one
(omit #47571) and title of #44538 is changed to
"command window problem for windows for non-ASCII characters"

2. bug #43099 (this bug) and bug #48306 integrated to one
and remove command windows problem and re-titled.
(I do not find good title at present.
Colin, do you have a good title?)

You didn't mention the documentation browser. Can you test that as well? Some help strings should have some UTF-8 quote characters or bullets.

Can someone look at related bugs and see if there is anything additional there or if they can be closed as duplicates of this report?

Check out bug #44538, bug #47571, and bug #48306.

Is the title correct?

utf-8 files: editor, command windows issues, workspace pane shows correctly

For editor, Japanese characters encoded in utf8 are represented correctly if the font is set properly on "preferences".

I think that the issue is limited to the command window.

> Windows seems to like utf-16, so maybe all we need to do is
> wrapper any functions to convert between utf-8 and 16.

Is this helpful?  http://utf8everywhere.org

@Colin:
No I wasn't offended at all of course, in fact I appreciated the joke. Seriously, there's quite some truth in it....

@JohnD:
Is it really true that "unicode support on Windows is broken at OS level"? I'd rather suspect it's the mingw tools that are at fault. On e.g. stackoverflow there are quite a few reports on unicode issues with mingw.
AFAIK MSVC++ doesn't have that many unicode problems.

There is now a collection of Windows utf-8 related bugs reported.

Some for display/font issues, some for file/os related functions.

And indications that windows unicode support is broken at the OS level.

So ... its probaly going to take a while to fix the issues.

> Is it all just a matter of a sympref setting? if so it might be easy to tackle this.

The point of this bug is that unicode doesn't work on Windows.

If I understand correctly, what you are suggesting is already done in symbolic: I detect Windows and default to ugly ascii art there to workaround this bug.  Whereas on GNU/Linux we have beauty. Which is of course true on many levels ;-)

My collaborator Andrés says:

> Unfortunately, Octave 4.0.0-rc1 fails to handle unicode
> strings correctly (see the attached screenshot). It has been
> tested using Windows 7 Professional (Service Pack 1).

I will attach a crop of that screenshot.  So no, we're not there yet.

Philip, thank you for testing but unfortunately that doesn't really check, because I changed the default to ascii on windows (b/c of this issue).

I'll try to test early this week.  But if easy for you to do so, try redoing your example with "sympref display unicode" (the default on unix).

Colin, could you please try the 4.0.0-rc1 build? The changeset http://hg.savannah.gnu.org/hgweb/octave/rev/3db04b75c7c0?revcount=40 might have fixed some of the issues.

Is there any difference between this bug and bug #42036?

I noticed that the command history has the encoding issue, too, but also noticed that this issue disappears if I manually change the encoding of the .octave_hist file in my personal directory to UTF-8 (via Notepad++, ...). So a really minor fix to one the bugs related to the encoding issues would be to make sure the encoding of the generated .octave_hist file is actually UTF-8. (Then the filter works, too). Of course this does not solve the original bug report but since the report is rather general, I posted my observations here.

See screenshots below for a simple demonstration.

Works for me on linux in the gui- and cli-version of the gui-release branch.

The font in the editor can be changed in the settings dialog (file menu / preferences) and there in the editor-styles tab. But I guess this won't change anything.

See attached screen shot.

Could be a font issue, although I tried changing the command windows from "Courier New" to "DejaVu Sans Mono" (the font I use on GNU/Linux) and it looks essentially the smae (see 2nd screenshot).  I didn't see a way to change the font in the editor.

Its also broken in the "cmd.exe" terminal.

.m file attached.