bugGNU Octave - Bugs: bug #49386, Non-latin characters in GUI

 
 

bug #49386: Non-latin characters in GUI

Submitter:  Igor Melentyev <igor_melentyev>
Submitted:  Wed 19 Oct 2016 04:38:21 PM UTC
   
 
Category:  GUI Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Other
Status:  Duplicate Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 4.0.3 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 23 Mar 2017 11:31:37 AM UTC, comment #21: 

Closing this bug as a duplicate of bug #43099 where the remaining issue as described by Hartmut in comment #20 can be tracked.

Markus Mützel <mmuetzel>
Group administrator
Mon 31 Oct 2016 10:23:44 PM UTC, comment #20: 

The "65001-patch" (file #38784) has now also been applied to the repository: 8c749f33ce0a . So it should be part of the current Octave pre-release 4.2.0-rc3.

If I am not mistaken, the only remaining issue of this bug report is now the "missing support for international characters in the GUI console in Windows 7". We might change the title accordingly.

Hartmut <hardy>
Thu 27 Oct 2016 09:00:20 PM UTC, comment #19: 
Philip Nienhuis <philipnienhuis>
Group Member
Thu 27 Oct 2016 06:22:43 PM UTC, comment #18: 

What is "65001-patch"?  URL please?

John W. Eaton <jwe>
Group administrator
Thu 27 Oct 2016 06:01:16 PM UTC, comment #17: 

The "cstrings-patch" has just been applied to the default branch. So I think the only remaining issue (that has so far been mentioned in this bug report) are utf8 characters in the GUI console of Windows Octave. Could we change the title accordingly? (This might help to not totally loose track of all those current text incoding bug reports...)

This issue could further be narrowed down by applying the "65001-patch", which seem to fix this remaining text encoding of the Windows console at least for Win10 (but not for Win7).

Hartmut <hardy>
Wed 26 Oct 2016 03:08:57 PM UTC, comment #16: 

@Hartmut:  As to your 5th bullet remark:
The GUI console in Windows (up till at least Win7, I don't know about 8 or 8.1) has never worked as good as bash and friends.

Earlier Octave versions for Windows (from before mxe-octave & the GUI) invoked mingw-based terminals (for example, rxvt in octave-3.2.4). These may have performed better, but from that time we have little or no bug reports about non-ascii character display AFAIK.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 26 Oct 2016 06:38:03 AM UTC, comment #15: 

I have now tested default-octave built via mxe-octave with


The results under WinXP are fully the same as without the "toLocal8Bit-patch": GUI editor is fine, inputdlg is fine, variable explorer is fine, but the GUI console is as bad as before. Typing umlauts in the GUI console also still does not work. I will attach another screenshot to visualize this. So this "toLocal8bit-patch" does NOT HELP for anything (at least not under WinXP).

So I would agree with Philip's conclusions in bug #43099 comment #34:

  • We should use the "cstrings-patch", it fixes the text coding issues with Octave GUI windows, except for the console.
  • We should also use the "65001-patch", it fixes the GUI console at least for Win10.
  • Making the GUI console work with utf8 characters under Win7 seems to be harder. (But has this ever worked before under Windows? Otherwise this is "only" a bug, but it is not a regression like some of the other mentioned behaviors.)



Hartmut <hardy>
Tue 25 Oct 2016 09:15:12 PM UTC, comment #14: 

Just for the records (in this bug report): Markus' "65001-patch" DOES HELP with displaying UTF8 characters in the GUI console, but only under Win10 (as reported by Philip in bug #43099), and not under Win7 or WinXP.

Hartmut <hardy>
Tue 25 Oct 2016 08:56:20 PM UTC, comment #13: 

See also my comment #34 in bug #43099.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 25 Oct 2016 08:52:47 PM UTC, comment #12: 

(My next mxe-octave built is ready.)

I have now tested the same (fairly recent) default-octave built via mxe-octave, and applied


The results under WinXP are fully the same as without the "65001-patch": GUI editor is fine, inputdlg is fine, variable explorer is fine, but the GUI console is as bad as before. Typing umlauts in the GUI console also still does not work. I will attach a screenshot to visualize this.

So my conclusion would be that this "65001-patch" does NOT HELP for anything discussed so far. I have also not found any evicende in bug #43099 that Philip observed any improvement under Windows after applying this patch.

(I will now go ahead and built an mxe-octave with my "cstrings-patch" applied, as well as my "toLocal8Bit-patch".)






Hartmut <hardy>
Tue 25 Oct 2016 07:53:14 PM UTC, comment #11: 

(My mxe built is curently running, using only my "cstrings-patch" and your "65001-patch" so far.)

No, I cannot type any umlauts in the GUI console as well. Neither with Octave-4.2.0-rc2 under WinXP, nor using my "cstrings-patch", nor issuing your chcp command from within the console beforehand. (But I am strongly hesitating to open just another bug report for that, let's wait if some of those discussed patches will help.)

Hartmut <hardy>
Tue 25 Oct 2016 07:36:18 PM UTC, comment #10: 

Hartmut,

guessing from Philip's test results in bug #43099 (comments #32 and #33 there), it might be worth including that patch in a try build.

I also think that it is a good idea to use toLocal8Bit where toAscii was used before cset 2fd4f1a3f4a8 like you explained in bug #49118 (comment #25 there). Might that also solve the remaining differences between Windows 7 and Windows 10 in bug #43099?

Btw: Can you enter non-ASCII characters in the GUI console in Windows? Umlauts typed on my keyboard are completely ignored. Pasting them from the clipboard also does not work.

Markus Mützel <mmuetzel>
Group administrator
Tue 25 Oct 2016 06:51:45 PM UTC, comment #9: 

@Markus: I have just now discovered your recent comment #6, and had not started to compile, yet.

I have tested my mxe-octave binary with only the "cstrings-patch" applied (as before in comment #7) and issued the command system("chcp 65001") before doing the same tests. The result is:

  • Doing 'system("chcp 65001")' from inside the Octave GUI CONSOLE does NOT help. UTF-8 characters are still not displayed correctly afterwards.


I will attach another screenshot to visualize this result.

@Markus: Do you still consider it worth to try a new mxe-octave built with file #38784 applied? Could it act on a "deeper" level if this codepage was already used during the creation of the console (that's what I think file #38784 would do)?


Hartmut <hardy>
Tue 25 Oct 2016 06:19:44 PM UTC, comment #8: 

Hartmut,

thank you very much for testing.

Seems I was a little too late with my last post.

For the GUI CONSOLE: Can you try after executing

system("chcp 65001")


If this "fixes" the console, there is a good change file #38784 will fix it, too.

Markus Mützel <mmuetzel>
Group administrator
Tue 25 Oct 2016 06:15:11 PM UTC, comment #7: 

I have built a (only some days old) default-octave with mxe-octave (qt4) and used my "cstrings-patch" (from bug #49118 comment #26). The results on a virtual WinXP machine are:

  • Even with the "official" Octave-4.2.0-rc2 the GUI EDITOR seems FINE to me. It displays UTF-8 characters properly (if I have a corresponding font installed, chinese did not work therefore). But I do need to change the editor settings, such that the file encoding is UTF-8. The default value is "System" and does not display UTF-8 characters properly on Windows.


  • The Octave GUI CONSOLE does NOT display UTF-8 characters properly. Not with 4.2.0-rc2 and still not with this patch applied.


  • A user created dialog window (e.g. created with the INPUTDLG command) does not display UTF-8 characters with 4.2.0-rc2, but after applying this patch it does. So this is one aspect, where this "cstrings-patch"HELPS.


  • The VARIABLE EXPLORER does not display UTF-8 characters properly with Octave-4.2.0-rc2. But after applying this patch, it does. So this is another aspect, where this "cstrings-patch" HELPS.


I will attach two screenshots to visualize the results as well. Those are done with the demo script from bug #43099, with an additional inputdlg command.

So I would strongly vote to apply this "cstrings-patch" (from bug #49118 comment #26). Not only does it fully resolves bug #49118, it does also resolve half of this bug (bug #49386).

As a next step I will try to additionally apply my proposed "toLocal8Bit-patch" (explained in bug #49118 comment #25, but without a patch file yet) to mxe-octave. It did not help for bug #49118, but maybe it helps to finish off the remaining half of this bug (bug #49386)...



Hartmut <hardy>
Tue 25 Oct 2016 06:08:55 PM UTC, comment #6: 

Hartmut,

yes, you're right. My patch wouldn't work with qt5. Yours is definitely the one to chose.

  • For the test of the GUI command window with the demo script testing_utf8.m, you should also apply file #38784 from bug #43099.


  • If the results of testing_utf8.m display fine in the variable explorer, Russian characters should display fine the as well (this bug report). But testing this as well doesn't hurt, I guess...


  • For the second part of this bug report, make sure to set the editor to UTF-8 in the preferences if you use the editor.
Markus Mützel <mmuetzel>
Group administrator
Mon 24 Oct 2016 06:25:44 PM UTC, comment #5: 

I had a look at Markus patch  in comment #1. If I remember correctly the command setCodecForCStrings is obsolete in qt5. So this patch might break building mxe-octave with qt5.

My patch in bug #49118 comment #26 adds the very same code line to octave-gui.cc, but wraps it in "#if defined (HAVE_QT4)".

If I find the time I will take any of those setCodecForCStrings patches and cross-build mxe-octave with qt4. Am a right that the following should then be tested:

  • display utf-8 characters (like in the demo script testing_utf8.m) in the GUI command window (bug #43099).
  • create a string variable with a utf8 russian letter as content (this bug report) and have a look at it in the variable explorer.
  • create a uigetfile window with a utf8 russion letter in the title string (this bug report).
Hartmut <hardy>
Mon 24 Oct 2016 04:12:52 PM UTC, comment #4: 

The patches here and in bug #43099 are different. The one here (or better use Hartmut's patch in bug #49118) sets the codepage for C-strings to UTF-8. The one in bug #43099 sets the codepage for the Windows console in Octave to UTF-8.

I am hoping that setting the codepage for C-strings explicitly to UTF-8 in qt4 will also solve (at least some of) the issues in bug #43099 for qt4, too.
If you already tried this patch here (additionally to the patch in bug #49118), and it did not improve anything codepage related in Windows, there is no need to do it again with Hartmut's patch.

There are so many open bugs related to codepage issues, it is getting pretty confusing...

Markus Mützel <mmuetzel>
Group administrator
Mon 24 Oct 2016 03:01:20 PM UTC, comment #3: 

Sure I'll have a try with Hartmut's patch, somewhere this week.
However I do not have any files with Russian contents or Russian names to try. So I could only look at the GUI terminal.
I tried the patch in comment #1 (you also gave it in bug #43099 in comment #31 there) and I found it solves the terminal issues only on Win10, not on Win7.

Even with qt4 builds I had problems with cutting/copying/pasting using ctrl-X, ctrl-C and ctrl-V, in the editor and terminal panes.

BTW I'm getting somewhat lost in all the encoding/codepage related bug reports.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 24 Oct 2016 10:39:13 AM UTC, comment #2: 

Hartmut posted a very similar but better patch at bug #49118 (file #38798).
I am still unable to cross-build with qt4.
Adding Philip and Hartmut in CC.

@Philip: Did you already test this patch here (since you referred to it in bug #49348). Does it work? If not yet, could you please test a cross-build with Hartmut's patch in bug #49118 with qt4 and test whether this also solves the codepage related issues?
The motivation to ask for this is: Pasting (neither Crtl+V nor via right-click menu) in the editor pane does not work for me in a qt5 cross-build for Windows. (Is this only for me?) And maybe there are other bugs, too. So it might be better not to rush with shipping a qt5 cross-build before we have it tested a little bit more thoroughly...

Markus Mützel <mmuetzel>
Group administrator
Thu 20 Oct 2016 08:32:38 AM UTC, comment #1: 

This might be a dublicate of bug #43099 and is still present in current rc's and dev. This fixes itself when compiling with qt5.
If I understand [1] correctly, it might also be fixed for qt4 by the attached patch (might be a wild guess).
However, I cannot test at the moment. (I managed to break my cross-compilation environment for qt4.) Any help with testing would be greatly appreciated.

[1] https://wiki.qt.io/Strings_and_encodings_in_Qt

(file #38772)

Markus Mützel <mmuetzel>
Group administrator
Wed 19 Oct 2016 04:38:21 PM UTC, original submission:  

In the screenshot you see a one-byte character encoding "latin1" in GUI. It does not contain Cyrillic characters, it causes problems for some users.
There is no way to use Russian letters for the variables and the name of the dialog box.

It is known that are currently Octave uses one byte for char arrays. Is it possible to introduce an additional environment variable that controls the encoding in the GUI?


Igor Melentyev <igor_melentyev>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by hardy (Posted a comment)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by igor_melentyev (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-03-23 mmuetzel StatusNone Duplicate
        Open/ClosedOpen Closed
        Dependencies- Depends on bugs #43099
    2016-10-26 hardy Attached File- Added WinXP defaultoctave+cstring-patch + tolocal8bit-patch editor+dlg=nice, console=ugly.png, #38812
    2016-10-25 hardy Attached File- Added WinXP defaultoctave+cstring-patch + 65001-patch editor+dlg=nice, console=ugly.png, #38811
    2016-10-25 hardy Attached File- Added WinXP defaultoctave+cstring-patch + chcp-command editor+dlg=nice, console=ugly.png, #38810
    2016-10-25 hardy Attached File- Added WinXP 420rc2 utf8 editor=nice, console+dlg=ugly.png, #38808
        Attached File- Added WinXP defaultoctave+cstring patch edidtor+dlg=nice, console=ugly.png, #38809
    2016-10-24 mmuetzel Carbon-Copy- Added philipnienhuis
        Carbon-Copy- Added hardy
    2016-10-20 mmuetzel Attached File- Added bug49386_utf8_cstrings.patch, #38772
    2016-10-19 igor_melentyev Attached File- Added example.PNG, #38761

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code