bugGNU Octave - Bugs: bug #47571, Problem entering...

 
 

bug #47571: Problem entering non-ASCII-character (umlaut) in command line

Submitter:  None
Submitted:  Wed 30 Mar 2016 09:47:56 AM UTC
   
 
Category:  GUI Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Martin Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Microsoft Windows Fixed Release:  None
Planned Release:  8.1.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 14 Dec 2022 12:28:21 PM UTC, comment #49: 

This is working correctly now.

Closing as fixed.

I don't see how comment #48 is related to this report.

Markus Mützel <mmuetzel>
Group administrator
Fri 09 Dec 2022 04:29:37 PM UTC, comment #48: 

got it. seems there are so many of these separate utf / character related bugs since ~2014-15, i failed to notice the very obvious win7 tie in in the title.

is the difference in the :

>> 'Müller' == 'Möller'


outputs between octave and matlab a concern? (assuming it's not an artifact of how i entered them with the ALT-text vs an actual keyboard)

Nicholas Jankowski <nrjank>
Group Member
Fri 09 Dec 2022 07:21:30 AM UTC, comment #47: 

Re comment #45 (bug #43099): Afaict, the support for UTF-8 in the Windows Console of Windows versions prior to Windows 10 is still incomplete/broken. And since those versions are end-of-support by Microsoft, I don't think this will change.

Markus Mützel <mmuetzel>
Group administrator
Thu 08 Dec 2022 07:25:45 PM UTC, comment #46: 

updating comment #19 for Matlab 2022b (all entered with copy/paste)

>> tmp = 'Müller'
tmp =
    'Müller'
>> tmp+0
ans =
    77   252   108   108   101   114
>> 'Müller' == 'Möller'
ans =
  1×6 logical array
   1   0   1   1   1   1
>> char([195 188])
ans =
    'ü'
>> str = '囊囂囇囒' + 0
str =
       22218       22210       22215       22226
>>  str = 'ಠ_ಠ' + 0
str =
        3232          95        3232

(apparently my screen font doesn't display 'ಠ_ಠ', it just looks like <square>_<square>, but copy paste back into here retains the correct characters.

in Octave 8.0.1 (hg id: 24bd675bceab) with MXE Default, trying to use the ALT-codes when possible, copy/paste for others:


>> tmp = 'Müller'
tmp = Müller
>> tmp+0
ans =

    77   195   188   108   108   101   114

>> 'Müller' == 'Möller'
ans =

  1  1  0  1  1  1  1

>> char([195 188])
ans = ü
>> str = '¥¥¥¥¥¥¥¥' + 0
str =

 Columns 1 through 12:

   239   191   165   155   138   239   191   165   155   130   239   191

 Columns 13 through 20:

   165   155   135   239   191   165   155   146

>> str = '¢¢_¢¢' + 0
str =

   239   191   160   178   160    95   239   191   160   178   160


(those last two are copy/paste only as I don't actually know if they can be done through alt-codes)

Nicholas Jankowski <nrjank>
Group Member
Thu 08 Dec 2022 03:56:16 PM UTC, comment #45: 

a comment further down suggested this might be related to bug #43099.  would those fixes be expected to address that as well?

Nicholas Jankowski <nrjank>
Group Member
Thu 08 Dec 2022 01:51:59 PM UTC, comment #44: 

I didn't find this bug report at the time. But most of this should be fixed with the following changes in MXE Octave:
https://hg.octave.org/mxe-octave/rev/a76f513700df
https://hg.octave.org/mxe-octave/rev/54f336bbb833
https://hg.octave.org/mxe-octave/rev/a6f1111a9f1d

And the following change in Octave:
https://hg.savannah.gnu.org/hgweb/octave/rev/ea0b06534a37

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Mon 16 May 2022 12:05:41 AM UTC, comment #43: 

C++23 has a proposed solution to print Unicode correctly on Windows. https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2093r11.html#motivation

Anonymous
Sun 09 Jun 2019 01:59:51 PM UTC, comment #42: 

Now that we are using a newer version of readline, I thought it might be easier to enable entering 8bit characters in the command window.
The attached patch sets the necessary flags in our inputrc. With this change, entering 8bit characters like "ä", "Ö", "ß" or "§" with a German keyboard layout is possible.

Remaining issues include:
- Any 8bit character appears on screen only after an additional key stroke for a 7bit character.
- Entering two 8bit characters in a row breaks the command window: it is no longer possible to enter anything.
- Using the left arrow keys to reverse over an 8bit character messes up the cursor position. With the cursor left of an 8bit character, entered characters override characters on-screen. These characters appear at an unexpected position in the resulting string (after hitting enter).
- Pasting strings with 8bit characters into the command window doesn't work at all. After trying to paste a string including an 8bit character, the paste functionality seems to be broken for the remainder of the Octave session.

I am not sure whether these are issues with readline or with our implementation of the command window.

(file #47058)

Markus Mützel <mmuetzel>
Group administrator
Sat 25 Aug 2018 04:26:04 PM UTC, comment #41: 

Please, dis-regard the last patch. It doesn't work.
But file #44715 would still be nice to be applied.

Markus Mützel <mmuetzel>
Group administrator
Sat 25 Aug 2018 03:47:07 PM UTC, comment #40: 

The attached patch updates readline to 7.0. Currently re-building to check whether that actually changes anything.

(file #44862)

Markus Mützel <mmuetzel>
Group administrator
Mon 20 Aug 2018 06:57:51 PM UTC, comment #39: 

Now that Octave 4.4.1 is out, can we consider applying the changeset from comment #38 to MXE Octave? That would mitigate this long standing bug.

Markus Mützel <mmuetzel>
Group administrator
Sun 05 Aug 2018 02:36:13 PM UTC, comment #38: 

GNU Readline 6.3 already added the "meta"-logic for encodings using more than 7 bits per character. And the MXE project has patches for that version that helped getting the code to cross-compile. So I opted for upgrading to 6.3.

With the attached changeset for MXE Octave, I can enter any non-ASCII characters without setting a local .inputrc file.

It still takes an additional keystroke for the multi-byte characters to appear and using the left and right keys to navigate over a multi-byte character seems to mess up the cursor position.
But it is definitely an improvement to the current situation where entering or pasting these characters isn't possible at all.

Maybe some of these issues would be fixed by upgrading to Readline 7.0 (or maybe not). But I won't have time to try that out myself within the next few weeks.

(file #44715)

Markus Mützel <mmuetzel>
Group administrator
Sun 05 Aug 2018 11:00:24 AM UTC, comment #37: 

This seems to be a problem with GNU Readline's handling of 8-bit characters. With the default settings, "Readline will convert characters with the eighth bit set to an ASCII key sequence by stripping the eighth bit and prefixing an ESC character, converting them to a meta-prefixed key sequence." [1]

If I use the following setting in my .inputrc, entering umlaut characters on the command line works (with a delay of one keystroke):

set input-meta on
set output-meta on
set convert-meta off


We could (over-)write .inputrc with the installer. Or we could change the defaults of the Readline we compile with MXE Octave with the attached patch.

However, the doc for "input-meta" continues to read:
"The default value is ‘off’, but Readline will set it to ‘on’ if the locale contains eight-bit characters."
And accordingly for the other two settings.

That logic doesn't seem to be present in Readline 6.2 that we use. So maybe we should better upgrade to Readline 7.0 (and also set an UTF-8 locale in case we don't do so yet).

[1]: https://www.gnu.org/software/bash/manual/html_node/Readline-Init-File-Syntax.html

(file #44713)

Markus Mützel <mmuetzel>
Group administrator
Thu 12 Apr 2018 03:21:31 PM UTC, comment #36: 

@Felipe: Even if this bug and bug #49118 both handle problems with non-ASCII characters, the underlying issues are different.

The Command Window in the GUI on Windows is becoming less functional with each feature update for Windows 10. Up to a point where it is hardly usable because it can no longer be scrolled up right now (bug #52496).
As far as I can tell, Octave uses different approaches how the Command Window is implemented on Windows and Linux. Maybe it would be a good idea to completely rewrite that code to get it cross-platform.
This might be a nice GSOC project (if anyone would like to mentor).

Markus Mützel <mmuetzel>
Group administrator
Mon 09 Apr 2018 01:22:25 AM UTC, comment #35: 

Bug #49118 seems related to this one.

Felipe G. Nievinski <fgnievinski>
Thu 22 Mar 2018 10:42:55 PM UTC, comment #34: 

Just for the files: Because of comment #4 and bug #42036 this might have never worked. Therefore gets marked as "feature request" rather than a regression.

Kai Torben Ohlhus <siko1056>
Group Member
Wed 12 Apr 2017 03:14:51 PM UTC, comment #33: 

Thank you, John. I got Octave to compile with these flags. Sadly, it did not solve the problem here directly.

The same problem exists on the CLI. Can someone give me a hint where stdin handling is done in the source code (which file)?

Markus Mützel <mmuetzel>
Group administrator
Fri 07 Apr 2017 12:41:49 PM UTC, comment #32: 

configure at least passes with the change in comment #30, however I see warnings and fails now in ov-java.cc for passing in / expecting wide characters to Win32 api calls.

John Donoghue <lostbard>
Group Member
Fri 07 Apr 2017 12:38:58 PM UTC, comment #31: 

Something like this may work:


diff -r 988745446d56 src/default-octave.mk
--- a/src/default-octave.mk     Mon Apr 03 07:52:35 2017 -0400
+++ b/src/default-octave.mk     Fri Apr 07 08:32:08 2017 -0400
@@ -115,7 +115,10 @@
 else
   $(PKG)_PREFIX := '$(HOST_PREFIX)'
   $(PKG)_EXTRA_CONFIGURE_OPTIONS := \
-    LDFLAGS='-Wl,-rpath-link,$(HOST_LIBDIR) -L$(HOST_LIBDIR) -L$($(PKG)_QTDIR)/lib'
+    LDFLAGS='-Wl,-rpath-link,$(HOST_LIBDIR) -L$(HOST_LIBDIR) -L$($(PKG)_QTDIR)/lib' \
+    XTRA_CXXFLAGS='-DUNICODE -D_UNICODE' \
+    XTRA_CFLAGS='-DUNICODE -D_UNICODE' \
+    XTRA_CRUFT_SH_LDFLAGS='-municode'
 endif



However, I believe a extern "C" wmain() function is also then required.


In addition, there is also a high probability that a number of function calls (maybe only windows API calls) that were sending in strings, or constants will then also need to change to put in thier wide character equivalents


John Donoghue <lostbard>
Group Member
Tue 04 Apr 2017 08:49:29 PM UTC, comment #30: 

If I understand correctly, Windows converts keystrokes and the paste bin fro the console to "ANSI" unless you compile with UNICODE and _UNICODE defined and link with -municode. [1]

I tried to do that by changing default-octave.mk in MXE Octave:

diff -r ce6c155a45ba src/default-octave.mk
--- a/src/default-octave.mk        Wed Mar 29 07:54:36 2017 -0400
+++ b/src/default-octave.mk        Tue Apr 04 22:15:01 2017 +0200
@@ -115,7 +115,9 @@
 else
   $(PKG)_PREFIX := '$(HOST_PREFIX)'
   $(PKG)_EXTRA_CONFIGURE_OPTIONS := \
-    LDFLAGS='-Wl,-rpath-link,$(HOST_LIBDIR) -L$(HOST_LIBDIR) -L$($(PKG)_QTDIR)/lib'
+    LDFLAGS='-Wl,-rpath-link,$(HOST_LIBDIR) -L$(HOST_LIBDIR) -L$($(PKG)_QTDIR)/lib -municode' \
+    CFLAGS='-DUNICODE -D_UNICODE' \
+    CXXFLAGS='-DUNICODE -D_UNICODE'
 endif

 ifeq ($(MXE_SYSTEM),mingw)


However, trying to build with that change fails at configure with the following error:

configure:6593: x86_64-w64-mingw32-gcc -DUNICODE -D_UNICODE -I/path_to_mxe_octave/usr/x86_64-w64-mingw32/include -Wl,-rpath-link,/path_to_mxe_octave/usr/x86_64-w64-mingw32/lib -L/path_to_mxe_octave/usr/x86_64-w64-mingw32/lib -L/path_to_mxe_octave/usr/x86_64-w64-mingw32/lib -municode conftest.c  >&5
/path_to_mxe_octave/usr/x86_64-w64-mingw32/lib/libmingw32.a(lib64_libmingw32_a-crt0_w.o): In function `wmain':
/path_to_mxe_octave/tmp-build-gcc/gcc-4.9.4/mingw-w64-v4.0.6/mingw-w64-crt/crt/crt0_w.c:23: undefined reference to `wWinMain'
collect2: error: ld returned 1 exit status
configure:6597: $? = 1
configure:6635: result: no
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "GNU Octave"
| #define PACKAGE_TARNAME "octave"
| #define PACKAGE_VERSION "4.3.0+"
| #define PACKAGE_STRING "GNU Octave 4.3.0+"
| #define PACKAGE_BUGREPORT "http://octave.org/bugs.html"
| #define PACKAGE_URL "http://www.gnu.org/software/octave/"
| #define PACKAGE "octave"
| #define VERSION "4.3.0+"
| #define OCTAVE_SOURCE 1
| /* end confdefs.h.  */
|
| int
| main ()
| {
|
|   ;
|   return 0;
| }
configure:6640: error: in `/path_to_mxe_octave/tmp-default-octave/octave-4.3.0+/.build':
configure:6642: error: C compiler cannot create executables


This is probably because with that switches the expected entry point is "wmain()".

I might have a patch for "main.in.cc", but cannot test (since configure fails).
Is there a way to pass linker switches only for the step when the octave executable is finally linked (but not during configure)? Where would I have to add those flags?

Adding John D to CC because he seems to know his way around in MXE Octave.

[1] https://sourceforge.net/p/mingw-w64/wiki2/Unicode%20apps/

Markus Mützel <mmuetzel>
Group administrator
Thu 23 Mar 2017 07:30:17 PM UTC, comment #29: 

Right. Those two characters you have mentioned are plain ASCII

"^" = 0x5E
"`" = 0x60

All others are not.

Armin Müller <arm_in>
Thu 23 Mar 2017 12:41:12 PM UTC, comment #28: 

Re-titling to better distinguish the scope of this bug from bug #43099 and changing the affected release to "dev".

I can reproduce the issue here in Windows 10 with Octave 4.2.1 and a fairly recent dev. It is not possible to enter any of the following characters (with a German keyboard layout) in Octave's command window: "ÄÖÜäöüßÂÊÎÔÛâêîôûÁÉÍÓÚáéíóúÀÈÌÒÙàèìòù´°§µ"
They are just ignored. It is however possible to enter these related accent characters: "^`"

Pasting the former string in the command window leads to nothing being pasted.

Entering any of these characters in the GUI editor works fine. All characters are also displayed correctly when I run a file with the following content:

a = "ÄÖÜäöüßÂÊÎÔÛâêîôûÁÉÍÓÚáéíóúÀÈÌÒÙàèìòù´°§µ"


Markus Mützel <mmuetzel>
Group administrator
Fri 01 Jul 2016 09:56:50 AM UTC, comment #27: 

This may be duplicated with
https://savannah.gnu.org/bugs/?43099

Tatsuro MATSUOKA <tmacchant>
Wed 29 Jun 2016 10:33:09 AM UTC, comment #26: 

Will compiling the existing source in a UTF-8 environment will make this work for 4.2.0?

Lachlan Andrew <lachlan>
Thu 21 Apr 2016 02:41:06 PM UTC, comment #25: 

From experimenting, the best way to save everything as UTF-8 on Windows is to ftp it as binary. Download Octave source to a network location that does UTF-8 natively (I used "hg clone" from an RHEL server), then use an ftp program from the Windows computer to copy the whole directory tree in binary mode. Do not compress to an archive and then re-expand in Windows; that munges line endings to "\r\n" and changes UTF settings.


Anonymous
Sat 09 Apr 2016 12:34:42 AM UTC, comment #24: 

The problem with the display is restricted to the GUI then; the UTF-8 chars seem to render properly in CLI mode if the command window is configured to accept UTF-8.

For the Qt GUI, I found these two pieces of information:

1. Set the locate to UTF-8, (re)save the entire Octave codebase as UTF-8 on the user's machine, and compile everything from source. Qt should pick up locale info and make the GUI work with UTF-8. [1] Does this make sense to someone with Qt experience? If this is the case, then entering umlauted text will also be solved by the GUI without any external hackery. This can be done by the end user without changing the code base, but can also be an annoying extra hoop for the end user to jump.

2. There is a Qt function called setCodecForCStrings, and it can be passed "UTF-8" as an argument. [2] Is this something that the Octave GUI code should have in a header file to "force" UTF-8 interpretation across the GUI? If this alternative works, it could make life easier for Windows users and precompiled binaries will also be UTF-8 aware.

Could someone with experience in Qt please comment here?

[1]: https://wiki.qt.io/Strings_and_encodings_in_Qt
[2]: http://doc.qt.io/qt-4.8/qtextcodec.html#setCodecForCStrings


The problem with entering an umlaut on the GUI could be potentially solved with [1] above; entering it on the CLI is still an unsolved problem. Let me read up more.

Anonymous
Fri 08 Apr 2016 06:31:42 AM UTC, comment #23: 

to comment #21:
In notepad++ and the Octave GUI editor the file is shown correctly. Notepad++ also shows that it is coded in UTF-8.

In cmd.exe with the default code page 850, the file is shown wrong:
C:\Users\lorenz.m\octave>type testumlaut.m
string='├╝'
type testumlaut.m

C:\Users\lorenz.m\octave>chcp
Aktive Codepage: 850.

C:\Users\lorenz.m\octave>chcp 65001
Aktive Codepage: 65001.

C:\Users\lorenz.m\octave>type testumlaut.m
string='ü'
type testumlaut.m

As you can see, with codepage 65001, the file is displayed correctly. With the same codepage I start ovtave-cli.exe:

octave:1> testumlaut
string = ü
testumlaut.m is the user-defined function defined from: C:\Users\lorenz.m\octave\testumlaut.m

string='ü'
type testumlaut.m

octave:2> Mller

Now the file is is displayed correctly in Octave! But I can still not enter the word Müller. The ü is omitted.

Martl <gsassle>
Fri 08 Apr 2016 01:52:59 AM UTC, comment #22: 

To comment #21: when you view the file in Notepad, or type it in cmd.exe, do you get the Umlaut? If you do, then can you try "octave --no-gui" from that cmd.exe window? In that Octave session, try running the file again. Hopefully that cmd.exe window will know how to handle the UTF-8 chars coming from Octave and render them properly.

If that works, then the problem is localized to Octave's GUI. If it doesn't work, but cmd.exe works with umlauts, then we need to somehow tell cmd.exe to expect UTF-8 from Octave.

Anonymous
Thu 07 Apr 2016 06:38:30 AM UTC, comment #21: 

To comment #20:

executing the test script yields:

>> testumlaut

string = ├╝
testumlaut.m is the user-defined function defined from: C:\Users\lorenz.m\octave\testumlaut.m
string='├╝'
type testumlaut.m

It does not print correctly, the two UTF-8 Bytes are displayed incorrectly.

To comment #17:
I can observe this 'delete rest of word' behaviour only in octave in Windows.

Martl <gsassle>
Tue 05 Apr 2016 04:18:48 AM UTC, comment #20: 

Make a file foo.m in a Unicode-aware text editor with two lines:

str = "ü"
type foo.m

and execute it. If it prints, then the problem is how Octave interacts with keystrokes. If it didn't print, then the problem is in Octave's display code, or its mismatch with UTF-16.

Anonymous
Tue 05 Apr 2016 01:56:25 AM UTC, comment #19: 

FWIW, results from Matlab on Windows from replicating previous commenters:


>> tmp = 'Müller'
tmp =
Müller
>> tmp + 0
ans =
    77   252   108   108   101   114
>> 'Müller' == 'Möller'
ans =
     1     0     1     1     1     1
>> char([195 188])
ans =
ü
>> str = '囊囂囇囒' + 0
str =
    26    26    26    26
>> str = 'ಠ_ಠ' + 0
str =
    26    95    26
>>
>>


Maps ü to 252, but fails for non-Latin characters like CJK. Can enter ü from keyboard and copy/paste, but cannot type 'ಠ_ಠ' (Kannada block) or '囊囂囇囒' (CJK). Everything is converted to character 26.

Anonymous
Mon 04 Apr 2016 07:28:54 PM UTC, comment #18: 

On GNU/Linux systems, yes UTF-8 works because Octave treats all strings as byte sequences. If a certain sequence of bytes makes up a multibyte character, that's left to the terminal to render, Octave doesn't care.

I can open a rxvt terminal (non-Unicode aware) and a urxvt (Unicode-aware) and run octave in them side-by-side and see different results for


>> char ([195, 188])


It appears that some users are able to enter UTF-8 characters in Octave on Windows and some are not.

Mike Miller <mtmiller>
Group Member
Mon 04 Apr 2016 07:05:23 PM UTC, comment #17: 

The lowercase ü is correctly interpreted as bytes 195 and 188 in that sequence, according to UTF-8. To verify: bitand(195,31) * 64 + bitand(188,63) == 252 == 0xFC == the Unicode character for ü. The UTF-8 stream for Latin-1 is of the form 110xxxxx 10yyyyyy. Discard the 110 and 10 to get xxxxxyyyyyy. Longer byte sequences do similar calculations. This means Octave is internally and implicitly doing UTF-8 normalization even on Windows -- JWE please correct if wrong.

Not able to reproduce the "delete rest of word" action. Continuing. Probably copy-pasting UTF-16 on Windows is being misinterpreted by Octave as UTF-8, and excess chars are being executed as word processing commands. Need to test more.

Martl, do you get "delete rest of word" only on Octave or on, say, Notepad and other applications as well?


Anonymous
Mon 04 Apr 2016 03:19:39 PM UTC, comment #16: 

To comment #14:

After entering or copy&Pasting length('Müller') I get

>> length('Mller')

ans =  5

I also observed that an 'ä' is interpreted as deletion of the rest of a word. For example, if I enter 12345 67890 in the CLI, move the cursor between 2 and 3 and press 'ä' then 345 disappears.

It seems that the umlauts are misinterpreted or not entered at all in the commend buffer of Octave in Windows. So where are they going to and where are they filtered out or misinterpreted?


Martl <gsassle>
Sat 02 Apr 2016 02:26:46 PM UTC, comment #15: 

In comment #14, problem 1 is not a problem. It is the nature of UTF-8. Using UTF-8 APIs and algorithms in place of naive char* processing will solve that. Most inner loops on char*, like copying, do not even change with char width.

Anonymous
Sat 02 Apr 2016 01:07:43 PM UTC, comment #14: 

There seem to be two different issues.

One is how Octave represents multi-byte chars internally, where Latin1 chars take up to 2 bytes and Chinese chars take up to 3 bytes. This has ramifications for string comparisons, strfind, strrep, etc. It looks bizarre at first but having variable width characters is common in most Unicode / localization efforts. The hard part is preventing false positives. For example, umlauted characters have the leading byte 195, so searching for character 195 returns true even if character 195 is different.

The other is how to enter extended chars at the command line and how that varies across different platforms. It looks like most Windows installations of Octave cannot accept umlauts even if Windows itself can. Is there code in the Octave interpreter that rejects characters outside a certain range even if they're inside string delimiters? If not, where are the umlauted chars going? Are they just not being displayed or are they not entered in the command buffer at all?

Anonymous
Sat 02 Apr 2016 08:46:31 AM UTC, comment #13: 

Last bit should read:
Copy-pasting it (= Müller) from e.g.,notepad++ to Octave the ü is lost)

Philip Nienhuis <philipnienhuis>
Group Member
Sat 02 Apr 2016 08:45:14 AM UTC, comment #12: 

Where you write:

>> tmp+0
ans =
    77   195   188   108   108   101   114
>> "Müller" == "Möller"
ans =
   1   1   0   1   1   1   1


you can see that Octave accepts a mix of single- and double byte strings. When the ö and ü are compared, only the second part of the double byte char is found to be differing. That looks a bit slippery to me.

What does
uint8 (tmp)
or
uint16 (tmp)
return for the first statement? (can't try myself as umlaut here on my US intl keyboard. Copying it from e.g.,notepad++ the ü is lost)

Philip Nienhuis <philipnienhuis>
Group Member
Fri 01 Apr 2016 11:06:12 PM UTC, comment #11: 

Looks OK...?


>> tmp = 'Müller'
tmp = Müller
>> whos
Variables in the current scope:

   Attr Name        Size                     Bytes  Class
   ==== ====        ====                     =====  =====
        tmp         1x7                          7  char

Total is 7 elements using 7 bytes

>> tmp+0
ans =
    77   195   188   108   108   101   114
>> "Müller" == "Möller"
ans =
   1   1   0   1   1   1   1
>> tmp = repmat(tmp, 1, 32)
tmp = MüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüllerMüller
>> strrep(tmp, "Müller", "Möller")
ans = MöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöller
>> strrep(tmp, "ü", "ö")
ans = MöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöllerMöller
>>


Anonymous
Fri 01 Apr 2016 07:38:30 AM UTC, comment #10: 

update to comment #9:

Now I tried directly in the commandline of the GUI of octave 4.0.0:

>> [Status, Text]=dos("chcp 65001")

Status = 0
Text = Aktive Codepage: 65001.

But I still cannot enter umlauts and also cut&paste of umlauts does not work.

Martl <gsassle>
Fri 01 Apr 2016 07:15:53 AM UTC, comment #9: 

The Tip of comment #8 does not work for me:
I started cmd.exe, changed the code page from 850 to 65001, and I can enter umlauts (but this worked also with default 850). Then I startet octave-cli in cmd, but I still cannot enter umlauts.

Concerning comment #1: Another strange thing is, that after downgrading from 4.0.1 to 4.0.0, now 4.0.0 does not understand umlauts either! But the old installation of 4.0.0 from last July did understand umlauts without any special configuration, hacks etc. Of course, there have been plenty windows updates in the meantime...

Martl <gsassle>
Thu 31 Mar 2016 05:49:45 PM UTC, comment #8: 

Here is one way to get umlauts in Windows. This works on Windows 7 Professional. Not tested on other versions.

1. Start cmd (Windows command line interface).

2. Type alt-space, then click Properties, go to the Fonts tab, and pick something Unicode-aware like Consolas. Do not leave it as a bitmap font.

3. Type chcp at the prompt and enter. It will probably say 437. Now type "chcp 65001" and enter. That will set it to UTF-8.

4. Now you'll be able to run some command like "echo Müller". Copy-paste Müller from somewhere else, or if you have a German keyboard you can type it directly. Now if you run Octave in CLI mode from the same cmd window, you should be able to see and input UTF-8 characters.

The command "chcp" changes the code page -- Windows by default uses code page 437 which is the original IBM PC character set from the 1980s, but it also treats all of UTF-8 as yet another code page (!!!) numbered 65001.

This hack probably needs much more testing with other languages and scripts, not just German. But if you use it often enough, you can probably write a batch file to chcp 65001 before running Octave.

Anonymous
Thu 31 Mar 2016 03:59:02 PM UTC, comment #7: 

Does Option 1 solve the problem for someone on Windows? I have no idea how to configure that, but if you or someone can confirm that inputting UTF-8 into Octave allows non-ASCII characters to be entered, then we can say we have the same experience on Windows as on GNU/Linux systems.

Mike Miller <mtmiller>
Group Member
Thu 31 Mar 2016 03:22:17 PM UTC, comment #6: 

Situation: wchar_t and std::wstring are almost never needed in Linux & Mac, where UTF-8 is used. Using wchar_t and std::wstring are common in Windows, which uses UTF-16.

Option 1: keep Octave as it is, and configure the OS to convert UTF encodings as needed. Can be painful for the user if they use legacy software that needs Windows-1252 encoding. Otherwise, converting everything to UTF-8 will help, as described here: http://www.utf8everywhere.org/

Option 2: make extensive use of typedefs or (worse) #IFDEF statements where needed to support cross-platform UTF encoding differences. Very complicated and not practical.

Anonymous
Thu 31 Mar 2016 02:47:49 PM UTC, comment #5: 

Character strings in Octave are simple std::string and char* objects.  They won't properly handle wide character sets, or necessarily even be 8-bit clean.  As far as I know, there is no simple fix for this problem.

John W. Eaton <jwe>
Group administrator
Thu 31 Mar 2016 02:24:28 PM UTC, comment #4: 

See Ticket #42036

"4) When trying to enter "cd 'München'" in the command window by hand, the Umlaut "ü" will be replaced by a verical bar '|'.

[...]

5) Getting the Umlaut via copy & paste from an external ANSI editor is not pssible, either."

Armin Müller <arm_in>
Thu 31 Mar 2016 09:21:24 AM UTC, comment #3: 

I can confirm the German umlaut problem in the command line (GUI), running Octave 4.0.1 under Windows 7 (64 bit).

Anonymous
Wed 30 Mar 2016 11:26:16 PM UTC, comment #2: 


Here is more data, as this isn't just an umlaut problem:

Latin, extended Latin, Greek, Cyrillic: print properly on GUI, can be entered properly via GUI, print OK on CLI with some cramped accents occasionally, not reliable entering them from keyboard on CLI.

CJK, Indic and Thai: print properly on GUI, some rare difficulties in entering via GUI (combination chars), print very badly on CLI (no combinations at all), cannot be entered without mangling badly on CLI.

Armenian, Georgian, Arabic, Hebrew, Farsi, Ethiopic scripts: cannot evaluate due to unfamiliarity.

Textread and fprintf work fine for a Unicode sample file with multiple scripts listed above.

Strfind, strrep, and textscan are OK for Latin, extended Latin, Greek, Cyrillic. Not OK for Thai, Indic, (fail on combination characters). CJK looks OK but needs more testing.


All the above on Debian Jessie, standard terminal for CLI, Qt for Octave GUI.

Anonymous
Wed 30 Mar 2016 09:56:02 PM UTC, comment #1: 

Using a German Win 7 and keyboard, I cannot enter umlauts in neither v4.0.0 nor v4.0.1 (both CLI and GUI) command line. I also tried "ALT+142" and so on. In my regular Win CMD.exe it works. As described before, the editor can handle UTF-8 characters.

If you reinstall v.4.0.0, can you still type umlauts? Did you do any special configuration (e.g. usage of certain console fonts, installed some Windows updates?)

Kai Torben Ohlhus <siko1056>
Group Member
Wed 30 Mar 2016 09:47:56 AM UTC, original submission:  

I use Octave 4.0.1 in Windows with a german Keyboard.

In new Version 4.0.1, I cannot enter Umlauts like ö,ä, ß, or µ in the command line (Befehlsfenster). In the editor, this is possible.

In the old V4.0.0, entering umlauts was no problem.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #47058:  bug47571_rl_8bit.patch added by mmuetzel (1KiB - application/octet-stream)
file #44862:  bug47571_readline7.0.patch added by mmuetzel (15KiB - application/octet-stream)
file #44715:  bug47571_readline6.3.patch added by mmuetzel (12KiB - application/octet-stream)
file #44713:  bug47571_readline_meta.patch added by mmuetzel (2KiB - application/octet-stream)

 

Carbon-Copy List
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by marczellm
  • -email is unavailable- added by fgnievinski (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by tiagoma
  • -email is unavailable- added by tmacchant (Posted a comment)
  • -email is unavailable- added by lachlan (Posted a comment)
  • -email is unavailable- added by gsassle (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by arm_in (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 25 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-02-05 mmuetzel Dependencies- bugs #63717 is dependent
    2022-12-14 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2022-12-08 mmuetzel StatusPatch Submitted Ready For Test
        Planned ReleaseNone 8.1.0
    2022-06-03 mmuetzel Dependencies- bugs #62573 is dependent
    2022-05-28 ineiev Carbon-CopyRemoved 0 -
    2021-08-04 mmuetzel Dependencies- bugs #61006 is dependent
    2019-09-18 mmuetzel Dependencies- bugs #56789 is dependent
    2019-06-09 mtmiller Carbon-CopyRemoved 80942 -
    2019-06-09 mmuetzel Attached File- Added bug47571_rl_8bit.patch, #47058
        Summary[MXE-Octave] Problem entering non-ASCII-character (umlaut) in command line Problem entering non-ASCII-character (umlaut) in command line
    2018-08-25 mmuetzel Attached File- Added bug47571_readline7.0.patch, #44862
    2018-08-05 mmuetzel Attached File- Added bug47571_readline6.3.patch, #44715
        StatusNeed Info Patch Submitted
        SummaryProblem entering non-ASCII-character (umlaut) in command line [MXE-Octave] Problem entering non-ASCII-character (umlaut) in command line
    2018-08-05 mmuetzel Attached File- Added bug47571_readline_meta.patch, #44713
    2018-07-14 marczellm Carbon-Copy- Added marczellm
    2018-03-22 siko1056 Item GroupRegression Feature Request
    2017-04-12 mmuetzel Dependencies- bugs #50784 is dependent
    2017-04-12 mmuetzel SummaryProblem entering umlaut in command line Problem entering non-ASCII-character (umlaut) in command line
    2017-04-04 mmuetzel Carbon-Copy- Added lostbard
    2017-04-04 mmuetzel Dependencies- bugs #50284 is dependent
    2017-03-23 mmuetzel Release4.0.1 dev
        SummaryProblem with umlaut in command line Problem entering umlaut in command line

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code