bugGNU Octave - Bugs: bug #47571, Problem entering...

 
 

bug #47571: Problem entering non-ASCII-character (umlaut) in command line

Submitted by:  None
Submitted on:  Wed 30 Mar 2016 09:47:56 AM UTC  
 
Category: GUISeverity: 3 - Normal
Priority: 5 - NormalItem Group: Regression
Status: Need InfoAssigned to: None
Originator Name: MartinOriginator Email: -unavailable-
Open/Closed: OpenRelease: dev
Operating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 12 Apr 2017 03:14:51 PM UTC, comment #33:

Thank you, John. I got Octave to compile with these flags. Sadly, it did not solve the problem here directly.

The same problem exists on the CLI. Can someone give me a hint where stdin handling is done in the source code (which file)?

Markus Mützel <mmuetzel>
Project Member
Fri 07 Apr 2017 12:41:49 PM UTC, comment #32:

configure at least passes with the change in comment #30, however I see warnings and fails now in ov-java.cc for passing in / expecting wide characters to Win32 api calls.

John Donoghue <lostbard>
Project Member
Fri 07 Apr 2017 12:38:58 PM UTC, comment #31:

Something like this may work:

However, I believe a extern "C" wmain() function is also then required.

In addition, there is also a high probability that a number of function calls (maybe only windows API calls) that were sending in strings, or constants will then also need to change to put in thier wide character equivalents

John Donoghue <lostbard>
Project Member
Tue 04 Apr 2017 08:49:29 PM UTC, comment #30:

If I understand correctly, Windows converts keystrokes and the paste bin fro the console to "ANSI" unless you compile with UNICODE and _UNICODE defined and link with -municode. [1]

I tried to do that by changing default-octave.mk in MXE Octave:

However, trying to build with that change fails at configure with the following error:

This is probably because with that switches the expected entry point is "wmain()".

I might have a patch for "main.in.cc", but cannot test (since configure fails).
Is there a way to pass linker switches only for the step when the octave executable is finally linked (but not during configure)? Where would I have to add those flags?

Adding John D to CC because he seems to know his way around in MXE Octave.

[1] https://sourceforge.net/p/mingw-w64/wiki2/Unicode%20apps/

Markus Mützel <mmuetzel>
Project Member
Thu 23 Mar 2017 07:30:17 PM UTC, comment #29:

Right. Those two characters you have mentioned are plain ASCII

"^" = 0x5E
"`" = 0x60

All others are not.

Armin Müller <arm_in>
Thu 23 Mar 2017 12:41:12 PM UTC, comment #28:

Re-titling to better distinguish the scope of this bug from bug #43099 and changing the affected release to "dev".

I can reproduce the issue here in Windows 10 with Octave 4.2.1 and a fairly recent dev. It is not possible to enter any of the following characters (with a German keyboard layout) in Octave's command window: "ÄÖÜäöüßÂÊÎÔÛâêîôûÁÉÍÓÚáéíóúÀÈÌÒÙàèìòù´°§µ"
They are just ignored. It is however possible to enter these related accent characters: "^`"

Pasting the former string in the command window leads to nothing being pasted.

Entering any of these characters in the GUI editor works fine. All characters are also displayed correctly when I run a file with the following content:

Markus Mützel <mmuetzel>
Project Member
Fri 01 Jul 2016 09:56:50 AM UTC, comment #27:

This may be duplicated with
https://savannah.gnu.org/bugs/?43099

Tatsuro MATSUOKA <tmacchant>
Wed 29 Jun 2016 10:33:09 AM UTC, comment #26:

Will compiling the existing source in a UTF-8 environment will make this work for 4.2.0?

Lachlan Andrew <lachlan>
Project Member
Thu 21 Apr 2016 02:41:06 PM UTC, comment #25:

From experimenting, the best way to save everything as UTF-8 on Windows is to ftp it as binary. Download Octave source to a network location that does UTF-8 natively (I used "hg clone" from an RHEL server), then use an ftp program from the Windows computer to copy the whole directory tree in binary mode. Do not compress to an archive and then re-expand in Windows; that munges line endings to "\r\n" and changes UTF settings.

Anonymous
Sat 09 Apr 2016 12:34:42 AM UTC, comment #24:

The problem with the display is restricted to the GUI then; the UTF-8 chars seem to render properly in CLI mode if the command window is configured to accept UTF-8.

For the Qt GUI, I found these two pieces of information:

1. Set the locate to UTF-8, (re)save the entire Octave codebase as UTF-8 on the user's machine, and compile everything from source. Qt should pick up locale info and make the GUI work with UTF-8. [1] Does this make sense to someone with Qt experience? If this is the case, then entering umlauted text will also be solved by the GUI without any external hackery. This can be done by the end user without changing the code base, but can also be an annoying extra hoop for the end user to jump.

2. There is a Qt function called setCodecForCStrings, and it can be passed "UTF-8" as an argument. [2] Is this something that the Octave GUI code should have in a header file to "force" UTF-8 interpretation across the GUI? If this alternative works, it could make life easier for Windows users and precompiled binaries will also be UTF-8 aware.

Could someone with experience in Qt please comment here?

[1]: https://wiki.qt.io/Strings_and_encodings_in_Qt
[2]: http://doc.qt.io/qt-4.8/qtextcodec.html#setCodecForCStrings

The problem with entering an umlaut on the GUI could be potentially solved with [1] above; entering it on the CLI is still an unsolved problem. Let me read up more.

Anonymous
Fri 08 Apr 2016 06:31:42 AM UTC, comment #23:

to comment #21:
In notepad++ and the Octave GUI editor the file is shown correctly. Notepad++ also shows that it is coded in UTF-8.

In cmd.exe with the default code page 850, the file is shown wrong:
C:\Users\lorenz.m\octave>type testumlaut.m
string='├╝'
type testumlaut.m

C:\Users\lorenz.m\octave>chcp
Aktive Codepage: 850.

C:\Users\lorenz.m\octave>chcp 65001
Aktive Codepage: 65001.

C:\Users\lorenz.m\octave>type testumlaut.m
string='ü'
type testumlaut.m

As you can see, with codepage 65001, the file is displayed correctly. With the same codepage I start ovtave-cli.exe:

octave:1> testumlaut
string = ü
testumlaut.m is the user-defined function defined from: C:\Users\lorenz.m\octave\testumlaut.m

string='ü'
type testumlaut.m

octave:2> Mller

Now the file is is displayed correctly in Octave! But I can still not enter the word Müller. The ü is omitted.

Martl <gsassle>
Fri 08 Apr 2016 01:52:59 AM UTC, comment #22:

To comment #21: when you view the file in Notepad, or type it in cmd.exe, do you get the Umlaut? If you do, then can you try "octave --no-gui" from that cmd.exe window? In that Octave session, try running the file again. Hopefully that cmd.exe window will know how to handle the UTF-8 chars coming from Octave and render them properly.

If that works, then the problem is localized to Octave's GUI. If it doesn't work, but cmd.exe works with umlauts, then we need to somehow tell cmd.exe to expect UTF-8 from Octave.

Anonymous
Thu 07 Apr 2016 06:38:30 AM UTC, comment #21:

To comment #20:

executing the test script yields:

>> testumlaut

string = ├╝
testumlaut.m is the user-defined function defined from: C:\Users\lorenz.m\octave\testumlaut.m
string='├╝'
type testumlaut.m

It does not print correctly, the two UTF-8 Bytes are displayed incorrectly.

To comment #17:
I can observe this 'delete rest of word' behaviour only in octave in Windows.

Martl <gsassle>
Tue 05 Apr 2016 04:18:48 AM UTC, comment #20:

Make a file foo.m in a Unicode-aware text editor with two lines:

str = "ü"
type foo.m

and execute it. If it prints, then the problem is how Octave interacts with keystrokes. If it didn't print, then the problem is in Octave's display code, or its mismatch with UTF-16.

Anonymous
Tue 05 Apr 2016 01:56:25 AM UTC, comment #19:

FWIW, results from Matlab on Windows from replicating previous commenters:

Maps ü to 252, but fails for non-Latin characters like CJK. Can enter ü from keyboard and copy/paste, but cannot type 'ಠ_ಠ' (Kannada block) or '囊囂囇囒' (CJK). Everything is converted to character 26.

Anonymous
Mon 04 Apr 2016 07:28:54 PM UTC, comment #18:

On GNU/Linux systems, yes UTF-8 works because Octave treats all strings as byte sequences. If a certain sequence of bytes makes up a multibyte character, that's left to the terminal to render, Octave doesn't care.

I can open a rxvt terminal (non-Unicode aware) and a urxvt (Unicode-aware) and run octave in them side-by-side and see different results for

It appears that some users are able to enter UTF-8 characters in Octave on Windows and some are not.

Mike Miller <mtmiller>
Project Administrator
Mon 04 Apr 2016 07:05:23 PM UTC, comment #17:

The lowercase ü is correctly interpreted as bytes 195 and 188 in that sequence, according to UTF-8. To verify: bitand(195,31) * 64 + bitand(188,63) == 252 == 0xFC == the Unicode character for ü. The UTF-8 stream for Latin-1 is of the form 110xxxxx 10yyyyyy. Discard the 110 and 10 to get xxxxxyyyyyy. Longer byte sequences do similar calculations. This means Octave is internally and implicitly doing UTF-8 normalization even on Windows -- JWE please correct if wrong.

Not able to reproduce the "delete rest of word" action. Continuing. Probably copy-pasting UTF-16 on Windows is being misinterpreted by Octave as UTF-8, and excess chars are being executed as word processing commands. Need to test more.

Martl, do you get "delete rest of word" only on Octave or on, say, Notepad and other applications as well?

Anonymous
Mon 04 Apr 2016 03:19:39 PM UTC, comment #16:

To comment #14:

After entering or copy&Pasting length('Müller') I get

>> length('Mller')

ans = 5

I also observed that an 'ä' is interpreted as deletion of the rest of a word. For example, if I enter 12345 67890 in the CLI, move the cursor between 2 and 3 and press 'ä' then 345 disappears.

It seems that the umlauts are misinterpreted or not entered at all in the commend buffer of Octave in Windows. So where are they going to and where are they filtered out or misinterpreted?

Martl <gsassle>
Sat 02 Apr 2016 02:26:46 PM UTC, comment #15:

In comment #14, problem 1 is not a problem. It is the nature of UTF-8. Using UTF-8 APIs and algorithms in place of naive char* processing will solve that. Most inner loops on char*, like copying, do not even change with char width.

Anonymous
Sat 02 Apr 2016 01:07:43 PM UTC, comment #14:

There seem to be two different issues.

One is how Octave represents multi-byte chars internally, where Latin1 chars take up to 2 bytes and Chinese chars take up to 3 bytes. This has ramifications for string comparisons, strfind, strrep, etc. It looks bizarre at first but having variable width characters is common in most Unicode / localization efforts. The hard part is preventing false positives. For example, umlauted characters have the leading byte 195, so searching for character 195 returns true even if character 195 is different.

The other is how to enter extended chars at the command line and how that varies across different platforms. It looks like most Windows installations of Octave cannot accept umlauts even if Windows itself can. Is there code in the Octave interpreter that rejects characters outside a certain range even if they're inside string delimiters? If not, where are the umlauted chars going? Are they just not being displayed or are they not entered in the command buffer at all?

Anonymous
Sat 02 Apr 2016 08:46:31 AM UTC, comment #13:

Last bit should read:
Copy-pasting it (= Müller) from e.g.,notepad++ to Octave the ü is lost)

Philip Nienhuis <philipnienhuis>
Project Member
Sat 02 Apr 2016 08:45:14 AM UTC, comment #12:

Where you write:

you can see that Octave accepts a mix of single- and double byte strings. When the ö and ü are compared, only the second part of the double byte char is found to be differing. That looks a bit slippery to me.

What does
uint8 (tmp)
or
uint16 (tmp)
return for the first statement? (can't try myself as umlaut here on my US intl keyboard. Copying it from e.g.,notepad++ the ü is lost)

Philip Nienhuis <philipnienhuis>
Project Member
Fri 01 Apr 2016 11:06:12 PM UTC, comment #11:

Looks OK...?

Anonymous
Fri 01 Apr 2016 07:38:30 AM UTC, comment #10:

update to comment #9:

Now I tried directly in the commandline of the GUI of octave 4.0.0:

>> [Status, Text]=dos("chcp 65001")

Status = 0
Text = Aktive Codepage: 65001.

But I still cannot enter umlauts and also cut&paste of umlauts does not work.

Martl <gsassle>
Fri 01 Apr 2016 07:15:53 AM UTC, comment #9:

The Tip of comment #8 does not work for me:
I started cmd.exe, changed the code page from 850 to 65001, and I can enter umlauts (but this worked also with default 850). Then I startet octave-cli in cmd, but I still cannot enter umlauts.

Concerning comment #1: Another strange thing is, that after downgrading from 4.0.1 to 4.0.0, now 4.0.0 does not understand umlauts either! But the old installation of 4.0.0 from last July did understand umlauts without any special configuration, hacks etc. Of course, there have been plenty windows updates in the meantime...

Martl <gsassle>
Thu 31 Mar 2016 05:49:45 PM UTC, comment #8:

Here is one way to get umlauts in Windows. This works on Windows 7 Professional. Not tested on other versions.

1. Start cmd (Windows command line interface).

2. Type alt-space, then click Properties, go to the Fonts tab, and pick something Unicode-aware like Consolas. Do not leave it as a bitmap font.

3. Type chcp at the prompt and enter. It will probably say 437. Now type "chcp 65001" and enter. That will set it to UTF-8.

4. Now you'll be able to run some command like "echo Müller". Copy-paste Müller from somewhere else, or if you have a German keyboard you can type it directly. Now if you run Octave in CLI mode from the same cmd window, you should be able to see and input UTF-8 characters.

The command "chcp" changes the code page -- Windows by default uses code page 437 which is the original IBM PC character set from the 1980s, but it also treats all of UTF-8 as yet another code page (!!!) numbered 65001.

This hack probably needs much more testing with other languages and scripts, not just German. But if you use it often enough, you can probably write a batch file to chcp 65001 before running Octave.

Anonymous
Thu 31 Mar 2016 03:59:02 PM UTC, comment #7:

Does Option 1 solve the problem for someone on Windows? I have no idea how to configure that, but if you or someone can confirm that inputting UTF-8 into Octave allows non-ASCII characters to be entered, then we can say we have the same experience on Windows as on GNU/Linux systems.

Mike Miller <mtmiller>
Project Administrator
Thu 31 Mar 2016 03:22:17 PM UTC, comment #6:

Situation: wchar_t and std::wstring are almost never needed in Linux & Mac, where UTF-8 is used. Using wchar_t and std::wstring are common in Windows, which uses UTF-16.

Option 1: keep Octave as it is, and configure the OS to convert UTF encodings as needed. Can be painful for the user if they use legacy software that needs Windows-1252 encoding. Otherwise, converting everything to UTF-8 will help, as described here: http://www.utf8everywhere.org/

Option 2: make extensive use of typedefs or (worse) #IFDEF statements where needed to support cross-platform UTF encoding differences. Very complicated and not practical.

Anonymous
Thu 31 Mar 2016 02:47:49 PM UTC, comment #5:

Character strings in Octave are simple std::string and char* objects. They won't properly handle wide character sets, or necessarily even be 8-bit clean. As far as I know, there is no simple fix for this problem.

John W. Eaton <jwe>
Project Administrator
Thu 31 Mar 2016 02:24:28 PM UTC, comment #4:

See Ticket #42036

"4) When trying to enter "cd 'München'" in the command window by hand, the Umlaut "ü" will be replaced by a verical bar '|'.

[...]

5) Getting the Umlaut via copy & paste from an external ANSI editor is not pssible, either."

Armin Müller <arm_in>
Thu 31 Mar 2016 09:21:24 AM UTC, comment #3:

I can confirm the German umlaut problem in the command line (GUI), running Octave 4.0.1 under Windows 7 (64 bit).

Anonymous
Wed 30 Mar 2016 11:26:16 PM UTC, comment #2:

Here is more data, as this isn't just an umlaut problem:

Latin, extended Latin, Greek, Cyrillic: print properly on GUI, can be entered properly via GUI, print OK on CLI with some cramped accents occasionally, not reliable entering them from keyboard on CLI.

CJK, Indic and Thai: print properly on GUI, some rare difficulties in entering via GUI (combination chars), print very badly on CLI (no combinations at all), cannot be entered without mangling badly on CLI.

Armenian, Georgian, Arabic, Hebrew, Farsi, Ethiopic scripts: cannot evaluate due to unfamiliarity.

Textread and fprintf work fine for a Unicode sample file with multiple scripts listed above.

Strfind, strrep, and textscan are OK for Latin, extended Latin, Greek, Cyrillic. Not OK for Thai, Indic, (fail on combination characters). CJK looks OK but needs more testing.

All the above on Debian Jessie, standard terminal for CLI, Qt for Octave GUI.

Anonymous
Wed 30 Mar 2016 09:56:02 PM UTC, comment #1:

Using a German Win 7 and keyboard, I cannot enter umlauts in neither v4.0.0 nor v4.0.1 (both CLI and GUI) command line. I also tried "ALT+142" and so on. In my regular Win CMD.exe it works. As described before, the editor can handle UTF-8 characters.

If you reinstall v.4.0.0, can you still type umlauts? Did you do any special configuration (e.g. usage of certain console fonts, installed some Windows updates?)

Kai Torben Ohlhus <siko1056>
Project Member
Wed 30 Mar 2016 09:47:56 AM UTC, original submission:

I use Octave 4.0.1 in Windows with a german Keyboard.

In new Version 4.0.1, I cannot enter Umlauts like ö,ä, ß, or µ in the command line (Befehlsfenster). In the editor, this is possible.

In the old V4.0.0, entering umlauts was no problem.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -unavailable- added by lostbard (Posted a comment)
  • -unavailable- added by mmuetzel
  • -unavailable- added by mmuetzel (Posted a comment)
  • -unavailable- added by tiagoma
  • -unavailable- added by tmacchant (Posted a comment)
  • -unavailable- added by lachlan (Posted a comment)
  • -unavailable- added by gsassle (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by arm_in (Posted a comment)
  • -unavailable- added by siko1056 (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 9 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 12 Apr 2017 03:20:19 PM UTCmmuetzelDependencies-=>bugs #50784 is dependent
    Wed 12 Apr 2017 03:14:51 PM UTCmmuetzelSummaryProblem entering umlaut in command line=>Problem entering non-ASCII-character (umlaut) in command line
    Tue 04 Apr 2017 08:49:29 PM UTCmmuetzelCarbon-Copy-=>Added lostbard
    Tue 04 Apr 2017 08:10:34 PM UTCmmuetzelDependencies-=>bugs #50284 is dependent
    Thu 23 Mar 2017 12:41:12 PM UTCmmuetzelRelease4.0.1=>dev
      SummaryProblem with umlaut in command line=>Problem entering umlaut in command line
    Thu 04 Aug 2016 05:49:54 PM UTCtiagomaCarbon-Copy-=>Added tiagoma
    Wed 30 Mar 2016 09:56:02 PM UTCsiko1056Item GroupNone=>Regression
      StatusNone=>Need Info

    Back to the top


    Powered by Savane 3.1-cleanup1