bugGNU Octave - Bugs: bug #54391, Incorrect result when attempting...

 
 

bug #54391: Incorrect result when attempting to type or paste UTF-8 Cyrillic text into octave CLI

Submitter:  Alan W. Irwin <airwin>
Submitted:  Thu 26 Jul 2018 11:16:31 PM UTC
   
 
Category:  Libraries Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 24 Sep 2018 05:49:42 PM UTC, comment #11: 

Thanks, Mike, for clearing this up for me. I didn't understand what the intended behaviour of the bound function was. I now understand that they never worked for me as well.

I pushed the patch here:
http://hg.savannah.gnu.org/hgweb/octave/rev/483e74e0c1c3

Markus Mützel <mmuetzel>
Group administrator
Mon 24 Sep 2018 05:20:44 PM UTC, comment #10: 

On the contrary, I think the Meta-N and Meta-P key bindings in liboctave/util/cmd-edit.cc are not working at all, and maybe haven't been for quite a long time, and no one noticed.

The readline library itself binds Meta-N and Meta-P to the functions 'non-incremental-forward-search-history' and 'non-incremental-reverse-search-history', respectively. When these functions are activated, the input shows a ':' prompt, the user types some search string, presses Enter, and the first history substring match is recalled.

This is very different from the functions that Octave is attempting to bind Meta-N and Meta-P to. In cmd-edit.cc, the keys are bound to 'history-search-forward' and 'history-search-backward', respectively. When these functions are called, whatever the user has already typed in the input buffer is used as the substring search term.

When I run an unpatched Octave, built with readline 7, Meta-N and Meta-P give me the readline built in behavior, not the behavior that Octave intended with the bindings in cmd-edit.cc. So I agree that these can be safely deleted because they are not effective with a current version of readline.

Mike Miller <mtmiller>
Group Member
Sun 23 Sep 2018 01:02:14 PM UTC, comment #9: 

Those hard-coded bindings have been part of cmd-edit.cc since it was added with hg id 66ef74ee5d9f in May 97:
http://hg.savannah.gnu.org/hgweb/octave/rev/66ef74ee5d9f
Maybe those bindings were necessary back then and are no longer now?

Markus Mützel <mmuetzel>
Group administrator
Sat 22 Sep 2018 11:35:15 PM UTC, comment #8: 

Forgot to mention that the patch doesn't break the default behaviour of Alt+P and Alt+N at the Octave prompt.

Markus Mützel <mmuetzel>
Group administrator
Sat 22 Sep 2018 11:31:19 PM UTC, comment #7: 

I don't know why we ever needed the hard-coded readline bindings in the first place. Alt+P and Alt+N are already bound to "non-incremental-reverse-search-history" and "non-incremental-forward-search-history" by default. The only difference to "history-search-backward" and "history-search-forward" seems to be that the latter should only match the start of a line. [1] (But that isn't how Alt+P behaved for me before the patch.)

Readline 6.3 and newer should be handling UTF-8 charsets correctly automatically. At least as long as we (or the user) aren't messing with the meta-settings.
There still seems to be a problem with rl_add_defun and Meta-key shortcuts.
It might be best to remove the calls to rl_add_defun with Meta-key shortcuts entirely. If it should be necessary, a user could still switch back to the old functions in their ~/.inputrc with:

"\M-p": history-search-backward
"\M-n": history-search-forward


With the patch applied I can paste the Cyrillic string from comment #0 also in Octave for Windows (cross-compiled with readline 6.3 from bug #47571) which previously failed as well.
Alt+P and Alt+N never worked for me on Windows. (But I don't know to which key the Meta key corresponds on Windows.)

[1]: https://tiswww.case.edu/php/chet/readline/readline.html#SEC15

Markus Mützel <mmuetzel>
Group administrator
Sat 22 Sep 2018 10:06:59 PM UTC, comment #6: 

What I don't understand is why this affects Octave, but not bash. The compiled in defaults of readline and bash bind Meta+P to the function "non-incremental-reverse-search-history", and Alt+P on my keyboard triggers this. But I am still able to enter Cyrillic multibyte characters at the bash prompt. Same with the rlwrap command. Is there something that Octave can do when initializing readline to handle multibyte characters better?

Mike Miller <mtmiller>
Group Member
Sat 22 Sep 2018 09:49:49 PM UTC, comment #5: 

Good catch. Agree with this analysis, Meta+P is 0x80 + 0x50 == 0xD0. I didn't realize Octave was doing readline bindings internally outside of the configurable inputrc file.

I was also able to work around this by adding either of the following two equivalent lines to my ~/.inputrc:


"\320": self-insert
"\xd0": self-insert


This allows the byte 0xD0 to be recognized specifically by readline and passed through as is, which seems to override the Meta+P interpretation. Meta+P will still work in this case if the user uses the Escape key as a prefix.

Mike Miller <mtmiller>
Group Member
Sat 22 Sep 2018 09:04:32 PM UTC, comment #4: 

I could track this down to GNU readline's implementation of the Meta-key and how we use readline in Octave:

Essentially, readline uses the highest bit of an 8-bit char to indicate that the Meta (Alt) key was pressed at the same time as a key for that character. That is all well as long as we are only dealing with ASCII characters.
However with UTF-8, the highest bit of a 8-bit char is also set in multi-byte characters.

The first byte of "Ч" (and the other characters that failed pasting) is decimal 208. It just so happens that this is the same as Latin "P" (decimal 80 in ASCII/UTF-8) with the highest bit set to 1.
We are registering Meta-P as a shortcut for readline's "history-search-backward" function.

If I remove that shortcut, the Cyrillic example string from comment #0 can be pasted just fine on Octave's CLI:

octave:1> x="Частота"
x = Частота
octave:2> uint8(x)
ans =

  208  167  208  176  209  129  209  130  208  190  209  130  208  176


The attached patch removes the two functions that we are registering with the Meta key.

I think the Ctrl key shortcuts are save when it comes to ASCII vs. UTF-8.
For reference I'll paste the corresponding code from readlines's "chardefs.h":

/* Some character stuff. */
#define control_character_threshold 0x020   /* Smaller than this is control. */
#define control_character_mask 0x1f            /* 0x20 - 1 */
#define meta_character_threshold 0x07f            /* Larger than this is Meta. */
#define control_character_bit 0x40            /* 0x000000, must be off. */
#define meta_character_bit 0x080            /* x0000000, must be on. */
#define largest_char 255                    /* Largest character value. */

#define CTRL_CHAR(c) ((c) < control_character_threshold && (((c) & 0x80) == 0))
#define META_CHAR(c) ((c) > meta_character_threshold && (c) <= largest_char)

#define CTRL(c) ((c) & control_character_mask)
#define META(c) ((c) | meta_character_bit)

#define UNMETA(c) ((c) & (~meta_character_bit))
#define UNCTRL(c) _rl_to_upper(((c)|control_character_bit))



Interestingly, the two removed shortcuts (Meta-P and Meta-N) still work for me even with the patch applied (with GNU readline 7.0).


(file #45076)

Markus Mützel <mmuetzel>
Group administrator
Fri 27 Jul 2018 12:11:43 PM UTC, comment #3: 

This might be a long shot: There is "linenoise ng" which is [1]:
"A small, portable GNU readline replacement for Linux, Windows and MacOS which is capable of handling UTF-8 characters."

That might mean that UTF-8 handling is a known issue with readline. But a better reference is needed.

There is also bug #47571 for similar issues in Windows which might (or more probably?) might not be related.

[1]: https://github.com/arangodb/linenoise-ng

Markus Mützel <mmuetzel>
Group administrator
Fri 27 Jul 2018 08:28:11 AM UTC, comment #2: 

Mike Miller asked:

> Have you tried experimenting with different readline settings?


Not until now when I tried your suggestion of -no-line-editing which I confirm on my (Debian Buster) system allows cut and paste
of

x="Частота"

to work properly with the octave CLI.

I also tried the experiment of cutting and pasting the above command in the Python CLI (since I am virtually positive that the Python CLI
uses the readline library).  In that case I got the (correct)
result:

>>> x="Частота"
>>> x

'\xd0\xa7\xd0\xb0\xd1\x81\xd1\x82\xd0\xbe\xd1\x82\xd0\xb0'

So I am fairly sure the issue is not due to some bug in libreadline, but might be caused by some difference between how the Python and Octave CLI's use libreadline.

Alan W. Irwin <airwin>
Fri 27 Jul 2018 12:01:33 AM UTC, comment #1: 

Confirmed, with 4.4 and with the development version. I think you are right that it is specific to Cyrillic characters.

In fact it seems to be specific to Cyrillic characters whose UTF-8 encoding starts with the byte 0xD0. These are the 64 characters with UTF-16 code point U+0400 through U+043F.

The first byte is being interpreted by the readline library as some kind of control character. On my system it seems to do something similar to the Up arrow key or the Ctrl+P binding.

Running octave with --no-line-editing disables readline and allows this to work.

Have you tried experimenting with different readline settings?

Mike Miller <mtmiller>
Group Member
Thu 26 Jul 2018 11:16:31 PM UTC, original submission:  

To demonstrate this issue, execute (using cut and paste)

x="Частота"

in the octave CLI.  The result is gibberish text for x, and incorrect values for the components of x as can be seen from

# The gibberish below comes from the cut and paste attempt
# of the above command
octave:3> x="��ст�т�"
x = ��ст�т�
octave:4> for i = 1:length(x)

> printf("%x\n", x(i));
> endfor

a7
b0
d1
81
d1
82
be
d1
82
b0

If you execute the x assignment in test_cyrillic.m, then you
get the correct result:

test_cyrillic
x
octave:5> test_cyrillic
octave:6> x
x = Частота
octave:7> for i = 1:length(x)

> printf("%x\n", x(i));
> endfor

d0
a7
d0
b0
d1
81
d1
82
d0
be
d1
82
d0
b0

which can be verified (subject to endianess swaps) using od on the bash command line as follows:

software@merlin> echo "Частота" |od -t x2
0000000 a7d0 b0d0 81d1 82d1 bed0 82d1 b0d0 000a
0000017

Note, I tested "Частота" was valid UTF-8 within the PLplot library (since that string is used within one of our examples)
and also by converting it using iconv, e.g.,

software@merlin> echo "Частота" |iconv --from-code UTF8 --to-code UTF8 |od -t x2
0000000 a7d0 b0d0 81d1 82d1 bed0 82d1 b0d0 000a
0000017

I think there might be something specially wrong with cut and paste of Cyrillic for the octave CLI, because I have not encountered these issues with UTF-8 strings for Mandarin, and UTF-8 strings representing math symbols.

Alan W. Irwin <airwin>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #45076:  bug54391_readline_meta.patch added by mmuetzel (1KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by airwin (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-10-02 mtmiller StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2018-09-24 mmuetzel StatusPatch Submitted Ready For Test
    2018-09-23 mtmiller Carbon-CopyRemoved 80942 -
    2018-09-22 mmuetzel Operating SystemGNU/Linux Any
    2018-09-22 mmuetzel Attached File- Added bug54391_readline_meta.patch, #45076
        StatusConfirmed Patch Submitted
        Release4.4.0 dev
    2018-07-27 mtmiller CategoryNone Libraries
        StatusNone Confirmed
        SummaryIncorrect result when attempting to cut and paste UTF-8 Cyrillic text into octave CLI Incorrect result when attempting to type or paste UTF-8 Cyrillic text into octave CLI

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code