bugGNU Octave - Bugs: bug #54391, Incorrect result when attempting...

 
 

bug #54391: Incorrect result when attempting to type or paste UTF-8 Cyrillic text into octave CLI

Submitted by:  Alan W. Irwin <airwin>
Submitted on:  Thu 26 Jul 2018 11:16:31 PM UTC  
 
Category:  Libraries Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  4.4.0 Operating System:  GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 27 Jul 2018 12:11:43 PM UTC, comment #3:

This might be a long shot: There is "linenoise ng" which is [1]:
"A small, portable GNU readline replacement for Linux, Windows and MacOS which is capable of handling UTF-8 characters."

That might mean that UTF-8 handling is a known issue with readline. But a better reference is needed.

There is also bug #47571 for similar issues in Windows which might (or more probably?) might not be related.

[1]: https://github.com/arangodb/linenoise-ng

Markus Mützel <mmuetzel>
Project Member
Fri 27 Jul 2018 08:28:11 AM UTC, comment #2:

Mike Miller asked:

> Have you tried experimenting with different readline settings?


Not until now when I tried your suggestion of -no-line-editing which I confirm on my (Debian Buster) system allows cut and paste
of

x="Частота"

to work properly with the octave CLI.

I also tried the experiment of cutting and pasting the above command in the Python CLI (since I am virtually positive that the Python CLI
uses the readline library). In that case I got the (correct)
result:

>>> x="Частота"
>>> x

'\xd0\xa7\xd0\xb0\xd1\x81\xd1\x82\xd0\xbe\xd1\x82\xd0\xb0'

So I am fairly sure the issue is not due to some bug in libreadline, but might be caused by some difference between how the Python and Octave CLI's use libreadline.

Alan W. Irwin <airwin>
Fri 27 Jul 2018 12:01:33 AM UTC, comment #1:

Confirmed, with 4.4 and with the development version. I think you are right that it is specific to Cyrillic characters.

In fact it seems to be specific to Cyrillic characters whose UTF-8 encoding starts with the byte 0xD0. These are the 64 characters with UTF-16 code point U+0400 through U+043F.

The first byte is being interpreted by the readline library as some kind of control character. On my system it seems to do something similar to the Up arrow key or the Ctrl+P binding.

Running octave with --no-line-editing disables readline and allows this to work.

Have you tried experimenting with different readline settings?

Mike Miller <mtmiller>
Project Administrator
Thu 26 Jul 2018 11:16:31 PM UTC, original submission:

To demonstrate this issue, execute (using cut and paste)

x="Частота"

in the octave CLI. The result is gibberish text for x, and incorrect values for the components of x as can be seen from

# The gibberish below comes from the cut and paste attempt
# of the above command
octave:3> x="��ст�т�"
x = ��ст�т�
octave:4> for i = 1:length(x)

> printf("%x\n", x(i));
> endfor

a7
b0
d1
81
d1
82
be
d1
82
b0

If you execute the x assignment in test_cyrillic.m, then you
get the correct result:

test_cyrillic
x
octave:5> test_cyrillic
octave:6> x
x = Частота
octave:7> for i = 1:length(x)

> printf("%x\n", x(i));
> endfor

d0
a7
d0
b0
d1
81
d1
82
d0
be
d1
82
d0
b0

which can be verified (subject to endianess swaps) using od on the bash command line as follows:

software@merlin> echo "Частота" |od -t x2

              1. a7d0 b0d0 81d1 82d1 bed0 82d1 b0d0 000a

0000017

Note, I tested "Частота" was valid UTF-8 within the PLplot library (since that string is used within one of our examples)
and also by converting it using iconv, e.g.,

software@merlin> echo "Частота" |iconv --from-code UTF8 --to-code UTF8 |od -t x2

              1. a7d0 b0d0 81d1 82d1 bed0 82d1 b0d0 000a

0000017

I think there might be something specially wrong with cut and paste of Cyrillic for the octave CLI, because I have not encountered these issues with UTF-8 strings for Mandarin, and UTF-8 strings representing math symbols.

Alan W. Irwin <airwin>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by airwin (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-07-27 mtmiller CategoryNone => Libraries
        StatusNone => Confirmed
        SummaryIncorrect result when attempting to cut and paste UTF-8 Cyrillic text into octave CLI => Incorrect result when attempting to type or paste UTF-8 Cyrillic text into octave CLI

    Back to the top


    Powered by Savane 3.3