bugGNU Octave - Bugs: bug #55974, Octave hangs on plotting text...

 
 

bug #55974: Octave hangs on plotting text object with invalid UTF-8 byte sequence

Submitter:  Roland Baudin <roland65>
Submitted:  Thu 21 Mar 2019 01:57:55 PM UTC
   
 
Category:  Plotting Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Regression
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 5.1.0 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 04 Jun 2019 07:01:29 PM UTC, comment #10: 

@Imad: Thanks. I finally pushed the patch to stable in my name with credits to you here:

http://hg.savannah.gnu.org/hgweb/octave/rev/8a07083c8cfc

I changed the replacement character in EPS files to be the question mark as proposed by Markus.

Closinr report as fixed.

Pantxo Diribarne <pantxo>
Group Member
Tue 04 Jun 2019 01:18:16 PM UTC, comment #9: 

Thanks for providing a fix for both issues.
Regarding the latter (printing): Would you consider using the question mark ? as the replacement character in ASCII? There doesn't seem to be a strict rule. But e.g. according to Wikipedia [1]:

In many Web browsers and other computer programs, when converting text between encodings, it may not be possible to map some characters into the target character set. In this situation it is common to replace each unmappable character with a question mark ?, inverted question mark ¿, or the Unicode replacement character, usually rendered as a white question mark in a black diamond: U+FFFD � REPLACEMENT CHARACTER.


I also believe that gnulib is doing the same in some functions operating on strings.

[1]: https://en.wikipedia.org/wiki/Question_mark?wprov=sfla1

Markus Mützel <mmuetzel>
Group administrator
Tue 04 Jun 2019 09:07:37 AM UTC, comment #8: 

Hi @pantxo,

Thank you for preparing these patches, and for the additional fix.
I have sent you an e-mail with my name and e-mail address, but I suggest that you push both fixes in your name since you bothered to prepare the patches.

Thanks.

ies

ImadES <ies>
Mon 03 Jun 2019 10:31:18 PM UTC, comment #7: 

The same infinite loop happens if you try to print the resulting figure. I attached a patch that also handles this (but uses a blank space since there is no replacement char in ascii).

(file #47020)

Pantxo Diribarne <pantxo>
Group Member
Mon 03 Jun 2019 10:06:21 PM UTC, comment #6: 

@ies: I attached a patch for stable that implements your solution (but with a proper replacement character instead of "?"). Can you add your email and name so that I can push it in your name?

(file #47019)

Pantxo Diribarne <pantxo>
Group Member
Mon 03 Jun 2019 09:34:34 PM UTC, comment #5: 

@ies: I like the proposed fix for the infinite loop (and FWIW Matlab also uses a replacement character in this situation). Can you produce a proper patch?

>> But I must admit at this point that I am not sure why we ended up calling it like this.


We want to use UTF-8 as Octave's internal string encoding, hence the assumption that the string that is passed to the text renderer is UTF-8 encoded.
Octave can make sure a string typed in a terminal, or read from a file (provided the file is loaded with the proper encoding) is translated to UTF-8 before being passed to internal functions. But it can't prevent you from loading files with wrong encodings or building foreign non UTF-8 strings, so the replacement character looks like the best approach.

Pantxo Diribarne <pantxo>
Group Member
Mon 03 Jun 2019 08:40:46 AM UTC, comment #4: 

Hi,

The problem occurs in this function found in 'libinterp/corefcn/ft-text-renderer.cc':


  void
  ft_text_renderer::visit (text_element_string& e)
  {
    if (font.is_valid ())
      {
...
        while (n > 0)
          {
            // Retrieve the length and the u32 representation of the current
            // character
            int mblen = octave_u8_strmbtouc_wrapper (&u32_c, c + icurr);
            n -= mblen;
...
            icurr += mblen;
          }



If and when 'octave_u8_strmbtouc_wrapper' returns -1, it results in an infinite loop.

An obvious way to get rid of this infinite loop and at least avoid having Octave hang is to add:


+            if (mblen < 1)
+              {
+                mblen = 1;
+                u32_c = 63;  // replace with '?'
+              }


right after the call to 'octave_u8_strmbtouc_wrapper'. This is similar to what is done in 'libinterp/octave-value/ov-ch-mat.cc', for instance, and results in display of 'Fr?quence', which I believe is an acceptable outcome.

Now to why the problem occurs in the first place.

In the first (working) example:


s = "Fréquence";
text (0.2, 0.5, s)


the character U+00E9 'é' is coded as 195 169 and is correctly processed by 'octave_u8_strmbtouc_wrapper'.

In the second example based on


$ echo é | iconv -f utf8 -t latin1 | hd
00000000  e9 0a                                             |..|
00000002


'octave_u8_strmbtouc_wrapper' is called with a first byte holding 233=0xE9 and deems this not to be a correct UTF8 code value. But I must admit at this point that I am not sure why we ended up calling it like this.

I would suggest to fix the infinite loop problem, and then discuss further the observed behaviour in the second example.

ImadES <ies>
Thu 11 Apr 2019 10:28:40 PM UTC, comment #3: 

Re-summarizing this bug, and posting your example as an inline snippet.

This works correctly in a UTF-8 locale


s = "Fréquence";
text (0.2, 0.5, s)


The following causes Octave to hang and needs to be killed with SIGKILL


[~, s] = system ("echo Fréquence | iconv -f utf8 -t latin1");
s = strtrim (s);
text (0.2, 0.5, s)


The hang occurs in FreeType, here is the top of the call stack in gdb when stuck


#0  0x00007f5fdd3e151e in ?? () from /lib/x86_64-linux-gnu/libfreetype.so.6
#1  0x00007f5fdd3e5006 in ?? () from /lib/x86_64-linux-gnu/libfreetype.so.6
#2  0x00007f5fdd3e5c58 in ?? () from /lib/x86_64-linux-gnu/libfreetype.so.6
#3  0x00007f5fdd3e83d6 in ?? () from /lib/x86_64-linux-gnu/libfreetype.so.6
#4  0x00007f5fdd3eab73 in ?? () from /lib/x86_64-linux-gnu/libfreetype.so.6
#5  0x00007f5fdd39a9de in ?? () from /lib/x86_64-linux-gnu/libfreetype.so.6
#6  0x00007f5fdd374641 in FT_Load_Glyph () from /lib/x86_64-linux-gnu/libfreetype.so.6
#7  0x00007f5fe0479403 in octave::ft_text_renderer::process_character (this=this@entry=0x55cd357ca940, code=114,
    previous=previous@entry=83) at ../libinterp/corefcn/ft-text-renderer.cc:726
#8  0x00007f5fe047a2e6 in octave::ft_text_renderer::visit (this=0x55cd357ca940, e=...)
    at ../libinterp/corefcn/ft-text-renderer.cc:930
#9  0x00007f5fe0478d65 in octave::text_processor::visit (e=..., this=0x55cd357ca940)
    at ../libinterp/corefcn/text-engine.h:330
#10 octave::ft_text_renderer::visit (this=0x55cd357ca940, e=...) at ../libinterp/corefcn/ft-text-renderer.cc:970
#11 0x00007f5fe047726a in octave::ft_text_renderer::render (this=0x55cd357ca940, elt=0x55cd35835600, box=...,
    rotation=0) at ../libinterp/corefcn/ft-text-renderer.cc:1165
#12 0x00007f5fe04784a0 in octave::ft_text_renderer::text_to_pixels (this=0x55cd357ca940, txt="Fr\351quence\n",
    pxls=..., box=..., _halign=<optimized out>, valign=1, rotation=<optimized out>, interpreter=...,
    handle_rotation=true) at ../libinterp/corefcn/ft-text-renderer.cc:1302
#13 0x00007f5fe078baa2 in octave::text_renderer::text_to_pixels (this=this@entry=0x55cd35873460,
    txt="Fr\351quence\n", pxls=..., bbox=..., halign=halign@entry=0, valign=valign@entry=1,


Mike Miller <mtmiller>
Group Member
Fri 22 Mar 2019 08:38:16 AM UTC, comment #2: 

This is clearly a bug that Octave hangs in this case.

You can find attached another example. This new script 'test_iso88591_2.m' (file encoded as ISO 8859-1) is simpler than the previous one and clearly shows the bug on Linux. It contains only two lines:

display('Fréquence');
xlabel('Fréquence');

On Linux, the first one is executed and the string is displayed as 'Fr�quence', which is what is expected.

The second line makes Octave 5.1.0 hang (OpenGL or fltk graphics toolkit).

In Octave 4.4.1, the second line does silently nothing, which is also a bug.

When a string is not encoded as expected, Octave should display it, with garbage characters where the encoding differs, in console or graphics.


(file #46610)

Roland Baudin <roland65>
Thu 21 Mar 2019 05:01:44 PM UTC, comment #1: 

The short answer is that Octave does not currently have a heuristic way to detect the file encoding simply by looking at the bytes.

If you change the setting in the Preferences, Editor tab, that says "Text encoding used for loading and saving" to be "ISO-8859-1", then your test_iso88591 script works correctly but the UTF-8 encoded script shows garbage characters.

Maybe something can be done about Octave hanging when it tries to decode a file that is not valid UTF-8 and hits an invalid byte sequence. Is that what you'd like to see fixed here?

Mike Miller <mtmiller>
Group Member
Thu 21 Mar 2019 01:57:55 PM UTC, original submission:  

Sometimes I use on Linux some script (m-file) that comes from the Windows world. Such scripts are usually encoded in ISO 8859-1.

If there is in the script some non ASCII character (like 'é', 'à' and other french accents), in a call to functions like xlabel(), ylabel(), title(), etc., then Octave 5.1.0 gets stuck and seems to be lost in an infinite loop.

I attached two scripts to test this issue. Both contain the same code (below, 'Fréquence' is the french word for 'Frequency'):

plot( sin(1 : 1000) );
xlabel('Fréquence');

Note that the script 'test_utf8.m' is encoded in UTF-8 and the script 'test_iso88591.m' is encoded in ISO 8859-1.

In Octave 5.1.0 (system is Ubuntu Linux 18.04 64 bits), test_utf8.m is executed with no error and the word 'Fréquence' is correctly displayed, while test_iso88591.m makes Octave stuck (empty figure window, prompt never returns).

In Octave 4.4.1 (same system), test_iso88591.m runs, but displays the following error message:

warning: ft_text_renderer: failed to decode string `Fr�quence x' with locale `fr_FR.UTF-8'
warning: called from
    _axis_label_ at line 31 column 3
    xlabel at line 51 column 8
    test at line 4 column 1
warning: ft_text_renderer: failed to decode string `Fr�quence x' with locale `fr_FR.UTF-8'
warning: called from
    _axis_label_ at line 31 column 3
    xlabel at line 51 column 8
    test at line 4 column 1

In Windows, both scripts run as expected.

Roland Baudin <roland65>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #47020:  replacemnt_char2.patch added by pantxo (2KiB - text/x-patch)
file #47019:  replacemnt_char.patch added by pantxo (1KiB - text/x-patch)
file #46610:  test2.zip added by roland65 (216B - application/zip - Another (simpler) test)
file #46604:  test.zip added by roland65 (423B - application/zip - Scripts to test the issue)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by pantxo (Posted a comment)
  • -email is unavailable- added by ies (Posted a comment)
  • -email is unavailable- added by roland65 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-06-04 pantxo StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2019-06-03 pantxo Attached File- Added replacemnt_char2.patch, #47020
    2019-06-03 pantxo Attached File- Added replacemnt_char.patch, #47019
    2019-04-11 mtmiller StatusNone Confirmed
        SummaryOctave stuck on Linux when figure contains a string with non ASCII characters encoded in ISO 8859-1 Octave hangs on plotting text object with invalid UTF-8 byte sequence
    2019-03-22 roland65 Attached File- Added test2.zip, #46610
    2019-03-21 roland65 Attached File- Added test.zip, #46604

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code