bugGNU Octave - Bugs: bug #56805, Using unrecognized char in text...

 
 

bug #56805: Using unrecognized char in text results in infinite warning loop

Submitter:  None
Submitted:  Mon 26 Aug 2019 06:16:32 AM UTC
   
 
Category:  Plotting Severity:  4 - Important
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Fixed Assigned to:  None
Originator Name:  bk Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 5.1.0
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 03 Sep 2019 08:35:00 PM UTC, comment #10: 

I applied all three patches to the development branch.

The next bug fix version of Octave (on the stable branch in Mercurial) does not have an infinite loop problem so I think it's okay to close this report.

Rik <rik5>
Group administrator
Tue 03 Sep 2019 06:47:17 PM UTC, comment #9: 

Attached is another patch that improves the error message for the code in comment #0:

>> text(0,0,char(169))
error: regexprep: internal error calling pcre_exec; the input string is invalid UTF-8
error: called from
    text at line 171 column 14



(file #47445)

Markus Mützel <mmuetzel>
Group administrator
Tue 03 Sep 2019 06:13:56 PM UTC, comment #8: 

@Rik: You are right. I didn't read the sources before commenting.

The two attached patches fix the orientation of the output vector of native2unicode and remove the duplicate input checks from the internal functions.

AKAIK, Matlab uses only the BMP of UTF-16 (i.e. UCS-2). So there is probably no example where a single character in any encoding would lead to a two element result of native2unicode in Matlab.
We decided we want to use UTF-8 instead. If I remember correctly the reasoning was that most third party libraries are using 8bit chars as their strings. Thus, if our strings are of 8bit chars as well we would need to convert them at less interfaces.

This still leaves the original issue.

(file #47443, file #47444)

Markus Mützel <mmuetzel>
Group administrator
Tue 03 Sep 2019 04:43:37 PM UTC, comment #7: 

According to the input validation, arrays and N-D matrices are improper inputs to native2unicode and will cause an error before processing even begins.

Rik <rik5>
Group administrator
Tue 03 Sep 2019 04:41:04 PM UTC, comment #6: 

Seems like there might be a couple things to debug here.

First, the m-file has this input validation,


  if (! ischar (codepage))
    error ("native2unicode: CODEPAGE must be a string")
  endif


The internal C++ function _native2unicode_ in strfns.cc also has


  std::string tmp = args(1).xstring_value ("CODEPAGE must be a string");


In general, we only want to do input validation once for efficiency which usually means the m-file.  The internal function, as signified by the leading "__", expects to be called in the proper way since it should never be reached accidentally.

Second, I ran


x = native2unicode(169, 'latin1')


in Matlab and get just a scalar ([169]) in return.  Is that correct that Octave does not produce the same output as Matlab for the same input?  It may be because they use UTF-16 encoding natively, I believe.

Third, the internal function always returns a row vector.  Hence, I think you're right that you want to test for "! isrow (native_bytes)" and only transpose if that turns out to be true.

Rik <rik5>
Group administrator
Tue 03 Sep 2019 04:40:45 PM UTC, comment #5: 

I forgot to draw matrices and nd arrays into the picture. So please ignore the second option in comment #4. It is wrong.

Markus Mützel <mmuetzel>
Group administrator
Tue 03 Sep 2019 04:11:44 PM UTC, comment #4: 

@Rik: I was trying with Octave 5.1.0 installed on Windows 10 and the result for "native2unicode(169, 'latin1')" was a column vector. (Hence the transpose operator in comment #2.)

You are right. It would be better to use the transpose operator in the code you cited.
And the condition in this code is probably what causes the problem. Both isrow and iscolumn are true for a scalar input. The transpose should only be done for "true" column vectors. Maybe we could use something like:

if (numel (native_bytes) > 1 && iscolumn (native_bytes))
  utf8_str = utf8_str.';
endif


or maybe easier:

if (! isrow (native_bytes))
  utf8_str = utf8_str.';
endif


Markus Mützel <mmuetzel>
Group administrator
Tue 03 Sep 2019 03:42:33 PM UTC, comment #3: 

@Markus: On the last point about the orientation of the returned results being a column vector, do you have an example?

The last few lines of native2unicode.m are


  if (iscolumn (native_bytes))
    utf8_str = utf8_str';
  endif


and seem designed to ensure that the result is a row output.  Also, since this is just about orientation, the code should really use the transpose operator <.'> rather than the Hermitian conjuate operator.

Rik <rik5>
Group administrator
Tue 03 Sep 2019 12:56:26 PM UTC, comment #2: 

Octave uses UTF-8 as its internal encoding. "char(169)" isn't valid UTF-8. You could use the following code to convert from UTF-16 to UTF-8:

native2unicode([0 169], 'utf-16')


Or similarly from Latin1 to UTF-8:

native2unicode(169, 'latin1').'


Notwithstanding, the error message on the development branch could be clearer.

Additionally, it looks like "native2unicode" produces a column vector for a scalar input which is probably an additional bug.

Markus Mützel <mmuetzel>
Group administrator
Thu 29 Aug 2019 02:30:19 PM UTC, comment #1: 

Confirmed.  This result is also the same on a Linux machine so it is not confined to the Windows build.  On the development branch of Octave there is no infinite loop, but it is still not handled correctly.  I get


text(0,0,char(169))
error: regexprep: internal error calling pcre_exec; error code from pcre_exec is -10
error: called from
    text at line 171 column 14



Rik <rik5>
Group administrator
Mon 26 Aug 2019 06:16:32 AM UTC, original submission:  

Using the text function with an unknown char (not displayable?), results in an infinite loop on my machine, showing two warnings infinitely:


text(0,0,char(169))
warning: text_renderer: skipping missing glyph for character '0'
warning: called from
    text at line 170 column 10
warning: text_renderer: skipping missing glyph for character '0'
warning: called from
    text at line 170 column 10
...


This shall display the Copyright char (C), but doesn't.

It seems to be quite similiar to #55974, but I'm unsure whether the issue has been adressed to other sub functions in plotting since I get a different warning.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #47445:  bug56805_regexp_utf8_error.patch added by mmuetzel (1KiB - application/octet-stream)
file #47443:  bug56805_native2unicode_row_vector.patch added by mmuetzel (732B - application/octet-stream)
file #47444:  bug56805_native2unicode_input_checks.patch added by mmuetzel (1KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-09-03 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2019-09-03 mmuetzel Attached File- Added bug56805_regexp_utf8_error.patch, #47445
    2019-09-03 mmuetzel Attached File- Added bug56805_native2unicode_row_vector.patch, #47443
        Attached File- Added bug56805_native2unicode_input_checks.patch, #47444
    2019-08-29 rik5 Severity3 - Normal 4 - Important
        Item GroupNone Segfault, Bus Error, etc.
        StatusNone Confirmed
        Operating SystemMicrosoft Windows Any

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code