bugGNU Octave - Bugs: bug #59314, [octave forge] (mapping) str2angle...

 
 

bug #59314: [octave forge] (mapping) str2angle fails to parse string with UTF-8 degree (°) character

Submitter:  Rafael Laboissière <rlaboiss>
Submitted:  Wed 21 Oct 2020 03:06:18 PM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * other Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 23 Oct 2020 07:15:51 AM UTC, comment #12: 

The issue that is fixed by this change will no longer occur with Octave 6 and later. I don't think we need to apply this patch in MXE Octave.

Markus Mützel <mmuetzel>
Group administrator
Thu 22 Oct 2020 08:37:51 PM UTC, comment #11: 

Verified patch on Linux with Octave-"6.0.1" & 7.0.0, on Windows w. 6.0.90a and same 7.0.0, and pushed.

@Rafael: thanks for the report and the patch.
@Kai & Markus: thanks for the discussion.

Should there be a patch for mxe-octave? I suppose we'll see Octave-6.1 before the end of this year and then no more patch is needed for the Windows builds.

Anyway, closing report.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 22 Oct 2020 11:43:24 AM UTC, comment #10: 

Sorry again. Please disregard my comment #8. I mis-read my own comment #5...
Afaict, the change is save. Something like it should be done imho.

Markus Mützel <mmuetzel>
Group administrator
Thu 22 Oct 2020 10:34:04 AM UTC, comment #9: 

I forgot to mention, in my original report, that the bug happens in version 5.2.0 of Octave.

Rafael Laboissière <rlaboiss>
Thu 22 Oct 2020 09:01:36 AM UTC, comment #8: 

Aaaah. I just now noticed my typo in comment #5. I wanted to write:

> If you are: I believe it is not.


D'oh!

Markus Mützel <mmuetzel>
Group administrator
Thu 22 Oct 2020 08:56:56 AM UTC, comment #7: 

I believe, it makes sense to apply the original patch.

Look at this part of the original expression: "([^+-]\s?)".
It matches one single character potentially followed by a white space character. IIUC, that will e.g. match "°" or "° " in Octave 6 and later. But because the UTF-8 encoded "°" is actually two bytes, it won't match any of these strings in Octave 5 or earlier.
If you don't like that "°" is matched literally, you could also use "([^+-]{1,4}\s?)". But I'd argue that the original patch is probably safer.

If you are worrying that the additional complexity might slow down processing, you could run performance tests. If you find that there is an impact on performance, only use the more complex regex in Octave 5 and earlier.

Markus Mützel <mmuetzel>
Group administrator
Thu 22 Oct 2020 08:15:51 AM UTC, comment #6: 

@Markus, Sorry for the confusion.  I understand your concern with non-ASCII in regular expressions and I would neither be happy about the OP's patch to be applied.

It seems to be not a general problem, showing up on every system.  For me the problem only showed up once in a particular system setup.  I don't know what setup the OP is using, maybe that is helpful for identifying the circumstances this bug shows up.


Kai Torben Ohlhus <siko1056>
Group Member
Thu 22 Oct 2020 07:16:39 AM UTC, comment #5: 

I think I lost you somewhere. Are you debating whether this change is save for Octave prior to version 6?
If you are: I believe it is.
A problematic pattern would e.g. be "[D°]" because in previous Octave versions that would match any of the three bytes 0x44, 0xC2, and 0xB0 independently. That means it would also match the first byte of e.g. UTF-8 encoded "µ" (0xC2 0xB5).

If I read the regular expression correctly, there are no non-ASCII characters in bracket expressions. My comment was just meant to warn about potential changes to that regular expression.

Markus Mützel <mmuetzel>
Group administrator
Thu 22 Oct 2020 05:36:14 AM UTC, comment #4: 

Regarding comment #3, can you confirm, that Octave 6 has trouble with it?  I cannot.  Can you be more verbose about the used Octave version and operating system?

Regarding comment #2, Philip, I hit this problem with a special compiled version of Octave 5.2.0 (https://sourceforge.net/p/octave/package-releases/425/#d41d).  Maybe the OP uses something similar?

Kai Torben Ohlhus <siko1056>
Group Member
Thu 22 Oct 2020 05:24:15 AM UTC, comment #3: 

I did not test it, but I guess that my patch would work in both versions 5 and 6 of Octave. Could anyone test it, please?

Rafael Laboissière <rlaboiss>
Wed 21 Oct 2020 09:35:48 PM UTC, comment #2: 

Should anything be done with angl2str.m?

I didn't hit this with Octave-6.0.9a and dev Octave. Nor did Kai (OF admin) and other testers.
Obviously it isn't nice for current Octave-5.2.0. I suppose the tests can be made conditional on Octave version?

(typos in title fixed)

Philip Nienhuis <philipnienhuis>
Group Member
Wed 21 Oct 2020 04:13:23 PM UTC, comment #1: 

Please note that `regexp` in Octave before version 6 doesn't correctly handle UTF-8 encoded characters. Before that version the `regexp` function matched "byte-wise". That means that a pattern might match an incomplete UTF-8 character.
Afaict, this won't be an issue with this particular pattern. Just something to keep in mind.

Markus Mützel <mmuetzel>
Group administrator
Wed 21 Oct 2020 03:06:18 PM UTC, original submission:  

In version 1.4.1 of the mapping package, the str2angle function fails to parse strings containing the degree (°) symbol.  For instance :


> pkg load mapping
> str2angle ("22°52'44.25\"")
ans =  Inf


With the patch attached to this bug report, we get this result:


> str2angle str2angle ("22°52'44.25\"")
ans =  22.879


This is the reason the unit tests are failing:


octave:5> test str2angle
***** shared tst, res
 tst = '191E21''3.1"\n12e 22''33.24"\n13E 23'' 33.344"\n14w24'' 33.4444"\n';
 tst = [tst '15S25''33.54444"\n16W26''33.644444''''\n17s27''33.7444444"\n'];
 tst = [tst '18N28''33.84444444"\n19d29m33.944444444s\n20D20M33.04444444Se\n'];
 tst = [tst '21°51''4.1"\n22°52''44.25"\n23° 53''33.34"\n24°54'' 33.44"N\n'];
 tst = [tst '25° 55'' 33.544"\n26°56''33.644''''S\n27°57'' 33.744''''\n'];
 tst = [tst '28°58''33.844"w'];
 tst = strrep (tst, '\n', char(10));
 res = [191.351, 12.376, 13.393, -14.409, -15.426, -16.443, -17.459, 18.476, ...
        19.493, 20.343, 21.851, 22.879, 23.893, 24.909, 25.926, 26.943, ...
        27.959, -28.976];
 assert (str2angle (tst), res, 1e-3);
!!!!! test failed
ASSERT errors for:  assert (str2angle (tst),res,1e-3)

  Location  |  Observed  |  Expected  |  Reason
     .         O(1x10)      E(1x18)      Dimensions don't match
shared variables   scalar structure containing the fields:

    tst = [](0x0)
    res = [](0x0)


Rafael Laboissière <rlaboiss>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by rlaboiss (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-10-22 philipnienhuis StatusNeed Info Fixed
        Open/ClosedOpen Closed
    2020-10-22 siko1056 StatusNone Need Info
    2020-10-21 philipnienhuis Summary[octave forge] (mapping) str2angle fail to parse sgring with UTF-8 degree (°) character [octave forge] (mapping) str2angle fails to parse string with UTF-8 degree (°) character
        Carbon-Copy- Added philipnienhuis
    2020-10-21 rlaboiss Attached File- Added str2angle-accept-degree-symbol.patch, #50035

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code