bugGNU Octave - Bugs: bug #60711, textscan with '.' char read as...

 
 

bug #60711: textscan with '.' char read as float

Submitter:  Paul Netsaver <netsaver>
Submitted:  Mon 31 May 2021 01:35:59 PM UTC
   
 
Category:  Octave Function Severity:  2 - Minor
Priority:  3 - Low Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  None
Originator Name:  Netsaver Paul Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 07 Mar 2022 10:10:54 AM UTC, comment #7: 

The new BIST seems to be passing on all platforms for which we have CI. And no complaints from testers so far.

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Mon 28 Feb 2022 04:59:09 PM UTC, comment #6: 


> This time, I didn't forget to add you to the list of contributors that appears at the beginning of the manual.

Thanks!

Gaël Bonithon <tamaranch>
Mon 28 Feb 2022 04:39:06 PM UTC, comment #5: 

Your analysis sounds reasonable and your proposed change fixes the issue for me.
Thank you for the contribution.

I pushed your change with minor changes to the default branch here:
https://hg.savannah.gnu.org/hgweb/octave/rev/5fbc41f7a8d1

This time, I didn't forget to add you to the list of contributors that appears at the beginning of the manual.

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Mon 28 Feb 2022 03:48:38 PM UTC, comment #4: 

Hello, attached is a proposed fix.

(file #52941)

Gaël Bonithon <tamaranch>
Fri 04 Jun 2021 03:56:25 PM UTC, comment #3: 

Marking as Confirmed since there is a demonstrated Matlab incompatibility.  However, I'm not sure when any developer will have time to take this up.

Rik <rik5>
Group administrator
Wed 02 Jun 2021 08:55:10 PM UTC, comment #2: 


>>I'm not sure why Matlab decides there are two values here when the delimiter was clearly specified to be the comma ','. 

I think the problem is not the delimiter, it is the dot.
Replace the dot with another char, e.h. 'a':

>> q=textscan([',,a,,3'],'%f','delimiter',',')
q =
{
  [1,1] =
     NaN
     NaN
}

You see that textscan stops when doesn't find a match (this results is the same in Octave and Matlab), while NaN does not stop the scan as it's a match with an EmptyValue.
I think that the true difference is that according to Matlab the dot '.' does not match a float '%f' (therefore stops the scan and does not read the following NaNs), while Octave does match with a NaN float, irrespective of the delimiter.
Probabily Octave starts a matching with '.' waiting for a number (e.g. '.0') and initializes a NaN, then when it finds the limiter and cannot convert to a number leaves the NaN.
Matlab instead understands that '.' alone cannot resolve to a number (even empty) and stops the scan.
By the way I can resolve with a dedicated check, it was only to understand if this was the expected behavior or not.

Thanks and regards,
Netsaver
 




Paul Netsaver <netsaver>
Wed 02 Jun 2021 12:02:50 AM UTC, comment #1: 

There is a difference with Matlab, but Matlab's approach doesn't make as much sense to me as Octave's.

Take a slightly different example for illustrative purposes.


q=textscan([',,2,,'],'%f','delimiter',',', 'emptyvalue', -1)


I specifically set the EmptyValue option to something besides NaN so the processing of textscan is more obvious.  In both Octave and Matlab this returns


q =
{
  [1,1] =

    -1
    -1
     2
    -1

}


This makes sense to me.  There is a missing value at the start of the string, the second position is also missing, the third value is the number 2, and then then there is another missing value.

Now try reading just a single '.' character.


textscan ('.', '%f')


Both Octave and Matlab return [] for this.

Based on this data, I would expect that ',,.,,' would be treated as EmptyValue EmptyValue EmptyValue EmptyValue which is exactly what Octave does.  I'm not sure why Matlab decides there are two values here when the delimiter was clearly specified to be the comma ','.

Of course, if you have built your textscan parser to rely on this Matlab-specific behavior then it will be a problem.  A simple hack would be to use regular expressions and search&replace bare instances of '.' with something else including possibly nothing.

Rik <rik5>
Group administrator
Mon 31 May 2021 01:35:59 PM UTC, original submission:  

According to matlab, textscan function "stops when it cannot match formatSpec to the data."
While writing a wrapper for textscan in my application, I could see that the char '.' (alone), when matched with a float with '%f' specifier, is read as 'NaN', letting the scan continue.
In matlab it doesn't match with '%f', stopping the scan at that position.
Here the text in the first parameter position emulates a file with 'eol' chars, but the results were the same with a real file...

q=textscan([',,.',newline,',,'],'%f','delimiter',',')
q{:}
matlab: NaN NaN
octave: q =
{
  [1,1] =

     NaN
     NaN
     NaN
     NaN
     NaN

}


the 3rd nan matches the '.' char
Note that (as with matlab), empty values before the delimiter are converted to NaN, while empty values after the delimiter and before the eol are not converted to nan (but does not stop the scan).

Marked as Matlab compatibility issue, but don't know if wanted or not.

regards,
Netsaver

Paul Netsaver <netsaver>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #52941:  bug60711.patch added by tamaranch (2KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by tamaranch (Updated the item)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by netsaver (Submitted the item)
  • -email is unavailable- added by netsaver
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-03-07 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2022-02-28 mmuetzel StatusConfirmed Ready For Test
        Release6.2.0 dev
    2022-02-28 tamaranch Attached File- Added bug60711.patch, #52941
    2021-06-04 rik5 Severity3 - Normal 2 - Minor
        Priority5 - Normal 3 - Low
        StatusNone Confirmed
    2021-06-02 rik5 Operating SystemMicrosoft Windows Any
    2021-05-31 netsaver Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code