bugGNU Octave - Bugs: bug #53277, textscan(...) does not parse...

 
 

bug #53277: textscan(...) does not parse sequence of integers separated by dots (.)

Submitter:  Matthias Brennwald <mbrennwa>
Submitted:  Sat 03 Mar 2018 05:58:45 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  3 - Low Item Group:  None
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * 4.2.1 Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 08 Mar 2018 07:25:36 PM UTC, comment #13: 

@Matthias;
That the old textscan.m (in fact, strread.m) worked fine for your use case turns out to be just coincidence.
In about any general programming/scripting language, trying to read integers that are separated by decimal separators or exponent characters, in fact by any character that can be part of a number, is a risky business to start with, no matter how long the world has been using that convention. Just live with that, be prudent, never blindly trust docs, and use simple tools rather than dinosaurs like textscan.

BTW I saw that Matlab's textscan now supports an option to use locales when reading numbers. Following that in Octave may help solve this issue, OTOH I fear it'll introduce only more concealed bugs and clashes with other options.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 08 Mar 2018 01:08:07 PM UTC, comment #12: 

@Philip: Octave has both scanf and sscanf and they are both in libinterp/corefcn/file-io.cc.  The textscan function is in the same file.  Eventually, textscan seems to be implemented in oct-stream.cc.  jwe did the work on this so I think it would be good to ask him at OctConf 2018 about this.

I can confirm that exponent separators are also a problem, although they work fine with sscanf.


octave:1> str = "1e2e3"
str = 1e2e3
octave:2> sscanf (str, "%de%de%d")
ans =

   1
   2
   3

octave:3> textscan (str, "%de%de%d")
ans =
{
  [1,1] = 100
  [1,2] = 3
  [1,3] = 0
}



Rik <rik5>
Group administrator
Thu 08 Mar 2018 11:12:24 AM UTC, comment #11: 

I can relate to the concept that using dots to separate integers may seem like a bad idea. However, large parts of the world HAVE used this notation for dates for a long time, and certainly for much longer than before Octave or its commercial colleague came to exist. I don't think Octave (or any other software out there) is in a position to dictate the world what date formats are wrong or right.

I don't care about Matlab. But if an Octave function does not work as described in the docs, it should be fixed (either the function or the documentation). In this case, the textscan(...) function changed somewhere from version 4.0 (where it works correctly) to 4.2 (where it does not work correctly).

I am just an Octave user who does not know a thing about C/C++. I can't help fixing the source code, and I leave it to the Octave maintainers to do whatever they think is right.

Matthias Brennwald <mbrennwa>
Thu 08 Mar 2018 10:53:56 AM UTC, comment #10: 

As I expected, TMW attempted to put me off with:
"... recommend to avoid that a dot only follows a number which is meant to be integer."
I nudged them to rethink that a bit I still have little faith there.

While I second TMW's advice (sorry Matthias) I agree that Octave's textscan might do better than Matlab as regards reading data specified as integers.
Is there a C++ equivalent of sscanf in Octave's code base that can be used for this?

BTW 1: We now hit this bug with dots, but exponent characters lead to similar problems. Didn't dare yet to try with i and j (complex).

BTW2: I'll add a note to strread.m's texinfo header to avoid dots and exponent characters as literals or separators for numeric fields; experimenting a bit with those did uncover unexpected / unpredictable results.
Maybe it is better to (provisionally) add such a note to textscan's header as well.


Philip Nienhuis <philipnienhuis>
Group Member
Mon 05 Mar 2018 06:23:25 PM UTC, comment #9: 

@Rik
Sure, I agree.

I am just a little surprised that Matthias' use case apparently has never come up before with Octave or Matlab - having dates with periods between the days.months.years entries isn't very far out IMO.

Anyway I filed a bug report with TMW.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 05 Mar 2018 06:53:27 AM UTC, comment #8: 

Ok, the %2d and %4d is useful to work around the bug. But it's still a bug, because textscan(...) does not work as intended. It does not do what is described in the documentation. The bug did not exist in previous versions of Octave, and it *did* break my code when it appeared. The bug should be fixed.

Matthias Brennwald <mbrennwa>
Mon 05 Mar 2018 05:55:38 AM UTC, comment #7: 

@Philip: I think we can allow the bug report.  I agree with your opinion that textscan is far too complicated, and it is unlikely that TMW will actually fix this.  But because it is so rare, I don't think it hurts to be Matlab-incompatible here and do the right thing.  In addition, we have a guarantee that anyone using Matlab will never use this pattern because it doesn't read the data correctly.  Hence, no one who uses Matlab will actually experience any incompatibility.

I'm marking the bug as confirmed, but also lowering the priority as there are many, many workarounds.  I personally don't have any time to work on this.

For the original reporter, there is no schedule for when this might get fixed so I would just choose one of the workarounds and move on.

If your data is really structured this simply then dlmread() would be a good choice.  If you absolutely must use a file descriptor then try scanf--it gets it right.  Demonstrating with sscanf


 sscanf ('5.2.1975', '%d.%d.%d')
ans =

      5
      2
   1975


You could change the FORMAT string that you give to textscan as Philip suggested to include field widths ('%2d.%2d.%4d').

You could also switch the file to use a different delimiter using something like this Perl command.


perl -i.bak -pe 's/\./#/g' file_name


followed by reading with the pattern '%d#%d#%d'.



Rik <rik5>
Group administrator
Sun 04 Mar 2018 09:22:44 PM UTC, comment #6: 

BTW, this:

>> textscan ('5.2.1975','%2d.%2d.%4d')
ans =
{
  [1,1] = 5
  [1,2] = 2
  [1,3] = 1975
}


works (but still not in Matlab)

Philip Nienhuis <philipnienhuis>
Group Member
Sun 04 Mar 2018 09:05:45 PM UTC, comment #5: 

@ comment #3
Surely this is unexpected behavior, I already wrote that bug-compatibility may be at play here. The actual issue is probably that strread.m (behind the old textscan.m in Octave 4.0.3 and luckily still present in 4.3+) can handle cuddling literals (the "." in your example) while textscan and Matlab's textscan and strread.m can't; they can only process literals surrounded by whitespace and/or delimiters.
OTOH chances are that if the behavior is changed, someone else will complain that his/her Matlab code won't run in Octave. I'm not the one to decide which argument should be leading.

Your implicit analysis that textscan in both Octave and the proprietary competition probably first reads doubles and later on casts them to int32 looks valid enough. The old textscan, in fact strread.m, first splits up the input into tokens along the delimiters, literals (in your case ".") and whitespace, only then tries to read numbers from the tokens alone, so that may explain a few things.
My own opinion is that Matlab's textscan has acquired quite a bit of overweight as far as functionality goes, it has so many options that gotchas like this one are IMO simply unavoidable and maybe even not fixable.


Anyway, I can't fix it, textscan is a binary function now and my C/C++ skills are insufficient. FYI there are several open bug reports for textscan ATM.
Please remember Octave is open source, you're free to try to come up with a fix yourself, we don't take orders, but sometimes take gentle nudges.

@ comment #4:
Try textread.m (a fid front-end for stread.m)
Alternatives that work for relatively homogeneous files are dlmread and csv2cell.

BTW I'll file a bug report about Matlab's textscan with TMW. However my experience with that is not favorable, usually they just change their docs.

Cc'ing Rik, let's await his opinion.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 04 Mar 2018 09:06:46 AM UTC, comment #4: 

Using strread(...) as an alternative to textscan(...) does not work for reading files because the strread(...) function cannot take a file ID as an argument.

Matthias Brennwald <mbrennwa>
Sun 04 Mar 2018 09:03:10 AM UTC, comment #3: 

This is a valid bug, and it should be fixed.

The textscan(...) function clearly does not work as described in the documentation. it has worked correctly in earlier versions of Octave. It's silly to argue that the bug should not be fixed because it also exists in some other software product. I am sure you can do better than this.

Matthias Brennwald <mbrennwa>
Sat 03 Mar 2018 09:04:40 PM UTC, comment #2: 

Surprise for you:
Matlab r2018a prerelease does the same as Octave:

>> u = textscan ('5.2.1975','%d.%d.%d')

u =

  1×3 cell array

    {[5]}    {[1975]}    {0×1 int32}


I agree it is at least unexpected if not confusing, but Octave is Matlab-compatible.
Yes we can argue about bug-for-bug compatibility :-)


Then:
IIRC In Octave 4.0.3 textscan was still an .m-file function. We knew it wasn't fully Matlab compatible then.

For the behavior you desire you can still use strread.m (FYI, before textscan was re-implemented as a binary function, strread.m was textscan.m's work horse behind the scenes):

>> [u, v, w] = strread ('5.2.1975','%d.%d.%d')
u = 5
v = 2
w = 1975
>>


For now I close this bug report as "Invalid".

Philip Nienhuis <philipnienhuis>
Group Member
Sat 03 Mar 2018 07:32:29 PM UTC, comment #1: 

I tried on my dev and result was same as said.
My version is 4.3.0+.

Dildar Sk <hodor123456>
Sat 03 Mar 2018 05:58:45 PM UTC, original submission:  

I am trying to read data from a file with dates in DAY.MONTH.YEAR format using the textscan(...) function. Below is a simple illustration of the problem.

This does not work as desired:

    u = textscan ('5.2.1975','%d.%d.%d')
    u =
    {
      [1,1] = 5
      [1,2] = 1975
      [1,3] = 0
    }

It seems textscan treats the 5.2 part as a decimal number and rounds it to 5. It returns the last number (1975) as the second element of u.

The conversion works as expected if the separator is not a dot:

    u = textscan ('5*2*1975','%d*%d*%d')
    u =
    {
      [1,1] = 5
      [1,2] = 2
      [1,3] = 1975
    }

I have asked about this on stackexchange. The feedback was that my dot example works as expected on Octave 4.0.3. See here: https://stackoverflow.com/questions/49075425/gnu-octave-use-textscan-function-to-read-sequence-of-integers-separated-by-a

Matthias Brennwald <mbrennwa>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by hodor123456 (Posted a comment)
  • -email is unavailable- added by mbrennwa (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-03-05 rik5 Priority5 - Normal 3 - Low
        StatusInvalid / Not an Octave Bug Confirmed
        Open/ClosedClosed Open
    2018-03-04 philipnienhuis Carbon-Copy- Added rik5
    2018-03-03 philipnienhuis StatusNone Invalid / Not an Octave Bug
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code