bugGNU Octave - Bugs: bug #47553, textscan Whitespace characters...


bug #47553: textscan Whitespace characters different from Matlab

Submitted by:  Rik <rik5>
Submitted on:  Mon 28 Mar 2016 04:56:30 AM UTC  
Category:  Octave Function Severity:  1 - Wish
Priority:  1 - Later Item Group:  Matlab Compatibility
Status:  Need Info Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Any

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Sat 16 Apr 2016 05:25:21 AM UTC, comment #10:

I'd like to look into it at some stage, so I've just downgraded it to minimum priority. You're welcome to close it, though; I can reopen it if/when I fix it.

Lachlan Andrew <lachlan>
Thu 14 Apr 2016 04:24:52 PM UTC, comment #9:

Is there a way to proceed forward on this issue report, or should it just be closed? It seems like we're mostly Matlab compatible with some potential small incompatibilities on the margin because the textscan algorithm is so complicated.

Rik <rik5>
Project Administrator
Fri 08 Apr 2016 09:52:02 PM UTC, comment #8:

That same non-textscan incompatibility exists on Windows as well.
I know because I sometime make counters in .m files that use CR to avoid scrolling up. But those do not work in Matlab - scrolling is unavoidable there.

Ruminating my last comments I think there appears to be some logic in how Matlab processes endofline/whitespace/delimiters. There must be some consistent logic, after all.

First it processes (eliminates) endofline, maybe add it to delimiters.
Next it seems to collapse multiple whitespace.
Then it adds remaining whitespace to the delimiter collection.
Then it splits up the text stream.

Some noise exists as delimiters can also be non-numeric characters cuddling a preceding numeric field.

At least that is how far I got trying to understand textscan's logic.

Philip Nienhuis <philipnienhuis>
Project Member
Fri 08 Apr 2016 06:07:20 AM UTC, comment #7:

Rik, Philip's comment reminded me why the Octave documentation disagrees with Matlab's: Matlab's was unclear to me, and I describes the best that I could reverse-engineer from Matlab's actual behaviour.

Philip, thanks for those test cases. I'll look into them, and (maybe) fix them and/or add BIST cases.

You have demonstrated another non-textscan incompatibility. On Linux at least, Octave gives

Note the absence of '4'. Octave's GUI treats char(13) as a true carriage return rather than Matlab's behaviour of treating it as NL+CR.

Lachlan Andrew <lachlan>
Mon 28 Mar 2016 07:18:44 PM UTC, comment #6:

As an example of the interplay/interference of the various whitespace/delimiter/endofline defaults, consider this (ML r2016a prerel.):


Philip Nienhuis <philipnienhuis>
Project Member
Mon 28 Mar 2016 07:09:44 PM UTC, comment #5:

Which docs do you refer to?
If it's the Matlab docs, yes they are confusing. I think that adequately reflects the "one tool for all you can think of" that textscan came to be.
As to those docs, I'm even unsure if it is "\b\t" or " \b\t" (note the space) that should be the default whitespace.

Anyway this bug report uncovers a can of worms in a way.
My understanding (as far as I could infer textscan's working) is as follows:

- whitespace (" \b\t") serves as a type of delimiter of which several consecutive ones are always collapsed into one;

- delimiter (default a space) also delimits text fields. Consecutive ones will not be collapsed into one ... unless multipledelimsasone is specified;

- endofline (\r\n, r\r, or \n) are default delimiters and added to the delimiter collection, unless it is set to empty in which case it is treated as whitespace.

So whitespace, delimiters and endofline overlap in some sense.

Things become interesting if whitespace, delimiters and endofline all occur in a file / text srring and in textscan's input arguments.
Although I largely rewrote the textscan.m/strread.m combo, my memory gets a bit confused, sorry. ATM I am morphing strread.m from a self-contained function into one invoking binary textscan as backend and I'm bitten by exactly these "uncertainties" - IOW in hindsight I sometimes doubt if I've been doing the right things all along.

Back to whitespace, Matlab 2016a prerelease does this:

... showing that whitespace somehow gets "promoted" to delimiters, regardless of numeric or text fields in the input. Only \r (char(13) is treated as special. But maybe that could be due to the default "endofline" character (intractably for us inferred from the input file/string).

It could be that ML's textscan doesn't explicitly turn whitespace into delimiters, but simply stops reading whenever a next character doesn't fit in the numerical scheme, or when reading text, assumes whitespace doesn't occur in text fields.

The rationale may be very logical and straightforward, but the results can be a bit confusing - for me at least :-)

As to the example in comment #3:

Rik's first command gives:

...IOW the same as in comment #3.

Philip Nienhuis <philipnienhuis>
Project Member
Mon 28 Mar 2016 06:12:18 PM UTC, comment #4:

I only reported this because I was re-working the documentation string and noticed the difference with Matlab.

Does the example in comment #3 yield something different in Matlab?

I tried

and it gave the same result as

so perhaps this is only a documentation issue to fix.

Rik <rik5>
Project Administrator
Mon 28 Mar 2016 06:01:57 PM UTC, comment #3:

Is there a simple motivating example for this that shows the incompatibility? The textscan help is confusing when talking abstractly about whitespace and input rows and delimiters and end of line characters. Something like this maybe?

Mike Miller <mtmiller>
Project Administrator
Mon 28 Mar 2016 10:40:14 AM UTC, comment #2:

Looking at libinterp/oct-stream.cc L.2533, whitespace is declared as " \b\t". So it is somewhere down the road that eol1 ("\r") and eol2 ("\n") appear to be added to the whitespace collection?

Philip Nienhuis <philipnienhuis>
Project Member
Mon 28 Mar 2016 09:32:23 AM UTC, comment #1:

Thanks, Rik. I've forgotten the reason for that, but as I recall, it was to make the behaviour match Matlab's in some context. It may break it in another (such as when the new-line character is defined to be something other than \r, \n or \r\n), but for now there are many bigger bugs to chase...

A related incompatibility is that Matlab only interprets delimiters \b, \n, \r, \t and \\ within a single-quoted string, whereas Octave interprets its full range of escapes, including \a (=7), \f (=12) and \v (=11). However, I consider Octave's behaviour better and unlikely to cause problems.

Lachlan Andrew <lachlan>
Mon 28 Mar 2016 04:56:30 AM UTC, original submission:

According to the documentation, the default Whitespace characters are ' \b\t'. Octave also includes the end-of-line characters '\r\n'.

See http://www.mathworks.com/help/matlab/ref/textscan.html.

Rik <rik5>
Project Administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by lachlan (Posted a comment)
  • -email is unavailable- added by rik5 (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-04-16 lachlan Severity2 - Minor => 1 - Wish
        Priority5 - Normal => 1 - Later
    2016-04-08 philipnienhuis Operating SystemGNU/Linux => Any
    2016-03-28 mtmiller StatusNone => Need Info

    Back to the top

    Powered by Savane 3.3