bugGNU Octave - Bugs: bug #47553, textscan Whitespace characters...

 
 

bug #47553: textscan Whitespace characters different from Matlab

Submitter:  Rik <rik5>
Submitted:  Mon 28 Mar 2016 04:56:30 AM UTC
   
 
Category:  Octave Function Severity:  1 - Wish
Priority:  1 - Later Item Group:  Matlab Compatibility
Status:  Wont Fix Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 12 Nov 2021 04:25:57 AM UTC, comment #11: 

5 years and no additional comments.  closing as per comment #10 until renewed interest appears. since comment #9 suggests there are still a few inconsistencies, setting status to won't fix.

Nicholas Jankowski <nrjank>
Group Member
Sat 16 Apr 2016 05:25:21 AM UTC, comment #10: 

I'd like to look into it at some stage, so I've just downgraded it to minimum priority.  You're welcome to close it, though; I can reopen it if/when I fix it.

Lachlan Andrew <lachlan>
Thu 14 Apr 2016 04:24:52 PM UTC, comment #9: 

Is there a way to proceed forward on this issue report, or should it just be closed?  It seems like we're mostly Matlab compatible with some potential small incompatibilities on the margin because the textscan algorithm is so complicated.

Rik <rik5>
Group administrator
Fri 08 Apr 2016 09:52:02 PM UTC, comment #8: 

That same non-textscan incompatibility exists on Windows as well.
I know because I sometime make counters in .m files that use CR to avoid scrolling up. But those do not work in Matlab - scrolling is unavoidable there.


Ruminating my last comments I think there appears to be some logic in how Matlab processes endofline/whitespace/delimiters. There must be some consistent logic, after all.

First it processes (eliminates) endofline, maybe add it to delimiters.
Next it seems to collapse multiple whitespace.
Then it adds remaining whitespace to the delimiter collection.
Then it splits up the text stream.

Some noise exists as delimiters can also be non-numeric characters cuddling a preceding numeric field.

At least that is how far I got trying to understand textscan's logic.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 08 Apr 2016 06:07:20 AM UTC, comment #7: 

Rik, Philip's comment reminded me why the Octave documentation disagrees with Matlab's: Matlab's was unclear to me, and I describes the best that I could reverse-engineer from Matlab's actual behaviour.


Philip, thanks for those test cases.  I'll look into them, and (maybe) fix them and/or add BIST cases.


You have demonstrated another non-textscan incompatibility.  On Linux at least, Octave gives


>> str = ['1' char(8) '2' char(9) '3' char(10) '4' char(13) '5']
str = 2 3
5
>>
octave:2>


Note the absence of '4'.  Octave's GUI treats char(13) as a true carriage return rather than Matlab's behaviour of treating it as NL+CR.

Lachlan Andrew <lachlan>
Mon 28 Mar 2016 07:18:44 PM UTC, comment #6: 

As an example of the interplay/interference of the various whitespace/delimiter/endofline defaults, consider this (ML r2016a prerel.):

>> str = ['1' char(8) '2' char(9) '3' char(10) '4' char(13) '5']
str =
2        3
4
5
>> uint8 (str)
ans =
   49    8   50    9   51   13   52   10   53
>> C = textscan (str, '%f');
>> C{1}'
ans =
     1     2     3     4
>> C = textscan (str, '%s', 'whitespace', [char(8) char(9) char(10) char(13)], 'delimiter', '');
>> C = C{1}
C =
    '2        3'
    '4…'
>> uint8 (C{1})
ans =
   49    8   50    9   51
>> uint8 (C{2})
ans =
   52   13   53
%% => endofline \n still is a default delimiter

>> C = textscan (str, '%s', 'whitespace', [char(8) char(9) char(10) char(13)], 'delimiter', '', 'endofline', '');
>> C{1}
ans =
    '2        3…'
>> C = C{1}
C =
    '2        3…'
>> uint8(C{1})
ans =
   49    8   50    9   51   10   52   13   53
>> C = textscan (str, '%s', 'endofline', '');
>> C{1}'
ans =
    '1'    '2'    '3…'
>> C = C{1}; uint8 (C{3})
ans =
   51   10   52   13   53
>>


Sigh.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 28 Mar 2016 07:09:44 PM UTC, comment #5: 

@Mike:
Which docs do you refer to?
If it's the Matlab docs, yes they are confusing. I think that adequately reflects the "one tool for all you can think of" that textscan came to be.
As to those docs, I'm even unsure if it is "\b\t" or " \b\t" (note the space) that should be the default whitespace.

Anyway this bug report uncovers a can of worms in a way.
My understanding (as far as I could infer textscan's working) is as follows:

- whitespace (" \b\t") serves as a type of delimiter of which several consecutive ones are always collapsed into one;

- delimiter (default a space) also delimits text fields. Consecutive ones will not be collapsed into one ... unless multipledelimsasone is specified;

- endofline (\r\n, r\r, or \n) are default delimiters and added to the delimiter collection, unless it is set to empty in which case it is treated as whitespace.

So whitespace, delimiters and endofline overlap in some sense.

Things become interesting if whitespace, delimiters and endofline all occur in a file / text srring and in textscan's input arguments.
Although I largely rewrote the textscan.m/strread.m combo, my memory gets a bit confused, sorry. ATM I am morphing strread.m from a self-contained function into one invoking binary textscan as backend and I'm bitten by exactly these "uncertainties" - IOW in hindsight I sometimes doubt if I've been doing the right things all along.

Back to whitespace, Matlab 2016a prerelease does this:

>> str = ['1' char(8) '2' char(9) '3' char(10) '4' char(13) '5']
str =
2        3
4
5
>> uint8 (str)
ans =
   49    8   50    9   51   13   52   10   53
>> C = textscan (str, '%f');
>> C{1}'
ans =
     1     2     3     4
>> C = textscan (str, '%f', 'whitespace', [char(8) char(9) char(10) char(13)]);
>> C{1}'
ans =
     1     2     3     4     5
>> C = textscan (str, '%s');  %% NOTE: now reading strings
>> C{1}'
ans =
    '1'    '2'    '3'    '4…'
>> C = C{1}; uint8 (C{4})
ans =
   52   13   53
>> C = textscan (str, '%s', 'whitespace', [char(8) char(9) char(10) char(13)]);
>> C{1}'
ans =
    '1'    '2'    '3'    '4'    '5'


... showing that whitespace somehow gets "promoted" to delimiters, regardless of numeric or text fields in the input. Only \r (char(13) is treated as special. But maybe that could be due to the default "endofline" character (intractably for us inferred from the input file/string).

It could be that ML's textscan doesn't explicitly turn whitespace into delimiters, but simply stops reading whenever a next character doesn't fit in the numerical scheme, or when reading text, assumes whitespace doesn't occur in text fields.

The rationale may be very logical and straightforward, but the results can be a bit confusing - for me at least :-)

As to the example in comment #3:

>> C = textscan(sprintf ('one\ntwo\nthree\nfour\n'), '%s %s')
C =
    {4x1 cell}    {4x1 cell}
>> C{1}
ans =
    'one'
    'two'
    'three'
    'four'
>> C{2}
ans =
    ''
    ''
    ''
    ''


Rik's first command gives:

>> D = textscan(sprintf('one\ntwo\nthree\nfour\n'), '%s %s', 'Whitespace', ' \b\t')
D =
    {4x1 cell}    {4x1 cell}
>> D{1}
ans =
    'one'
    'two'
    'three'
    'four'
>> D{2}
ans =
    ''
    ''
    ''
    ''


...IOW the same as in comment #3.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 28 Mar 2016 06:12:18 PM UTC, comment #4: 

I only reported this because I was re-working the documentation string and noticed the difference with Matlab.

Does the example in comment #3 yield something different in Matlab?

I tried


textscan(sprintf('one\ntwo\nthree\nfour\n'), '%s %s', "Whitespace", " \b\t")


and it gave the same result as


textscan(sprintf('one\ntwo\nthree\nfour\n'), '%s %s')


so perhaps this is only a documentation issue to fix.

Rik <rik5>
Group administrator
Mon 28 Mar 2016 06:01:57 PM UTC, comment #3: 

Is there a simple motivating example for this that shows the incompatibility? The textscan help is confusing when talking abstractly about whitespace and input rows and delimiters and end of line characters. Something like this maybe?


textscan(sprintf('one\ntwo\nthree\nfour\n'), '%s %s')


Mike Miller <mtmiller>
Group Member
Mon 28 Mar 2016 10:40:14 AM UTC, comment #2: 

Looking at libinterp/oct-stream.cc L.2533, whitespace is declared as " \b\t". So it is somewhere down the road that eol1 ("\r") and eol2 ("\n") appear to be added to the whitespace collection?

Philip Nienhuis <philipnienhuis>
Group Member
Mon 28 Mar 2016 09:32:23 AM UTC, comment #1: 

Thanks, Rik.  I've forgotten the reason for that, but as I recall, it was to make the behaviour match Matlab's in some context.  It may break it in another (such as when the new-line character is defined to be something other than \r, \n or \r\n), but for now there are many bigger bugs to chase...


A related incompatibility is that Matlab only interprets delimiters \b, \n, \r, \t and \\ within a single-quoted string, whereas Octave interprets its full range of escapes, including \a (=7), \f (=12) and \v (=11).  However, I consider Octave's behaviour better and unlikely to cause problems.

Lachlan Andrew <lachlan>
Mon 28 Mar 2016 04:56:30 AM UTC, original submission:  

According to the documentation, the default Whitespace characters are ' \b\t'.  Octave also includes the end-of-line characters '\r\n'.

See http://www.mathworks.com/help/matlab/ref/textscan.html.

Rik <rik5>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by lachlan (Posted a comment)
  • -email is unavailable- added by rik5 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-11-12 nrjank StatusNeed Info Wont Fix
        Open/ClosedOpen Closed
    2016-04-16 lachlan Severity2 - Minor 1 - Wish
        Priority5 - Normal 1 - Later
    2016-04-08 philipnienhuis Operating SystemGNU/Linux Any
    2016-03-28 mtmiller StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code