bugGNU Octave - Bugs: bug #38003, textscan() "endofline"...

 
 

bug #38003: textscan() "endofline" compatibility

Submitter:  Ben Abbott <bpabbott>
Submitted:  Sun 30 Dec 2012 10:13:05 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Invalid / Not an Octave Bug Assigned to:  None
Originator Name:  Ben Abbott Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 01 Jan 2013 05:44:59 PM UTC, comment #2: 

The possible EOL characters are nicely itemized on Wikipedia.

http://en.wikipedia.org/wiki/Newline#Representations

The "\n\r" is certainly not a common one.  The crux of my report was my understanding that Matlab's implementation respects all line endings simultaneously  ... which isn't clear from the documentation, as it may be interpreted to mean that a single line ending is used for each file and is detected automatically.

Essentially, I was thinking that Matlab's implementation was based on regexp (), and that they had implemented an effective and efficient way of splitting a text file into lines,


lines = regexp (text, '\r\n|\n\r|\r|\n', 'split');


However, a quick test shows that my understanding was wrong.


eol = '\r\n|\n\r|\r|\n';
a = textscan (sprintf ('A\r\nB\n\rC\rD\nE'), '%s', 'endofline', eol)
Error using textscan
End of line character must be no more than a single character or \r\n


Further the "\r\n" variant also throws an error.


 a = textscan (sprintf ('A\r\nB\n\rC\rD\nE'), '%s', 'endofline', '\n\r')
Error using textscan
End of line character must be no more than a single character or \r\n


On a related not, while poking about on the internet, I came across a reference to an undocumented functions as the backend to Matlab's strread() and textread().  When passing the 'endofline' as '\r\n|\n\r|\r|\n' to strread() or textread(), I get the same error as for textscan().  So it maybe that dataread() is also the backend for textscan()?


help dataread
 dataread Read formatted data from string or file.
    dataread is the backend support function for TEXTREAD and STRREAD.

    See also strread, textread.


In any event, I'm marking this as "invalid".


Ben Abbott <bpabbott>
Group Member
Tue 01 Jan 2013 11:15:45 AM UTC, comment #1: 

First of all, is this really a bug?

(1) At the time I added endofline processing to textscan/textread, the ML docs only mentioned "\n", "\r\n" and "\r" EOLs, plus that a default value for endofline param is to be determined from the file at hand.
==> there's at least a bit of undocumented ML behavior.

(2) In real life I've never seen "\n\r" EOLs.
I did encounter "\n\r" sequences but those usually were meant as special encodings, not EOL sequences.

(3) I wonder if your example is really valid as endofline stuff is only..., well, "supported"? .. when reading real files, not character strings.

FYI: strread.m (textscan's work horse) does:

octave:4> a = strread (sprintf ('A\r\nB\n\rC\rD\nE'), '%s')
a =
{
  [1,1] = A
  [2,1] = B
  [3,1] = C
  [4,1] = D
  [5,1] = E
}
octave:5>


Note that it may well be that strread.m does it OK just "by accident" :-)

Philip Nienhuis <philipnienhuis>
Group Member
Sun 30 Dec 2012 10:13:05 PM UTC, original submission:  

While thinking about the textscan()/strread() bugs in the tracker, it occurred to me that the Matlab variants may rely heavily on regexp.  To check I ran a test on how Matlab's textscan() handles "endofline" characters.

It appears that Matlab's default value for "endofline" is '\r\n|\n\r|\r|\n'.


a = textscan (sprintf ('A\r\nB\n\rC\rD\nE'), '%s')
a = {5x1 cell}
a{:}
ans =
    'A'
    'B'
    'C'
    'D'
    'E'


My tip is;


changeset:   15866:c1c6502fe52b
tag:         tip
user:        Rik <rik@octave.org>
date:        Sat Dec 29 08:15:04 2012 -0800
summary:     oct-rand.cc: Silence compiler warning about possible uninitialized variable.


And with the tip above, the default branch gives me


a = textscan (sprintf ('A\r\nB\n\rC\rD\nE'), '%s')
a =
{
  [1,1] =
  {
    [1,1] = A
    [2,1] = B
D   [3,1] = C
    [4,1] = E
  }
}


Which implies the Octave implementation does not recognize "\r" as an "endofline" character.

Ben Abbott <bpabbott>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by philipnienhuis (Posted a comment)
  • -email is unavailable- added by bpabbott (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-06-24 rik5 Open/ClosedOpen Closed
    2013-01-01 bpabbott StatusNone Invalid / Not an Octave Bug
        Originator Name Ben Abbott

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code