bugGNU Octave - Bugs: bug #52594, textscan continues past EOL

 
 

bug #52594: textscan continues past EOL

Submitter:  Ashwin Shrestha <niwhsa>
Submitted:  Wed 06 Dec 2017 03:26:57 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Confirmed Assigned to:  None
Originator Name:  Ash Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 26 Feb 2024 02:55:14 AM UTC, comment #6: 

, better function csvread than textscan

Anonymous
Sun 29 Mar 2020 08:20:52 PM UTC, comment #5: 

Verified that this is still an issue (3/29/2020).  Marking as a dependency of bug #51231 which appears to be the same issue.

Rik <rik5>
Group administrator
Wed 06 Dec 2017 03:54:26 PM UTC, comment #4: 

OK, a number of things there then.

The %d translated to NAN is a consequence of textscan::scan_one() using textscan::read_double() for integer formats (read_double is sort of a custom scanf for the buffered stream):


    double v;    // Matlab docs say 1e30 etc should be valid for %d and
                 // 1000 as a %d8 should be 127, so read as double.
                 // Some loss of precision for d64 and u64.


I suppose the easiest thing is after the double read to check if the value is +/-NaN or +/-Inf.  On the other hand, my suspicion is even that wouldn't be compatible with Matlab in odd cases like trying to read, say, 1.23e-04 with %d%s which in Matlab might produce 1 and ".23e-04".

I'm wondering if we could make better use of C++/C formatted read routines somehow rather than replicating C formatted I/O routines.  For example, the fscanf() routine is a variable argument routine:

http://www.cplusplus.com/reference/cstdio/fscanf/

But it is presented in terms of the way it might be used at compilation time.  However, I think that from a programming perspective there is a way to view that as simply a list of arguments.

On the other hand, the thing that this textscan routine adds is a delimiter.  A custom delimiter wouldn't work so well in the approach I described in the previous paragraph.  So, maybe rather than handle the delimiter as part of the delimited stream it would be better to first slice and dice the whole line, e.g.,:

1) Search for the next EOL and extract that string
2) Break that line up according to delimiters
3) Loop through each individual hunk scanning with sscanf() while interpretting d, u, s, etc.

The disadvantage of this approach is that the breaking up according to delimiters is somewhat dynamic, i.e., a list of strings.  But perhaps if that is done in a smart way the dynamic growth won't be much after the first one or two lines (i.e., the user will have utilized pretty much the full expected number of variables and max data width in the first few lines of data).

Dan Sebald <sebald>
Wed 06 Dec 2017 10:45:48 AM UTC, comment #3: 

In Matlab r2017b (and in r2012a albeit with different output format) the first example gives:

>> format compact   % to get rid of those pesky empty lines
>> a = textscan(sprintf('this,is\r\na,test'), '%s%s%s', 2, 'delimiter',',','EndofLine','\r\n')
a =
  1×3 cell array
    {2×1 cell}    {2×1 cell}    {1×1 cell}
>> a{1}
ans =
  2×1 cell array
    {'this'}
    {'a'   }
>> a{2}
ans =
  2×1 cell array
    {'is'  }
    {'test'}
>> a{3}
ans =
  1×1 cell array
    {0×0 char}


and the second example:

>> x = textscan(sprintf('1,2\r\n3,4'), '%d%d%d', 2, 'delimiter', ',','EndofLine','\r\n')
x =
  1×3 cell array
    {2×1 int32}    {2×1 int32}    {[0]}
>> x{1}
ans =
  2×1 int32 column vector
   1
   3
>> x{2}
ans =
  2×1 int32 column vector
   2
   4
>> x{3}
ans =
  int32
   0


Note that %d format specifiers give int32 and NaN doesn't exist in integer math; to get NaNs there %f is required.

AFAICS the "endofline" value is interpreted rather as a special delimiter, it isn't simply added to the delimiter list:

>> s = textscan ('ab:cd', '%s%s')
s =
    {1x1 cell}    {0x1 cell}
>> s{1}
ans =
    'ab:cd'
>> s{2}
ans =
   Empty cell array: 0-by-1

>> t = textscan ('ab:cd', '%s%s', 'endofline', ':')
t =
    {2x1 cell}    {1x1 cell}
>> t{1}
ans =
    'ab'
    'cd'
>> t{2}
ans =
    {''}

>> t = textscan ('ab:cd:', '%s%s', 'endofline', ':')
t =
    {2x1 cell}    {2x1 cell}
>> t{1}
ans =
    'ab'
    'cd'
>> t{2}
ans =
    ''
    ''


The latter example just shows that a trailing EOL serves to make all output columns equally sized. Otherwise columns are filled only up to the last read value and you can get unequal column lengths in the output.

The Matlab documentation is a lot more clear these days than when I fixed strread and textscan (the .m-file versions) several years ago.

Status -> confirmed
OS -> Any
Release -> dev

Philip Nienhuis <philipnienhuis>
Group Member
Wed 06 Dec 2017 05:40:04 AM UTC, comment #2: 

Yes that is correct. I believe that is what Matlab did.

Ashwin Shrestha <niwhsa>
Wed 06 Dec 2017 04:28:18 AM UTC, comment #1: 

Thanks for the report.  We've been in the process of correcting a few things related to textscan().  To be more accurate, you've mention NAN, but your example is scanning for %s, so I'm assuming you meant empty entry.

Let's consider this example:


octave:3> textscan(sprintf('this,is\r\na,test'), '%s%s%s', 2, 'delimiter', ',','EndofLine','\r\n')
ans =
{
  [1,1] =
  {
    [1,1] = this
    [2,1] = test
  }

  [1,2] =
  {
    [1,1] = is
  }

  [1,3] =
  {
    [1,1] = a
  }

}


Your description suggests the desired result should be:


ans =
{
  [1,1] =
  {
    [1,1] = this
    [2,1] = a
  }

  [1,2] =
  {
    [1,1] = is
    [2,1] = test
  }

  [1,3] =
  {
    [1,1] = [](0x0)
    [2,1] = [](0x0)
  }

}


??  That last entry ans{1,3}(2,1) being empty?

Similarly, for numeric input:


octave:16> x = textscan(sprintf('1,2\r\n3,4'), '%d%d%d', 2, 'delimiter', ',','EndofLine','\r\n')
x =
{
  [1,1] =

    1
    4

  [1,2] = 2
  [1,3] = 3
}


should be


x =
{
  [1,1] =

     1
     3

  [1,2] =

     2
     4

  [1,3] =

     NaN
     NaN

}


??

Dan Sebald <sebald>
Wed 06 Dec 2017 03:26:57 AM UTC, original submission:  

Textscan continues to read past EOL if the format specifier contain more element than the line has.

For example using the code below:

textscan(file_current, '%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s', 2, 'delimiter', ',','EndofLine','\r\n')


if i read something that has 16 values in a CSV, it would go to the next line and continue reading to fill the remaining "%s" are completely filled, ignoring the EOL at the end of line 1.

This code has been tested in Matlab and behaved differently. It would continue to read until EOL and fill the remaining elements with NaN. This would preserve the location of all the strings in the CSV.

Thanks,
Ashwin

Ashwin Shrestha <niwhsa>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by niwhsa (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-03-29 rik5 Dependencies- Depends on bugs #51231
    2017-12-06 philipnienhuis StatusNone Confirmed
        Release4.2.1 dev
        Operating SystemMicrosoft Windows Any
    2017-12-06 philipnienhuis CategoryNone Octave Function

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code