bugGNU Octave - Bugs: bug #52594, textscan continues past EOL

 
 

bug #52594: textscan continues past EOL

Submitted by:  Ashwin Shrestha <niwhsa>
Submitted on:  Wed 06 Dec 2017 03:26:57 AM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Matlab Compatibility
Status: ConfirmedAssigned to: None
Originator Name: AshOpen/Closed: Open
Release: devOperating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Wed 06 Dec 2017 03:54:26 PM UTC, comment #4:

OK, a number of things there then.

The %d translated to NAN is a consequence of textscan::scan_one() using textscan::read_double() for integer formats (read_double is sort of a custom scanf for the buffered stream):

I suppose the easiest thing is after the double read to check if the value is +/-NaN or +/-Inf. On the other hand, my suspicion is even that wouldn't be compatible with Matlab in odd cases like trying to read, say, 1.23e-04 with %d%s which in Matlab might produce 1 and ".23e-04".

I'm wondering if we could make better use of C++/C formatted read routines somehow rather than replicating C formatted I/O routines. For example, the fscanf() routine is a variable argument routine:

http://www.cplusplus.com/reference/cstdio/fscanf/

But it is presented in terms of the way it might be used at compilation time. However, I think that from a programming perspective there is a way to view that as simply a list of arguments.

On the other hand, the thing that this textscan routine adds is a delimiter. A custom delimiter wouldn't work so well in the approach I described in the previous paragraph. So, maybe rather than handle the delimiter as part of the delimited stream it would be better to first slice and dice the whole line, e.g.,:

1) Search for the next EOL and extract that string
2) Break that line up according to delimiters
3) Loop through each individual hunk scanning with sscanf() while interpretting d, u, s, etc.

The disadvantage of this approach is that the breaking up according to delimiters is somewhat dynamic, i.e., a list of strings. But perhaps if that is done in a smart way the dynamic growth won't be much after the first one or two lines (i.e., the user will have utilized pretty much the full expected number of variables and max data width in the first few lines of data).

Dan Sebald <sebald>
Wed 06 Dec 2017 10:45:48 AM UTC, comment #3:

In Matlab r2017b (and in r2012a albeit with different output format) the first example gives:

and the second example:

Note that %d format specifiers give int32 and NaN doesn't exist in integer math; to get NaNs there %f is required.

AFAICS the "endofline" value is interpreted rather as a special delimiter, it isn't simply added to the delimiter list:

The latter example just shows that a trailing EOL serves to make all output columns equally sized. Otherwise columns are filled only up to the last read value and you can get unequal column lengths in the output.

The Matlab documentation is a lot more clear these days than when I fixed strread and textscan (the .m-file versions) several years ago.

Status -> confirmed
OS -> Any
Release -> dev

Philip Nienhuis <philipnienhuis>
Project Member
Wed 06 Dec 2017 05:40:04 AM UTC, comment #2:

Yes that is correct. I believe that is what Matlab did.

Ashwin Shrestha <niwhsa>
Wed 06 Dec 2017 04:28:18 AM UTC, comment #1:

Thanks for the report. We've been in the process of correcting a few things related to textscan(). To be more accurate, you've mention NAN, but your example is scanning for %s, so I'm assuming you meant empty entry.

Let's consider this example:

Your description suggests the desired result should be:

?? That last entry ans{1,3}(2,1) being empty?

Similarly, for numeric input:

should be

??

Dan Sebald <sebald>
Wed 06 Dec 2017 03:26:57 AM UTC, original submission:

Textscan continues to read past EOL if the format specifier contain more element than the line has.

For example using the code below:

if i read something that has 16 values in a CSV, it would go to the next line and continue reading to fill the remaining "%s" are completely filled, ignoring the EOL at the end of line 1.

This code has been tested in Matlab and behaved differently. It would continue to read until EOL and fill the remaining elements with NaN. This would preserve the location of all the strings in the CSV.

Thanks,
Ashwin

Ashwin Shrestha <niwhsa>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by sebald (Posted a comment)
  • -unavailable- added by niwhsa (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 06 Dec 2017 10:45:48 AM UTCphilipnienhuisStatusNone=>Confirmed
      Release4.2.1=>dev
      Operating SystemMicrosoft Windows=>Any
    Wed 06 Dec 2017 10:45:47 AM UTCphilipnienhuisCategoryNone=>Octave Function

    Back to the top


    Powered by Savane 3.1-cleanup1