bugGNU Octave - Bugs: bug #50619, textscan weird behaviour when...


bug #50619: textscan weird behaviour when reading a csv

Submitted by:  Andrea <rackbox>
Submitted on:  Thu Mar 23 14:33:04 2017  
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: ConfirmedAssigned to: None
Originator Name: Open/Closed: Open
Release: devOperating System: Any

Add a New Comment (Rich MarkupRich Markup):

You are not logged in

Please log in, so followups can be emailed to you.


(Jump to the original submission Jump to the original submission)

Sat Mar 25 23:27:21 2017, comment #14:

Good detective work, Dan. Obviously it looked like some buffer size issue but which buffer and where it is to be found ...

Answer to comment #11 Q1:
< "endofline", "" > is perfectly legal syntax and can be useful when reading from strings (i.e., not a text file, or a file not standard organized along lines like e.g., XML or html)

AFAIU from my textscan.m/strread.m work from years gone by, endofline is whitespace but whitespace is a default delimiter unless "%s" (character string) format specifiers come into play, in which case endofline is moved to the delimiter collection. Or something like that ... the (=TMW's) logic is almost untractable.

Philip Nienhuis <philipnienhuis>
Project Member
Sat Mar 25 23:26:36 2017, comment #13:

To the original poster, the following mods work for me (add new line to the delimiters and increase the buffer size):

It's not a bug fix, but if you keep your string lengths well below 4096, the above should work.

Dan Sebald <sebald>
Sat Mar 25 22:38:58 2017, comment #12:

Here's the comment for the delimited_stream object:

I take it that using the std::stream routines are slow when reading one character at a time, so this class brings in a chunk of data to remain resident and have faster access. I don't know the value of a non-absolute seek and tell that only refer to some given buffer.

Also, what if there is no "last delimiter" in the buffer because the length of the string is greater than the length of the buffer?

Another issue, given chunks of data is the goal, is that the existing code doesn't go about it in a very efficient way. For example, here is the code that reads a string:

For every field read, there is a new standard string "on the stack". Now, "on the stack" probably means that the actual character data isn't on the stack, but the object's reference is (for otherwise the stack could easily overflow if the string is very long). But the point is, for every field there's a fresh std::string that has to be grown. Instead, why not keep a std::string as part of the delimited_stream object? One that grows as large as the largest field it ever sees, and when the delimiter stream is deleted, so is the memory associated with the std::string.

Because, looking at scan_string:

every time through the loop one has to check if the length of std::string is big enough. Why not use a fast character scanning function to determine the length to the next delimiter first, then expand the std::string, then copy data?

There are a lot of FIXMEs in this code, probably because of a not-fully-planned buffer scheme.

Dan Sebald <sebald>
Sat Mar 25 18:46:26 2017, comment #11:

Yes, that is a duplicate.

OK, I'm a little further along in the thought process, and I see now why this strange behavior and the formula for "he [deg]" works, but not "heading [deg]" does not work.

The delimiter buffer size is 80. Count 80 characters out of the below buffer:

and that points 4 characters BEFORE the semicolon of the second line. Snip six characters from the first line and then that first semicolon of the second line is AFTER 80 characters. So, the interaction of that delimited_stream with the end of its buffer and putting stuff back into buffer is where the error lies. And that is why when I put ";\n" in for the delimiter characters, the fields come out right, but the "5.25" is dropped--the delimiter_stream buffer has grabbed a new chunk of data from the std::stream, so what the delimiter stream is attempting to put back, is lost. (? That's the theory anyway.)

By that contorted thinking, lengthening the delimiter_stream buffer from 80 to 100 should fix this particular problem when I use ";\n" delimiters...

And that does, in fact, work:

but of course this isn't a general fix, because the first line could be any length.

OK, so there are two things wrong in the delimiter_stream code

1) The EOL character is not automatically included as a delimiter. I guess it should be in all cases, correct? That is, there isn't some form of syntax for textscan() for which the user can specify EOL is not a delimiter?

2) The buffer doesn't behave correctly at the end, most likely because valuable characters are dropped when delimiter_stream buffer does a refresh_buf():

Dan Sebald <sebald>
Sat Mar 25 15:48:51 2017, comment #10:

bug #50485 may be related

Philip Nienhuis <philipnienhuis>
Project Member
Sat Mar 25 15:44:10 2017, comment #9:

The EOL ("endofline") char / char sequence should automatically be included in the list of delimiters.

Be sure to not forget the \r char.
FYI textscan has an option <"endofline", <EOL char> >

Philip Nienhuis <philipnienhuis>
Project Member
Sat Mar 25 10:06:22 2017, comment #8:

I've tracked this down a bit, so I'm just writing some notes here for reference:

I printed out the "is.tellg()" for:

Here's the result for the test case:

What this is telling me is that the pointer advances as expected with the is.get(). That is, the count of +1, etc. is the number of characters added to the pointer's previous value to get (hopefully) the next pointer address. Except until the last field, the fourteen character "heading [deg]". In that case the pointer makes some odd jump, going backward (!), as we'd expect 7867381 + 14 = 7867395.

This stream:

isn't behaving nicely. The max_lookahead is 3, and the buf_size is 80. (I recall somewhere else there being a buffer size of 4096...but don't take that as being of some significance, as I don't quite understand the implication of buf_size.)

I can see what is wrong. See the delims passed into this delimited stream? Later in testing the ch = is.get() character with is_delim(ch), it's those delims (a C++ std::string) that are looked for. Going into that is() instantiation is only ";". So this delimited_stream doesn't recognize the new-line character as a delimiter. It's just another character, so the delimiter stream keeps reading until hitting another ";" character. There must be some odd relationship between line length and buf_size that causes the pointer to advance to some strange place in the next line for the next textscan(). Note: I think that even though the col_headers looks to be reading the "header [deg]" properly, I think it's not and somehow the new-line character-plus (i.e., "\n5.2500000000000") is dropped somewhere along the way when converted to cell-string.

So, as a little test, let's try putting ";\n" in for the delimiters in the test code, i.e., textscan(file, formatSpec, 1, 'Delimiter', ";\n"):

OK, now things look proper, i.e., 7866245 + 14 = 7866259. Unfortunately, the result still isn't quite correct:

Better! But the first entry isn't 5.25. Again, some strange interaction with the new-line character and placing it back into the stream, maybe?

That's where I am. On the trail, I think, but only close so far.

Dan Sebald <sebald>
Fri Mar 24 21:23:38 2017, comment #7:

The file does look perfectly suited to dlmread with a skip of the first line.

Does this work?

Rik <rik5>
Project Administrator
Fri Mar 24 12:23:47 2017, comment #6:

100,000's of lines is no big deal. 10e5 * 100 chars/line ~ 10 MB, that's still fairly tiny considering that char arrays can be 2 GB (with 64-bit Octave).
As soon as you hit 1 GB (a factor of 100 away) you should become a little wary as data are copied over internally into potentially much larger data objects. But do use 64-bit Octave.

Chances are that textscan() won't be fixed very soon. So you need a workaround.
I'd advise trying with dlmread, csvread or csv2cell anyway; in top (*nix) or Task Manager (Windows) you can follow how much memory is consumed. You can read big datafiles in big chunks at a time.

Philip Nienhuis <philipnienhuis>
Project Member
Fri Mar 24 08:29:20 2017, comment #5:

It is a log file whose production I can't control, I have to use it as-is. It can contain hundreds of thousands of lines.

Andrea <rackbox>
Fri Mar 24 08:25:34 2017, comment #4:

Define "huge".

I do see that it spoils huge amounts of disk space because of possibly unneeded output precision, i.e. 16-17 digits.
Cutting down there would make the file maybe half as huge :-)

Philip Nienhuis <philipnienhuis>
Project Member
Fri Mar 24 08:14:46 2017, comment #3:

I was using textscan because the file I want to read is huge, and I need to read it line by line...

Andrea <rackbox>
Thu Mar 23 23:43:22 2017, comment #2:

Confirmed on Windows 7 with Octave-4.3.0+
An intriguing bug.
I checked with Matlab r2017a but that does not portray this bug.

AFAICS, by starting with "h" for the last field on line 1 and adding characters, trouble starts as soon as the "[" (in fact, any character) is added after the trailing space after "heading". Then textscan starts reading in the middle of the "44.0000000000000" field (correctly at the faulty pos1 position) and gets out of sync, evidenced by the NaN (= missing value for empty field).
Adding a <'whitespace', ''> arg in the textscan call makes no difference.
Textcan correctly reads the last field on line 1 in all cases. So it is the file pointer computation that gets confused.

As a workaround you can also try to read the csv-file using csv2cell in the io package (I just tried). As of io-2.4.6 (now silently updated to 2.4.7) csv2cell accepts a spreadsheet-style address argument, see "help csv2cell".

Philip Nienhuis <philipnienhuis>
Project Member
Thu Mar 23 17:09:35 2017, comment #1:

I've confirmed this behavior in linux with the development version. The pertinent code resides in libinterp/corefun/oct-stream.cc, probably textscan::do_scan().

The only other detail I see is that ";he [deg]" also works, but not ";hea [deg]". With ";he [deg]", the total length of the first line is 64 characters, including the carriage-return character. I presume what is happening is the first textscan() is searching for the next field after the first group of reads and advances the file pointer. But the REPEAT field is present, "1", so the algorithm should stop after that seventh string field.

Dan Sebald <sebald>
Thu Mar 23 14:33:04 2017, original submission:

If I process this .csv file

with the following code

the output of the second textscan call is wrong. In particular, the first element (which should be 5.25) is 0. In this case the value of 'pos1' is 96 (why?).

If I change the .csv file substituting the word "heading" with "h" and leave everything else unchanged, the output is correct. In this case the value of 'pos1' is 62 (correct).

This behaviour is not reproducible in Matlab 2010, which produces the correct output in both cases.

Andrea <rackbox>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

No files currently attached


Depends on the following items: None found

   bug dependencies.


Carbon-Copy List
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by sebald (Posted a comment)
  • -unavailable- added by rackbox (Submitted the item)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue Apr 4 16:54:19 2017mmuetzelDependencies-=>bugs #50717 is dependent
    Thu Mar 23 23:43:22 2017philipnienhuisStatusNone=>Confirmed
      Operating SystemMicrosoft Windows=>Any

    Back to the top

    Powered by Savane 3.1-cleanup1