bugGNU Octave - Bugs: bug #41579, textscan: file offset after...

 
 

bug #41579: textscan: file offset after partial read differs from Matlab

Submitter:  Jérôme <jeromegnu>
Submitted:  Thu 13 Feb 2014 02:55:24 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Need Info Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 11 Aug 2023 09:23:15 PM UTC, comment #17: 

it's been quite some time since this was looked at, and the last comment mentioned a number of other bugs to textscan to be tackled first.  It would be useful if someone could identify an updated simple test to determine if this bug is still present for current versions of octave and matlab.

Nicholas Jankowski <nrjank>
Group Member
Tue 24 Oct 2017 10:51:12 PM UTC, comment #16: 

"Maybe the only difference in the examples I gave is that regular delimiters after the last read field are skipped but just not endofline. But when reading continues, endofline seems to be treated as leading whitespace (I have to confirm the latter)."

That's what I observed too, this was my goal for the modification.  That is, the EOL-skip should be done prior to the scan action, not after.  I believe there is currently a skip_whitespace prior to the scan action that does this.  And that's the reason I had to remove the skip_whitespace() from the skip_delim(), because otherwise it would remove the EOL.


"And then there is the thing that whitespace is treated differently when reading text, than when reading numeric data."

I believe that is the case.  But I think handling such a thing inside skip_whitespace() and skip_delim() makes for lots of conditional tests.  On the other hand, a separate skip_eol() means that context can be kept out of the lower-level routines.

Anyway, there is another collection of textscan() bugs being sorted through at the moment, so we'll let the dust settle there and come back to this after some thought.

Dan Sebald <sebald>
Tue 24 Oct 2017 07:53:29 PM UTC, comment #15: 

Dan,
Before venturing further I think I might try similar tricks with whitespace and EOL settings in Matlab as I did in comment #12, with text and numeric data. But I'm unsure if I have time for that the next days.

FYI, I took up textscan.m / textread.m / strread.m some years ago and tried to make them as much Matlab-compatible as possible, based on a lot of observations of Matlab behavior at the time by me and several others, and given (Octave-)strread's limitations.
The result is that in Octave's still existing strread.m, which used to be called by (now removed) textscan.m, it works as follows:

1. endofline character is first added to the delimiter collection

2. ...and removed from the whitespace collection if present there (strread.m by default contains " \r\n\b\t").

Now, endofline may not be a "regular" delimiter but Matlab surely treats it as such when reading multiple lines. Only if set to "" (empty) it is ignored, but then again not when reading numeric data.
Maybe the only difference in the examples I gave is that regular delimiters after the last read field are skipped but just not endofline. But when reading continues, endofline seems to be treated as leading whitespace (I have to confirm the latter).

And then there is the thing that whitespace is treated differently when reading text, than when reading numeric data.
Multiple whitespace incl. endofline is usually collapsed, multiple delimiters not. AFAIK when MultipleDelimsAsOne is specified, endofline preceding or following a delimiter is is ignored in the collapse action (I have to verify that too).

So endofline may be treated as whitespace and/or as delimiter, depending on context.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 24 Oct 2017 05:19:20 PM UTC, comment #14: 

Diff file attached: yanking EOL treatment that behaves similar to original ML example.

(file #42248)

Dan Sebald <sebald>
Tue 24 Oct 2017 05:16:22 PM UTC, comment #13: 

OK, thanks.  The take-away from these two examples is that ML does not consider EOLs to be delimiters.  (If there is one historical rule I've noticed about ML is that they go for simplicity of code and let idiosyncrasies be what they are.)

However, EOLs are considered delimiters for Octave, and it looks like they are also considered white-spaces.  That's a significant deviation from ML.  Octave has gone to great lengths to nuance the treatment of EOLs, so much so that I'm wondering what the developer had in mind and whether ML-compatibility was even a concern.

I'm attaching a diff file in which I've basically gutted the skip_delim() routine and the behavior is the same as the original example for ML.  No need to test this, it should be clear that basically I've removed all EOL treatment.  However, I did have to remove this line too:


//    int c1 = skip_whitespace (is, true);  // 'true': stop once EOL is read


which generally speaking is probably too much (but the test.txt file given is nice an clean as far as extraneous white space)--but again, I think there too the EOL is considered whitespace when it shouldn't be.

I think the design "flaw" (if one wants to call it that) is intermingling EOL treatment with the delimiter treatment and the whitespace treatment.  Instead, it would probably make more sense to be more OOP-mindful and pull out the EOL code and place it into its own "skip_eol()" method.  Then we can strategically place

skip_whitespace()
skip_eol()
skip_delim()

in various locations and various combinations to match ML behavior.  That's shaking the tree a bit, but hopefully the greater granularity makes the code easier to debug.

Do we want to go that route?

Dan Sebald <sebald>
Tue 24 Oct 2017 07:04:25 AM UTC, comment #12: 

Dan,

I think we run in Matlab's idiosyncrasies here.
The example below on this file:

123;456;789
012;345;678
901;234;567


(note: with CRLF EOLs)
shows that Matlab (r017b), when reading a specified nr. of fields, advances behind delimiters but before EOLs ater reading the last field as specified in the format repeat count.


>> format compact
>> fid = fopen ('txtscn.txt')
fid =
     3
>> C = textscan (fid, '%f %f', 1, 'delimiter', ';')
C =
  1×2 cell array
    {[123]}    {[456]}
>> pos = ftell (fid)
pos =
     8
>> txt = fread (fid, Inf, 'char=>char')'
txt =
    '789
     012;345;678
     901;234;567
     '
>> uint8 (txt)
ans =
  1×31 uint8 row vector
  Columns 1 through 18
   55   56   57   13   10   48   49   50   59   51   52   53   59   54   55   56   13   10
  Columns 19 through 31
   57   48   49   59   50   51   52   59   53   54   55   13   10
>> fseek (fid, 0, 'bof')
ans =
     0
>> C = textscan (fid, '%f %f %f', 1, 'delimiter', ';')
C =
  1×3 cell array
    {[123]}    {[456]}    {[789]}
>> pos = ftell (fid)
pos =
    11
>> txt = fread (fid, Inf, 'char=>char')'
txt =
    '
     012;345;678
     901;234;567
     '
>> uint8 (txt)
ans =
  1×28 uint8 row vector
  Columns 1 through 18
   13   10   48   49   50   59   51   52   53   59   54   55   56   13   10   57   48   49
  Columns 19 through 28
   59   50   51   52   59   53   54   55   13   10
>>


Philip Nienhuis <philipnienhuis>
Group Member
Mon 23 Oct 2017 06:18:57 PM UTC, comment #11: 

I took a quick look at this issue.  I've found the line of code responsible for advancing the filepointer past the EOL when, in the example, the first group of string variables are read.  In the diff hunk below, I put an "#if 0" around the line in question:


diff --git a/libinterp/corefcn/oct-stream.cc b/libinterp/corefcn/oct-stream.cc
--- a/libinterp/corefcn/oct-stream.cc
+++ b/libinterp/corefcn/oct-stream.cc
@@ -1603,6 +1603,7 @@ namespace octave
     int len = out.length ();
     int used = 0;
     int ch;
+fprintf(stderr, "GETLINE\n");
     while ((ch = get_undelim ()) != delim
            && ch != std::istream::traits_type::eof ())
       {
@@ -2685,6 +2686,7 @@ namespace octave
               }

             row_idx(0) = row;
+fprintf(stderr, "CALL read_format_once\n");
             err = read_format_once (is, fmt_list, out, row_idx, done_after);

             if ((err & ~1) > 0 || ! is || (lines >= ntimes && ntimes > -1))
@@ -3505,12 +3507,14 @@ namespace octave
         elem = fmt_list.next ();
         char *pos = is.tellg ();

+#if 0
         // FIXME: these conversions "ignore delimiters".  Should they include
         // delimiters at the start of the conversion, or can those be skipped?
         if (elem->type != textscan_format_elt::literal_conversion
             // && elem->type != '[' && elem->type != '^' && elem->type != 'c'
             )
           skip_delim (is);
+#endif

         if (is.eof ())
           {


The skip_delim() is being called after the read of the variable.  (And I believe there are plenty of delimiter skipping actions before reading a variable as well.)  By my way of thinking, the ML behavior doesn't do anything after a read, and commenting out the line of code as above results in Octave behaving like ML for this example.

Looking closely at skip_delim(),


         if (is_delim (c1) || c1 == eol1 || c1 == eol2)


the EOL characters are considered delimiters in this context.  I guess that is fine.

I'm not so sure it is as simple as dropping the lines associated with the FIXME above, because obviously they were there and modified for some reason and I am wondering what breaks by dropping that line.  Also, I think we need to cover all cases here and consider what happens when, say, the textscan() command asks for fewer columns than what exists in the file.  Say, for example, there are five columns in the data file and textscan only asks to read three of them:

v1 v2 v3 V4 V5

Do the fourth and fifth columns then get treated as the first and second variables of the next line?  That is, once in data scanning mode (as opposed to headerline skipping) is the file treated like EOL characters are irrelevant and just another white space or delimiter?  Consider adding Rik to the list for his attention to detail.

Dan Sebald <sebald>
Mon 28 Mar 2016 06:51:45 PM UTC, comment #10: 

I don't work with Matlab/Octave anymore.

We have a Matlab licence at the office so if you have a .m file with all the tests you need, maybe I can execute it and post the results here.

Jérôme <jeromegnu>
Mon 28 Mar 2016 05:38:33 PM UTC, comment #9: 

And if I add


fseek(fid, 40);


after the first textscan, then the script works without the Octave check.

So the core issue seems to be the file offset after textscan returns, not the interpretation of the headerlines parameter.

Retitling appropriately.

Some more examples in Matlab showing what the file offset is after a partial read would be helpful here.

Mike Miller <mtmiller>
Group Member
Mon 28 Mar 2016 05:30:56 PM UTC, comment #8: 

Example works again now on the default branch with the new textscan, but still showing this incompatibility with where the file position is after the first call to textscan.

Mike Miller <mtmiller>
Group Member
Fri 25 Mar 2016 11:24:01 PM UTC, comment #7: 

I filed a new bug for the new textscan reading the entire input file even if it is given a count of parameters to read, as in this example.

See bug #47537.

Mike Miller <mtmiller>
Group Member
Fri 25 Mar 2016 08:20:25 PM UTC, comment #6: 

Now that textscan is a built-in function, the example script in this bug report no longer works. Can someone please test and see what needs to be changed in the test script or what is not working compatibly with Matlab now?


>> fid = fopen ("test.txt");
>> t = textscan (fid, "%s%s%s%s%s", 1, "delimiter", "\t", "collectoutput", 1);
>> ftell (fid)
ans =  79
>> fclose (fid);
t{1}
ans =
{
  [1,1] = year
  [1,2] = day in year
  [1,3] = day in month
  [1,4] = month
  [1,5] = hour
}


Mike Miller <mtmiller>
Group Member
Fri 14 Feb 2014 10:50:24 PM UTC, comment #5: 

Thanks for asking "upstream".

In the mean time, I think on the Octave side this cannot be fixed to be ML-'compatible' - at present, that is.
Octave's textscan.m is more or less line-oriented, in contrast to Matlab's which acts more like a character-oriented binary file reader. Fiddling with textscan.m's whitespace/delimiter/endofline defaults and with the way they're processed won't help much I'm afraid, as the real issue is the way Octave's textscan.m reads from file (i.e., before it even does the first analyses).

For some years now there have been plans to create a binary textscan(); JWE has even made a rough skeleton (IIRC 2 or 3 years ago), but no one has had time and priority yet to finish it.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 14 Feb 2014 03:48:45 PM UTC, comment #4: 

Indeed, the \n\t workaround does the trick for me.

Anyway, here's the question:

http://www.mathworks.fr/matlabcentral/answers/116179-textscan-not-eating-eol

I should be notified and post here if I get a useful answer.

Thank you Philip.

Jérôme <jeromegnu>
Thu 13 Feb 2014 10:43:19 PM UTC, comment #3: 

Here's a thread on this subject from comp-soft-sys.matlab (tru Google Groups):

https://groups.google.com/forum/#!topic/comp.soft-sys.matlab/d03F4NsdA40

...so it seems this (IMO ML inconsistency) has been hit before.

Maybe Mathworks is prepared to fix this behavior, so do report it to them.
Otherwise we might have to fix the incompatibility in Octave.

Jérôme, I'll await your response.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 13 Feb 2014 10:22:25 PM UTC, comment #2: 

I can confirm the difference in behavior between Octave 3.8.0 & Matlab r2014a-prerelease on Windows.

I suspect un-/obscurely documented, or at least misunderstood, Matlab behavior, i.e., in how far the EOL char is included in delimiters when reading from file (see below for more on this).
Yet I do agree that Matlab's behavior is at least unexpected (to the unwary user).

Some analysis:
After executing the first textscan command from test.m, 'ftell (fid)' on Octave gets me 41 while on Matlab it is 40.
Counting the characters (incl. tabs) on the first line of test.txt gives 40; so it seems Matlab doesn't "eat" the EOL char. IOW, on the next call to textscan it seems to include that EOL in the 'headerlines' count rather than the next EOL (on L.2).
Now, the ML docs say that "... headerlines are skipped, incl. the remainder of the current line ...". But it is a bit ambiguous to me whether this would also apply to the remaining sole EOL on line 1.

In Matlab, specifying '\t\n' as delimiter value for all textscan calls gets ftell(fid) to report 41 after the first textscan call in test.m, and after the second call with "'headerlines', 1" Matlab correctly reports the output as a {2x5 double}.
Octave's behavior isn't affected by this; so there's a workaround for you to have Octave and Matlab behave identically.

At first sight I'd say this is a Matlab bug. When a "headerlines" parameter is specified it looks obvious to me that EOLs should be eaten by it, including EOL in "the remainder of the current line" as their docs state. But I'm not sure; so please report it to the Mathworks and report back their answer here.
Or for faster answers, ask in usenet NG comp.soft-sys.matlab (hint: don't mention Octave ;-) ); the resident Mathworks-employed guru there is often quick to reply.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 13 Feb 2014 05:23:20 PM UTC, comment #1: 

Hm, looks for me the same (octave 3.8 and matlab 2013a, 2014a)

>> ttttt

    'year'    'day in year'    'day in month'    'month'    'hour'

        1900           1           1           1           1
        1900           1           1           1           2

>> whos

  Name         Size            Bytes  Class     Attributes

  ans          1x1                 8  double             
  fid          1x1                 8  double             
  headers      1x5               632  cell               
  nb_col       1x1                 8  double             
  t            1x1               192  cell               
  values       2x5                80  double             

>> headers{1,1}


ans =

year

>> headers{1,3}


ans =

day in month

>> size(headers)


ans =

     1     5






octave:1> ttttt

{
  [1,1] = year
  [1,2] = day in year
  [1,3] = day in month
  [1,4] = month
  [1,5] = hour
}
                  1900                     1                     1                     1                     1
                  1900                     1                     1                     1                     2
octave:2> whos
Variables in the current scope:

   Attr Name         Size                     Bytes  Class
   ==== ====         ====                     =====  =====
        ans          1x1                          8  double
        fid          1x1                          8  double
        headers      1x5                         36  cell
        nb_col       1x1                          8  double
        t            1x1                         80  cell
        values       2x5                         80  double

Total is 19 elements using 220 bytes

octave:3> headers {1,1}
ans = year
octave:4> headers {1,3}
ans = day in month
octave:7> size(headers)
ans =

   1   5

Markus Bergholz <markuman>
Thu 13 Feb 2014 02:55:24 PM UTC, original submission:  

Hi.

It seems the HeaderLines parameter in textscan behaves differently between Matlab end Octave.

In the attached script, I test whether I'm using ML or Octave to get the same result.

My algo does the following :

- Read first line as headers

- Read all lines from the third, which are value. The second header line is ignored.

In ML, I need to enter HeaderLines=2, as if the pointer was still at the end of line 1. On Octave, I need to enter HeaderLines=1, as if the pointer was already at the beginning of line 2.

Hopefully, the example will make this clearer.

Here's the result on Octave 3.8 and ML 8.2.0.701 (R2013b) :

octave:15> test

{
  [1,1] = year
  [1,2] = day in year
  [1,3] = day in month
  [1,4] = month
  [1,5] = hour
}
   1900      1      1      1      1
   1900      1      1      1      2



>> test

    'year'    'day in year'    'day in month'    'month'    'hour'

        1900           1           1           1           1
        1900           1           1           1           2


They are the same, thanks to the test in the code. The HeaderLines parameter differ.

As a sidenote, the behaviour un Octave seems more logical to me... Should I submit the bug to ML ?

Jérôme <jeromegnu>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #30535:  test.m added by jeromegnu (467B - text/plain)
file #30536:  test.txt added by jeromegnu (79B - text/plain)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by markuman (Posted a comment)
  • -email is unavailable- added by jeromegnu (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-08-11 nrjank Operating SystemGNU/Linux Any
    2017-10-24 sebald Attached File- Added bug41579-major_textscan_skipdelim_mods-djs2017oct24.diff, #42248
    2016-03-28 mtmiller SummaryHeaderLines parameter in textscan different between Octave and ML textscan: file offset after partial read differs from Matlab
    2016-03-25 mtmiller Dependencies- Depends on bugs #47537
    2016-03-25 mtmiller Release3.8.0 dev
    2014-02-13 philipnienhuis StatusNone Need Info
    2014-02-13 jeromegnu Attached File- Added test.m, #30535
        Attached File- Added test.txt, #30536

    Back to the top

    Powered by Savane 3.13-caa5.
    Corresponding source code