bugGNU Octave - Bugs: bug #51707, textscan skips chunks of text on...

 
 

bug #51707: textscan skips chunks of text on files with newline line endings

Submitted by:  HJW <thrynae>
Submitted on:  Wed 09 Aug 2017 06:14:56 PM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: ConfirmedAssigned to: None
Originator Name: Open/Closed: Open
Release: 4.2.0Operating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Fri 25 Aug 2017 09:21:18 PM UTC, comment #11:

That is right.
I was not able to reproduce this on Linux with files with Windows or Linux line endings when reading in binary or text mode.

The only affected case seems to be when reading a file with Linux line endings in text mode on Windows.
Between 1 and 8 characters seem to be missing at random intervals.

Markus Mützel <mmuetzel>
Project Member
Fri 25 Aug 2017 05:54:58 PM UTC, comment #10:

Thanks.

This still confuses me, because my understanding of text-mode (rt) is that it translates CR-LF line endings into a single LF when reading a file. If the input file already has LF line endings, then it should have no effect.

Maybe textscan itself is doing something to try to compensate for native line endings, which is breaking down in the case when the file doesn't actually have native line endings?

So to summarize, this bug only occurs on Windows when the input file has Unix LF line endings and the file is opened with mode "rt", right?

Mike Miller <mtmiller>
Project Administrator
Fri 25 Aug 2017 05:42:38 PM UTC, comment #9:

Mike, fopening with "r" instead of "rt" does not skip chunks of text with the original file (LF line ending) nor with the modified one with CRLF line. Tested with Octave 4.2.1 on Windows 10.

Markus Mützel <mmuetzel>
Project Member
Fri 25 Aug 2017 04:05:59 PM UTC, comment #8:

Markus - does it matter whether the file is fopened with "r" vs "rt"? The original example used "rt", which I thought is supposed to normalize line endings in the input stream. But maybe it also screws up something with the buffering and indexing used in the textscan parser.

Mike Miller <mtmiller>
Project Administrator
Fri 25 Aug 2017 09:12:08 AM UTC, comment #7:

I can confirm that I see the same on Windows 10 with Octave 4.2.1 and 4.3.0+ hg_id 35e3d199527c: Some characters are missing.

After changing the line ending to CRLF in an editor, the file is read in without missing characters. So Mike is probably right in comment #4.

Markus Mützel <mmuetzel>
Project Member
Thu 24 Aug 2017 10:30:40 PM UTC, comment #6:

I just tested it on another machine (also W10x64), which does have 4.2.1, and the exact same issue arrises. However, all 3 alternatives offered in comment #3 seem to work.

A minor note: the textscan function in Matlab doesn't gobble empty lines by default, so am I correct in thinking that there is an incompatibility as well as a bug?

Anyway, I now have a workaround that should work for my function, so thank you for your help.

HJW <thrynae>
Thu 24 Aug 2017 09:36:01 PM UTC, comment #5:

Sorry Alternative C in comment #3 gets an exact copy, means 1137 from 1137 lines when writing the field "data" out for comparison.

Kai Torben Ohlhus <siko1056>
Project Member
Thu 24 Aug 2017 09:19:01 PM UTC, comment #4:

If I'm reading comment #2 correctly, it appears to be eating "itNav" out of the middle of a line of text. I don't see this at all with Octave 4.2.1 or dev on Debian.

HJW - maybe try upgrading to 4.2.1 to see if any fixes were made to textscan.

Someone with access to Windows may want to try to reproduce this to see if this is strictly a Windows library issue, or maybe a line ending conversion issue, that is somehow throwing the buffering off.

Mike Miller <mtmiller>
Project Administrator
Thu 24 Aug 2017 09:14:54 PM UTC, comment #3:

Be aware, that `textscan` "eats" empty lines, leading and trainling spaces! That makes the difference in the line count. It is not documented, but `textscan` is intended to read formatted lines (not empty data lines). Three alternatives are given below. To check the difference I just wrote out what Octave read and compared the files using a diff tool of your choice.

Kai Torben Ohlhus <siko1056>
Project Member
Thu 24 Aug 2017 06:19:39 PM UTC, comment #2:

I'm getting 1109 lines (apparently a line end is not read on line 1129)

This is one of the lines (line 579, first is original):
'&amp;quot;features.EventEdit.components.EventEditNav.details&amp;quot;: &amp;quot;Details&amp;quot;'
'&amp;quot;features.EventEdit.components.EventEd.details&amp;quot;: &amp;quot;Details&amp;quot;,'

Using Matlab, I made a list for lines in this file that go wrong. The following list is [line number, length in Matlab, length in Octave]. To account for the missing LF on line 1129, I've added an empty cell at the end.

[260 144 141;428 324 317;481 54 47;526 56 48;579 64 59;606 540 539;664 285 283;698 88 85;740 205 200;788 40 33;885 17 16;1129 11 31;1131 31 39;1132 39 9;1133 9 153;1134 153 7;1136 7 7;1137 7 0]

I can see no system in what bit sequence causes characters to drop or what characters are truncated. I will try uploading a .mat file with the textscan result.

HJW <thrynae>
Thu 24 Aug 2017 05:32:04 PM UTC, comment #1:

When I run this example, data is a cell array of 1110 lines of text. The original file is 1137 lines, containing 27 blank lines that are skipped by textscan. Other than that I don't see any differences between the input and output. Do you have lines that are truncated? If so, what is their length? Are the truncated lines all the same length?

Mike Miller <mtmiller>
Project Administrator
Wed 09 Aug 2017 06:14:56 PM UTC, original submission:

(context: I'm downloading webpages and merging specific content)

When opening the downloaded file in notepad++, all the text is there, but when I use textscan, some text is missing. This seems to be stable behavior (same text is missing each time). This is not the case under Matlab (tested on R2017a and R2012b). My OS is a 64bit Windows 10, and my Octave version is 4.2.0.

As far as I can tell, this is not yet reported. Other files with longer lines (77k on a single line) are not failing. I can't find any systematic reason for this. This occurs in multiple files.

MWE:

HJW <thrynae>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mmuetzel (Posted a comment)
  • -unavailable- added by siko1056 (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by thrynae (Submitted the item)
  • -unavailable- added by thrynae
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 25 Aug 2017 05:54:58 PM UTCmtmillerItem GroupMatlab Compatibility=>Incorrect Result
    Fri 25 Aug 2017 04:05:59 PM UTCmtmillerStatusNeed Info=>Confirmed
      Summarytextscan seems to skip chunks of text=>textscan skips chunks of text on files with newline line endings
    Thu 24 Aug 2017 05:32:04 PM UTCmtmillerStatusNone=>Need Info
    Wed 09 Aug 2017 06:14:56 PM UTCthrynaeCarbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup1