bugGNU Octave - Bugs: bug #51707, textscan skips chunks of text on...

 
 

bug #51707: textscan skips chunks of text on files with newline line endings

Submitter:  HJW <thrynae>
Submitted:  Wed 09 Aug 2017 06:14:56 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 4.2.0 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 24 Oct 2017 07:17:48 PM UTC, comment #12: 

After applying the textscan fix(es) by Dan in bug #52116 comment 12, this bug seems fixed.

I've compared a dev Octave with and without the patches.
Without patches, L.579 of the file (present as item {561, 1} in data{1} after executing L1-12 of the script in the OP) lacks the "itNav" (see comment #2 below), with the patches it is complete.

Before declaring this fixed I'll wait for confirmation from other people.

(status => Ready for test)
cc'ing Dan as well, I guess he'd like to know :-)

Philip Nienhuis <philipnienhuis>
Group Member
Fri 25 Aug 2017 09:21:18 PM UTC, comment #11: 

That is right.
I was not able to reproduce this on Linux with files with Windows or Linux line endings when reading in binary or text mode.

The only affected case seems to be when reading a file with Linux line endings in text mode on Windows.
Between 1 and 8 characters seem to be missing at random intervals.

Markus Mützel <mmuetzel>
Group administrator
Fri 25 Aug 2017 05:54:58 PM UTC, comment #10: 

Thanks.

This still confuses me, because my understanding of text-mode (rt) is that it translates CR-LF line endings into a single LF when reading a file. If the input file already has LF line endings, then it should have no effect.

Maybe textscan itself is doing something to try to compensate for native line endings, which is breaking down in the case when the file doesn't actually have native line endings?

So to summarize, this bug only occurs on Windows when the input file has Unix LF line endings and the file is opened with mode "rt", right?

Mike Miller <mtmiller>
Group Member
Fri 25 Aug 2017 05:42:38 PM UTC, comment #9: 

Mike, fopening with "r" instead of "rt" does not skip chunks of text with the original file (LF line ending) nor with the modified one with CRLF line. Tested with Octave 4.2.1 on Windows 10.

Markus Mützel <mmuetzel>
Group administrator
Fri 25 Aug 2017 04:05:59 PM UTC, comment #8: 

Markus - does it matter whether the file is fopened with "r" vs "rt"? The original example used "rt", which I thought is supposed to normalize line endings in the input stream. But maybe it also screws up something with the buffering and indexing used in the textscan parser.

Mike Miller <mtmiller>
Group Member
Fri 25 Aug 2017 09:12:08 AM UTC, comment #7: 

I can confirm that I see the same on Windows 10 with Octave 4.2.1 and 4.3.0+ hg_id 35e3d199527c: Some characters are missing.

After changing the line ending to CRLF in an editor, the file is read in without missing characters. So Mike is probably right in comment #4.

Markus Mützel <mmuetzel>
Group administrator
Thu 24 Aug 2017 10:30:40 PM UTC, comment #6: 

I just tested it on another machine (also W10x64), which does have 4.2.1, and the exact same issue arrises. However, all 3 alternatives offered in comment #3 seem to work.

A minor note: the textscan function in Matlab doesn't gobble empty lines by default, so am I correct in thinking that there is an incompatibility as well as a bug?

Anyway, I now have a workaround that should work for my function, so thank you for your help.

HJW <thrynae>
Thu 24 Aug 2017 09:36:01 PM UTC, comment #5: 

Sorry Alternative C in comment #3 gets an exact copy, means 1137 from 1137 lines when writing the field "data" out for comparison.

Kai Torben Ohlhus <siko1056>
Group Member
Thu 24 Aug 2017 09:19:01 PM UTC, comment #4: 

If I'm reading comment #2 correctly, it appears to be eating "itNav" out of the middle of a line of text. I don't see this at all with Octave 4.2.1 or dev on Debian.

HJW - maybe try upgrading to 4.2.1 to see if any fixes were made to textscan.

Someone with access to Windows may want to try to reproduce this to see if this is strictly a Windows library issue, or maybe a line ending conversion issue, that is somehow throwing the buffering off.

Mike Miller <mtmiller>
Group Member
Thu 24 Aug 2017 09:14:54 PM UTC, comment #3: 

Be aware, that `textscan` "eats" empty lines, leading and trainling spaces!  That makes the difference in the line count.  It is not documented, but `textscan` is intended to read formatted lines (not empty data lines).  Three alternatives are given below.  To check the difference I just wrote out what Octave read and compared the files using a diff tool of your choice.


filename='HB_SNG3.html';
if ~exist(filename,'file')
  %download file
  url='http://web.archive.org/web/20170807165834/https://www.bible.com/nl/bible/75/SNG.3.htb';
  urlwrite(url,filename);
end

fid=fopen(filename,'r');
data=textscan(fid,'%s\n','Delimiter','\n');
fclose(fid);
data = data{1};
length(data) # 1110/1137

# Write out what you read to diff the files
unlink([filename "_2"]);
fid=fopen([filename "_2"],'w');
fprintf (fid, "%s\n", data'{:});
fclose(fid);

# Alternative A
data = importdata (filename, "\n");
length(data) # 1115/1137

# Alternative B
data = fileread (filename);
data = strsplit (data, "\n");
length(data) # 1116/1137

# Alternative C from the `publish` function
# https://hg.savannah.gnu.org/hgweb/octave/file/e54e13ee99ce/scripts/general/publish.m#l700
data = cell();
fid = fopen (filename, "r");
i = 0;
do
  data{++i} = fgetl (fid);
until (! ischar (data{i}))
fclose (fid);
data = data(1:end-1);  # No EOL
length(data) # 1110/1137


Kai Torben Ohlhus <siko1056>
Group Member
Thu 24 Aug 2017 06:19:39 PM UTC, comment #2: 

I'm getting 1109 lines (apparently a line end is not read on line 1129)

This is one of the lines (line 579, first is original):
'&amp;quot;features.EventEdit.components.EventEditNav.details&amp;quot;: &amp;quot;Details&amp;quot;'
'&amp;quot;features.EventEdit.components.EventEd.details&amp;quot;: &amp;quot;Details&amp;quot;,'

Using Matlab, I made a list for lines in this file that go wrong. The following list is [line number, length in Matlab, length in Octave]. To account for the missing LF on line 1129, I've added an empty cell at the end.

[260 144 141;428 324 317;481 54 47;526 56 48;579 64 59;606 540 539;664 285 283;698 88 85;740 205 200;788 40 33;885 17 16;1129 11 31;1131 31 39;1132 39 9;1133 9 153;1134 153 7;1136 7 7;1137 7 0]

I can see no system in what bit sequence causes characters to drop or what characters are truncated. I will try uploading a .mat file with the textscan result.

HJW <thrynae>
Thu 24 Aug 2017 05:32:04 PM UTC, comment #1: 

When I run this example, data is a cell array of 1110 lines of text. The original file is 1137 lines, containing 27 blank lines that are skipped by textscan. Other than that I don't see any differences between the input and output. Do you have lines that are truncated? If so, what is their length? Are the truncated lines all the same length?

Mike Miller <mtmiller>
Group Member
Wed 09 Aug 2017 06:14:56 PM UTC, original submission:  

(context: I'm downloading webpages and merging specific content)

When opening the downloaded file in notepad++, all the text is there, but when I use textscan, some text is missing. This seems to be stable behavior (same text is missing each time). This is not the case under Matlab (tested on R2017a and R2012b). My OS is a 64bit Windows 10, and my Octave version is 4.2.0.

As far as I can tell, this is not yet reported. Other files with longer lines (77k on a single line) are not failing. I can't find any systematic reason for this. This occurs in multiple files.

MWE:

filename='HB_SNG3.html';
if ~exist(filename,'file')
  %download file
  url='http://web.archive.org/web/20170807165834/https://www.bible.com/nl/bible/75/SNG.3.htb';
  urlwrite(url,filename);
end
%load file
fid=fopen(filename,'rt','n');
data=textscan(fid,'%s','Delimiter','\n');
fclose(fid);
%convert file to a single long string
data=data{1};data(:,2)={' '};data=data';data=data(:)';data=cell2mat(data);
%remove the parts of the webpage that are not relevant for my goal.
pattern='<div class="book bk';
idx=strfind(data,pattern);
data=data(idx(1):end);
pattern='</div><div class="version-copyright"';
idx=strfind(data,pattern);
if isempty(idx),error('this is possibly a bug'),end
data=data(1:(idx(end)-1));


HJW <thrynae>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by thrynae (Submitted the item)
  • -email is unavailable- added by thrynae
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-10-26 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2017-10-24 philipnienhuis StatusConfirmed Ready For Test
        Carbon-Copy- Added sebald
    2017-08-25 mtmiller Item GroupMatlab Compatibility Incorrect Result
    2017-08-25 mtmiller StatusNeed Info Confirmed
        Summarytextscan seems to skip chunks of text textscan skips chunks of text on files with newline line endings
    2017-08-24 mtmiller StatusNone Need Info
    2017-08-09 thrynae Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code