bugGNU Octave - Bugs: bug #52116, Textscan filepointer bug is still...

 
 

bug #52116: Textscan filepointer bug is still present in Windows build

Submitted by:  None
Submitted on:  Tue 26 Sep 2017 07:05:24 AM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: NoneAssigned to: None
Originator Name: Cees MeijerOriginator Email: -unavailable-
Open/Closed: OpenRelease: 4.2.1
Operating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 27 Sep 2017 04:10:00 PM UTC, comment #12:

Yes, I forgot; patch attached.

I'm also attaching a non-bug patch to reuse some code of one delimited_stream::delimited_stream constructor from the other and remove a FIXME. (I think that's what that FIXME is about.) However, I don't know what script to use to test that it works.

(file #41900, file #41901)

Dan Sebald <sebald>
Wed 27 Sep 2017 10:28:44 AM UTC, comment #11:

Dan: I don't see a patch attached to this report. Did you forget to attach it before submitting comment #10?

John W. Eaton <jwe>
Project Administrator
Wed 27 Sep 2017 09:52:21 AM UTC, comment #10:

I may have found the bug, please test the attached patch.

This very subtle change takes some thought. As I mentioned, this delimited stream has to "put back" data into the conventional input stream, i.e., readjust the pointer, when it is done and there is unused data in its local buffer. Currently that pointer is recorded from the input stream every time some data is read from the input stream. But that is not correct because the input stream position is far ahead of the current delimited buffer's pointer, on average.

As an example, let's say the BUF_SIZE is N characters, something much larger than the data expected to be read, e.g. 1024. buf_in_file starts at 0, then we read 8 bytes, etc. The sequence of events would be

At this point, we restore the stream position after having read only 26 bytes. But the buf_in_file has a value of N+8+6+9, a value much greater than 26. So that is where the error is.

Note that if we pick N larger than the amount of data in the file, we luck out because the input stream is only read the first time and successive times eof() tests true, i.e.,

So, the change should fix the bug. But there are probably still other bugs. One thing is that because of the buffering of data there is an inherent assumption that a float value in ASCII form will be less than 80 characters, but someone could have test representations of numbers with more characters even though they wouldn't fit into a double float. In order to handle such a thing in buffered data, the low-level deciphering of floats needs to be done at the same level. That is we can't run scanf on something like

where that first portion is length N=16 buffer and there is still a portion of ascii characters to come in from the stream. I'm not going to concern myself with that though. This delimited stream is much too complex to experiment with.

Dan Sebald <sebald>
Tue 26 Sep 2017 10:09:06 PM UTC, comment #9:

I'm just adding a note here to diagnose the code.

I've printed out buf_size:

octave:4> A = textscan(fid,'%f %f %f %f %f %f %f %f %f %f %f %f %f',4,"endofline","\r\n");
BUF_SIZE=320

By my estimate there are 836 non-EOL characters in the data file. The file size is 855 characters. There look to be 10 lines in the file from an editor. That works out to two characters for EOL; seeing this is windows, no surprise. So starting from the end of line four in the data file, look 320 characters out and that puts us at

That's not 99.88, the first read value of lastline=fgetl(fid), but it is close. I'm probably not accounting for something, but the point is that the lost data has to do with the buffer size.

So, the obvious thought-o here is that specifying the number of lines to read, "4", means that the size of data is limited to 4*80, or some other value other than 80, without loss of generality. That obviously isn't true.

Dan Sebald <sebald>
Tue 26 Sep 2017 06:50:15 PM UTC, comment #8:

Thanks C.M. I've run the script on Linux and verified the bug--the lastline variable beginning with 99.88. So, this is not Windows OS specific.

This looks like a duplicate bug. Replacing the textscan() line with

gives me the correct result. See bug report

https://savannah.gnu.org/bugs/?50619#comment13

This textscan() looks like a high priority bug. It needs a pretty significant restructuring because of the way it descends into a subroutine that reads one line at a time and tries to put stuff back into the buffer. Information about what has been read is lost.

Dan Sebald <sebald>
Tue 26 Sep 2017 06:10:24 PM UTC, comment #7:

I think I succeeded in including the files this time. But if not, just let me know and I'll try again.

Cees Meijer <cees_meijer>
Tue 26 Sep 2017 04:12:33 PM UTC, comment #6:

you are right - I didn't look at the year, just month/day

John Donoghue <lostbard>
Project Member
Tue 26 Sep 2017 03:32:53 PM UTC, comment #5:

John - I don't think that's right, bug #47671 was fixed before the 4.2.0 release, so it should have been fixed in both 4.2.0 and 4.2.1. But there are definitely other open bugs with textscan, including some that are Windows specific.

Cees - please provide an example of a file and a textscan call and show the problem you are having. You said you added something in comment #2, but I don't see any attachments here.

Mike Miller <mtmiller>
Project Administrator
Tue 26 Sep 2017 02:42:44 PM UTC, comment #4:

Yes, I just followed the download link on the home page, and this leads to the ftp. So that could be the cause. But are there newer binaries available then ?

Cees Meijer <cees_meijer>
Tue 26 Sep 2017 12:06:23 PM UTC, comment #3:

Are you using the binaries from https://ftp.gnu.org/gnu/octave/windows/ ?

If so, they were created before Bug # 47671 was fixed

John Donoghue <lostbard>
Project Member
Tue 26 Sep 2017 10:34:39 AM UTC, comment #2:

I've added a sample file that comes close to the data I'm using, and an extract from my program that shows the problem.
The 'read_file' function should return the first line in the file, a matrix with all values from the next 4 lines, and finally the sixth line in 'lastline'. And there it fails, because lastline will actually start reading from somewhere halfway line 8.

Cees Meijer <cees_meijer>
Tue 26 Sep 2017 08:06:38 AM UTC, comment #1:

One of these bug reports might be relevant:

https://savannah.gnu.org/bugs/?50485
https://savannah.gnu.org/bugs/?50619

Do you have a simple example of partial file read? Say, a four line file in which only two lines are read at a time?

Dan Sebald <sebald>
Tue 26 Sep 2017 07:05:24 AM UTC, original submission:

Bug # 47671: When textscan reads only part of a file, the read pointer is left at the wrong position, so that subsequent reads re-read some of the data.

Though this bug is reported as patched and closed, it still seems to be present in the Windows build. I am using the Windows 10, 64 bit and get exactly the problem as described when using subsequent textscan() calls.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #41897:  read_file.m added by cees_meijer (227B - application/octet-stream - Data file and test program that show the textscan problem)
file #41898:  TestProfiles_1.TXT added by cees_meijer (855B - text/plain - Data file and test program that show the textscan problem)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by lostbard (Posted a comment)
  • -unavailable- added by cees_meijer (Posted a comment)
  • -unavailable- added by sebald (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 27 Sep 2017 04:10:00 PM UTCsebaldAttached File-=>Added octave-textscan_stream_pointer_restore-djs2017feb27.patch, #41900
      Attached File-=>Added octave-reuse_delimited_stream_constructor-djs2017feb27.patch, #41901
    Tue 26 Sep 2017 06:08:53 PM UTCcees_meijerAttached File-=>Added read_file.m, #41897
      Attached File-=>Added TestProfiles_1.TXT, #41898

    Back to the top


    Powered by Savane 3.1-cleanup1