bugGNU Octave - Bugs: bug #47539, textscan misinterprets escape...

 
 

bug #47539: textscan misinterprets escape sequences in single quoted strings

Submitted by:  Lars Kindermann <larskindermann>
Submitted on:  Sat 26 Mar 2016 04:10:50 AM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: FixedAssigned to: Lachlan Andrew <lachlan>
Originator Name: Lars KindermannOpen/Closed: Closed
Release: devOperating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon 28 Mar 2016 06:13:02 PM UTC, comment #11:

I've applied the second patch to fix the buffer corruption with non-default delimiters.

http://hg.savannah.gnu.org/hgweb/octave/rev/102b33b53ea4

The provided example works for me, closing this bug report.

Mike Miller <mtmiller>
Project Administrator
Mon 28 Mar 2016 05:51:09 PM UTC, comment #10:

I've applied the original patch to accept escape characters in single-quoted strings:

http://hg.savannah.gnu.org/hgweb/octave/rev/990c6c31a684

In the future please open new bug reports for separate issues. For now this bug remains open for the buffer corruption bug.

Whether or not textscan returns an EOF character should also be a separate bug.

Mike Miller <mtmiller>
Project Administrator
Mon 28 Mar 2016 05:16:07 PM UTC, comment #9:

Matlab 2012a does return ASCII 26 in textscan:

ends with
and is confirmed with

Anonymous
Mon 28 Mar 2016 04:27:48 PM UTC, comment #8:

Thanks, the two patches fix the bug for me.

(while *buffer = '\0' did not help on my system)

Note that the file I submitted ends with a 1Ah (Ctrl-Z) as EOF marker. The old textscan from 4.0 also returns it as the final element of f, so it is not a new bug. I am not sure what the "correct" behavior would be. Probably a trailing EOF should indeed be ignored by textscan. How does MATLAB behave?

Lars Kindermann <larskindermann>
Mon 28 Mar 2016 09:06:57 AM UTC, comment #7:

Here is a patch that fixes the random junk bug. Note that it is not a cumulative patch -- you need both it and file #36763. (Note to whoever commits this: please let me know if you'd prefer me to merge them.)

The problem was that the code does its own buffering to make lookahead and backtracking faster, and the set of delimiters used to prevent overrun of that buffer was different from the ones used to separate the %s strings.

It still has a bug: It returns an extra line containing only 1Ah (=26). I'll look into that next -- but it should be easy to work around in the mean time.

(file #36784)

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Mon 28 Mar 2016 07:39:11 AM UTC, comment #6:

Thanks, Anonymous. That will keep Lars going while we work out the right fix.

I think the problem is the difference between the set of delimiters used in delimited_stream::refresh_buf () and the set used in textscan::scan_string, but I don't understand exactly what is going on and don't have a fix yet.

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Mon 28 Mar 2016 01:30:25 AM UTC, comment #5:

In file oct-stream.cc:

I added a statement after line 1541 so that it now reads:

This fixes the problem with printing Lars's large text file for me, but it does not seem the most robust solution. Someone please check.

Anonymous
Mon 28 Mar 2016 01:08:36 AM UTC, comment #4:

I'm getting different output:

There's a periodic messup every 88 lines (lines 261, 349, 437, 525, then 961, 1049, etc). Almost certainly that's the buffer size for this read operation and there's some unterminated string somewhere. Looking...

Anonymous
Sat 26 Mar 2016 08:57:36 AM UTC, comment #3:

OK, the patch fixes the 'delimiter' '\n' problem for me.

But the problem with long files looks serious:
I attached a file 'textscantest.txt'
Running this script in current dev (4.0 works fine)

produces this garbage for me

Also octave crashes during textscan within another large program with

Speicherzugriffsfehler = Memory access error
Probably a pointer problem?

(file #36766)

Lars Kindermann <larskindermann>
Sat 26 Mar 2016 07:16:21 AM UTC, comment #2:

Lars, can you confirm that the attached patch fixes the problem? Thanks.

(file #36763)

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Sat 26 Mar 2016 06:34:50 AM UTC, comment #1:

I can confirm this issue. It only occurs with single-quoted strings.

I'll look into it now. Meanwhile, could you please post a bug report with a reproducible example of the bug with long files? If the file is too long to attach to the bug report, you may be able to put it on the web and attach the URL.

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Sat 26 Mar 2016 04:10:50 AM UTC, original submission:

in current dev (works fine in 4.0)

Save this as textscantest.m:

and execute! Result:

This looks like a wrong interpretion of '\n'

But there must be more bugs, when reading long textfiles some binary garbage is appearing in the output...

Lars Kindermann <larskindermann>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #36766:  textscantest.txt added by larskindermann (604KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by lachlan (Posted a comment)
  • -unavailable- added by larskindermann (Submitted the item)
  • -unavailable- added by larskindermann
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 10 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 28 Mar 2016 06:13:02 PM UTCmtmillerStatusPatch Submitted=>Fixed
      Open/ClosedOpen=>Closed
    Mon 28 Mar 2016 09:06:57 AM UTClachlanAttached File-=>Added bug_47539_textscan_buf_delim.cset, #36784
    Sat 26 Mar 2016 08:57:36 AM UTClarskindermannAttached File-=>Added textscantest.txt, #36766
    Sat 26 Mar 2016 07:16:21 AM UTClachlanAttached File-=>Added bug_47539_textscan_escapes.cset, #36763
      StatusConfirmed=>Patch Submitted
    Sat 26 Mar 2016 06:34:50 AM UTClachlanStatusNone=>Confirmed
      Assigned toNone=>lachlan
      Summarytextscan is broken=>textscan misinterprets escape sequences in single quoted strings
    Sat 26 Mar 2016 04:10:50 AM UTClarskindermannCarbon-Copy-=>Added larskindermann

    Back to the top


    Powered by Savane 3.1-cleanup1