bugGNU Octave - Bugs: bug #47539, textscan misinterprets escape...

 
 

bug #47539: textscan misinterprets escape sequences in single quoted strings

Submitter:  Lars Kindermann <larskindermann>
Submitted:  Sat 26 Mar 2016 04:10:50 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  lachlan
Originator Name:  Lars Kindermann Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 28 Mar 2016 06:13:02 PM UTC, comment #11: 

I've applied the second patch to fix the buffer corruption with non-default delimiters.

http://hg.savannah.gnu.org/hgweb/octave/rev/102b33b53ea4

The provided example works for me, closing this bug report.

Mike Miller <mtmiller>
Group Member
Mon 28 Mar 2016 05:51:09 PM UTC, comment #10: 

I've applied the original patch to accept escape characters in single-quoted strings:

http://hg.savannah.gnu.org/hgweb/octave/rev/990c6c31a684

In the future please open new bug reports for separate issues. For now this bug remains open for the buffer corruption bug.

Whether or not textscan returns an EOF character should also be a separate bug.

Mike Miller <mtmiller>
Group Member
Mon 28 Mar 2016 05:16:07 PM UTC, comment #9: 

Matlab 2012a does return ASCII 26 in textscan:


>> FID = fopen('textscantest.txt', 'r');
>> f = textscan(FID, '%s', 'delimiter', '\n');
>> fclose(FID);
>> tmp = f{1}

ends with

    '850;2016032422;   1;  38;   5.8;  92.0;eor'
    '850;2016032423;   1;  38;   5.8;  93.0;eor'
    ''

and is confirmed with

>> foo = tmp{end} + 0
foo =
    26


Anonymous
Mon 28 Mar 2016 04:27:48 PM UTC, comment #8: 

Thanks, the two patches fix the bug for me.

(while *buffer = '\0' did not help on my system)

Note that the file I submitted ends with a 1Ah (Ctrl-Z) as EOF marker. The old textscan from 4.0 also returns it as the final element of f, so it is not a new bug. I am not sure what the "correct" behavior would be. Probably a trailing EOF should indeed be ignored by textscan. How does MATLAB behave?

Lars Kindermann <larskindermann>
Mon 28 Mar 2016 09:06:57 AM UTC, comment #7: 

Here is a patch that fixes the random junk bug.  Note that it is not a cumulative patch -- you need both it and file #36763.  (Note to whoever commits this: please let me know if you'd prefer me to merge them.)

The problem was that the code does its own buffering to make lookahead and backtracking faster, and the set of delimiters used to prevent overrun of that buffer was different from the ones used to separate the %s strings.

It still has a bug: It returns an extra line containing only 1Ah (=26).  I'll look into that next -- but it should be easy to work around in the mean time.

(file #36784)

Lachlan Andrew <lachlan>
Mon 28 Mar 2016 07:39:11 AM UTC, comment #6: 

Thanks, Anonymous.  That will keep Lars going while we work out the right fix.

I think the problem is the difference between the set of delimiters used in delimited_stream::refresh_buf () and the set used in textscan::scan_string, but I don't understand exactly what is going on and don't have a fix yet.

Lachlan Andrew <lachlan>
Mon 28 Mar 2016 01:30:25 AM UTC, comment #5: 

In file oct-stream.cc:

1535           else  // Reading more than the whole buf; return it in buffer
1536             {
1537               retval = buffer;
1538               // FIXME -- read bufsize at a time
1539               int i;
1540               for (i = 0; i < size && ! eof (); i++)
1541                 *buffer++ = get_undelim ();
1542               if (eof ())
1543                 memset (buffer, 0, size - i);


I added a statement after line 1541 so that it now reads:

1541                 *buffer++ = get_undelim ();
1542               *buffer = '\0';
1543               if (eof ())
1544                 memset (buffer, 0, size - i);


This fixes the problem with printing Lars's large text file for me, but it does not seem the most robust solution. Someone please check.

Anonymous
Mon 28 Mar 2016 01:08:36 AM UTC, comment #4: 

I'm getting different output:

f =
{
  [1,1] =
  {
    [1,1] = STATIONS_ID; MESS_DATUM; QUALITAETS_NIVEAU; STRUKTUR_VERSION; LUFTTEMPERATUR;REL_FEUCHTE;eor
    [2,1] = 850;2014092200;   7;  38;  11.8;  91.0;eor
    [3,1] = 850;2014092201;   7;  38;  11.4;  91.0;eor
SNIP
    [260,1] = 850;2014100219;   7;  38;  16.6;  92.0;eor
    [261,1] = 850;2014100220;   e��Mh�؄
    [262,1] = 7;  38;  16.3;  93.0;eor
    [263,1] = 850;2014100221;   7;  38;  16.2;  93.0;eor
SNIP
    [348,1] = 850;2014100615;   7;  38;  17.4;  60.0;eor
    [349,1] = 850;2014100616;   7;  38;e��Mh�؄
    [350,1] = 16.7;  63.0;eor
    [351,1] = 850;2014100617;   7;  38;  16.2;  69.0;eor
SNIP
    [436,1] = 850;2014101006;   7;  38;  12.1;  91.0;eor
    [437,1] = 850;2014101007;   7;  38;  12.1;e��Mh�؄
    [438,1] = 90.0;eor
    [439,1] = 850;2014101008;   7;  38;  13.1;  86.0;eor
SNIP
    [524,1] = 850;2014101321;   7;  38;  12.3;  99.0;eor
    [525,1] = 850;2014101322;   7;  38;  12.2;  99.0;e��Mh�؄
    [526,1] = eor
    [527,1] = 850;2014101323;   7;  38;  11.2;  99.0;eor
SNIP
    [961,1] = 850;2014110101;   7;  38;  10.4;  99.0;eor
    [962,1] = 850;2014110102;   7;  3e��Mh�؄
    [963,1] = 8;  10.1;  98.0;eor
    [964,1] = 850;2014110103;   7;  38;   9.9;  99.0;eor
SNIP
    [1049,1] = 850;2014110417;   7;  38;   9.4;  95.0;eor
    [1050,1] = 850;2014110418;   7;  38;   9.e��Mh�؄
    [1051,1] = 2;  96.0;eor
    [1052,1] = 850;2014110419;   7;  38;   9.2;  97.0;eor
SNIP
    [1137,1] = 850;2014110808;   7;  38;   8.3;  86.0;eor
    [1138,1] = 850;2014110809;   7;  38;   9.3;  83.e��Mh�؄
    [1139,1] = 0;eor
    [1140,1] = 850;2014110810;   7;  38;  10.6;  79.0;eor
SNIP
    [1574,1] = 850;2014112612;   7;  38;   5.1;  76.0;eor
    [1575,1] = 850;2014112613;   7;  3e��Mh�؄
    [1576,1] = 8;   5.2;  76.0;eor
    [1577,1] = 850;2014112614;   7;  38;   5.3;  76.0;eor
SNIP
    [1662,1] = 850;2014113003;   7;  38;  -1.0;  92.0;eor
    [1663,1] = 850;2014113004;   7;  38;  -1.e��Mh�؄
    [1664,1] = 1;  92.0;eor
    [1665,1] = 850;2014113005;   7;  38;  -1.1;  92.0;eor
SNIP
    [1750,1] = 850;2014120318;   7;  38;  -2.6;  84.0;eor
    [1751,1] = 850;2014120319;   7;  38;  -2.5;  85.e��Mh�؄
    [1752,1] = 0;eor
    [1753,1] = 850;2014120320;   7;  38;  -2.4;  86.0;eor
SNIP
    [2187,1] = 850;2014122122;   7;  38;   8.1;  82.0;eor
    [2188,1] = 850;2014122123;   7;  3e��Mh�؄
    [2189,1] = 8;   8.6;  81.0;eor
ETC


There's a periodic messup every 88 lines (lines 261, 349, 437, 525, then 961, 1049, etc). Almost certainly that's the buffer size for this read operation and there's some unterminated string somewhere. Looking...

Anonymous
Sat 26 Mar 2016 08:57:36 AM UTC, comment #3: 

OK, the patch fixes the 'delimiter' '\n' problem for me.

But the problem with long files looks serious:
I attached a file 'textscantest.txt'
Running this script in current dev (4.0 works fine)


FID = fopen('textscantest.txt', 'r');
f = textscan(FID, '%s', 'delimiter', '\n');
fclose(FID);
f


produces this garbage for me


...
    [524,1] = 850;2014101321;   7;  38;  12.3;  99.0;eor
    [525,1] = 850;2014101322;   7;  38;  12.2;  99.0;^A^@^@^@^@^@^@^@%^@^@^@^@^@^@^@P^].$~^?^@^@*^@^@^@^A^@
^@^@<C0><DE>+$~^?^@^@U^@^@^@^@^@^@^@<A0>"<A5>h~^?^@^@^A^@^@^@08; X<<F4>h~^?^@^@<A8><DF>+$~^?^@^@<C0><DF>+$~
^?^@^@0<E0>+$~^?^@^@*^@^@^@ 7; ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@E^@^@^@^@^@^@^@850;2014092307;   7;  38;   8.;  83.0;eor4092307;   7; %^@^@^@^@^@^@^@^A^@^@^@^B^@^@^@^A^@^@^@*^@^@^@ ^@^@^@^@^@^@^@%^@^@^@^@^@^@^@^A^@^@@^B^@^@^@^A^@^@^@*^@^@^@ ^@*$~^?^@^@%^@^@^@^@^@^@^@0<E0>+$~^?^@^@*^@^@^@^A^@^@^@PK*$~^?^@^@U^@^@^@^@^@^@^@
<A0>"<A5>h~^?^@^@^A^@^@^@09; X<<F4>h~^?^@^@x<E0>+$~^?^@^@<90><E0>+$~^?^@^@<B0><E0>+$~^?^@^@*^@^@^@ 7; ^@^@^^@^@^@^@^@^@^@^@^@^@^@^@^@E^@^@^@^@^@^@^@850;2014092308;   7;  38;  11.3;  69.0;eor4092308;   7; %^@^@^@^@^^@^@^A^@^@^@^B^@^@^@^A^@^@^@*^@^@^@ ^@*$~^?^@^@%^@^@^@^@^@^@^@<B0><E0>+$~^?^@^@*^@^@^@^A^@^@^@<80>^S*$~^?^@
^@E^@^@^@^@^@^@^@850;2014092309;   7;  38;  13.7;  57.0;eor^@$~^?^@^@@^@^@^@^@^@^@^@U^@^@^@^@^@^@^@<A0>"<A5h~^?^@^@^A^@^@^@16; X<<F4>h~^?^@^@<88><E1>+$~^?^@^@<A0><E1>+$~^?^@^@^P<E2>+$~^?^@^@*^@^@^@ 7; ^@^@^@^@^@^@^^@^@^@^@^@^@^@^@^@E^@^@^@^@^@^@^@850;2014092215;   7;  38;  14.1;  52.0;eor4092215;   7; %^@^@^@^@^@^@^@^A^^@^@^B^@^@^@^A^@^@^@*^@^@^@ ^@j$~^?^@^@%^@^@^@^@^@^@^@^P<E2>+$~^?^@^@*^@^@^@^A^@^@^@<D0><E1>+$~^?^@^@U^@^@^^@^@^@^@<A0>"<A5>h~^?^@^@^A^@^@^@17; X<<F4>h~^?^@^@X<E2>+$~^?^@^@p<E2>+$~^?^@^@<E0><E2>+$~^?^@^@*^@^@^@ 7;
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@E^@^@^@^@^@^@^@850;2014092216;   7;  38;  13.7;  56.0;eor4092216;   7; %^@^^@^@^@^@^@^A^@^@^@^B^@^@^@^A^@^@^@*^@^@^@ ^@^@^@^@^@^@^@%^@^@^@^@^@^@^@<E0><E2>+$~^?^@^@*^@^@^@^A^@^@^@<D0>
<E0>+$~^?^@^@U^@^@^@^@^@^@^@<A0>"<A5>h~^?^@^@^A^@^@^@18; X<<F4>h~^?^@^@(<E3>+$~^?^@^@@<E3>+$~^?^@^@<B0><E3>$~^?^@^@*^@^@^@ 7; ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@E^@^@^@^@^@^@^@850;2014092217;   7;  38;  13.5;  58.0;eo4092217;   7; %^@^@^@^@^@^@^@^A^@^@^@^B^@^@^@^A^@^@^@*^@^@^@ ^@^@^@^@^@^@^@%^@^@^@^@^@^@^@<B0><E3>+$~^?^@^@^@^@^@^A^@^@^@Њj$~^?^@^@U^@^@^@^@^@^@^@<A0>"<A5>h~^?^@^@^A^@^@^@19; X<<F4>h~^?^@^@<F8><E3>+$~^?^@^@^P<E4>+$^?^@^@<80><E4>+$~^?^@^@*^@^@^@ 7; ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@E^@^@^@^@^@^@^@850;2014092218;   7;  38;13.1;  62.0;eor4092218;   7; %^@^@^@^@^@^@^@^A^@^@^@^B^@^@^@^A^@^@^@*^@^@^@ ^@.$~^?^@^@%^@^@^@^@^@^@^@<80>
<E4>+$~^?^@^@*^@^@^@^A^@^@^@.^@^@^@^@^@^@^@U^@^@^@^@^@^@^@<A0>"<A5>h~^?^@^@^A^@^@^@20; X<<F4>h~^?^@^@<C8>
<E4>+$~^?^@^@<90><9D>-$~^?^@^@p^P*$~^?^@^@*^@^@^@ 7; ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@E^@^@^@^@^@^@^@850;201092219;   7;  38;  12.7;  66.0;eorl$~^?^@^@<C0><D5>+$~^?^@^@%^@^@^@^@^@^@^@^A^@^@^@^B^@^@^@^A^@^@^@*^@^@^@
^@^@^@^@^@^@^@<95>^A^@^@^@^@^@^@^@^@^@^@~^?^@^@<B0>c*$~^?^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<A0><8D>,$~^?^^@#^@^@^@^@^@^@^@#^@^@^@^@^@^@^@^^^@^@^@^@^@^@^@p<E6>+$~^?^@^@#^@^@^@^@^@^@^@#^@^@^@^@^@^@^@pkg.m^@^@^@P<E5+$~^?^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@PKG_ADD^@^@^@^@^@^@^@^@^@<FB><86><F5>V^@^@^@^@^@^@^@^@^@^@^@^@<9F><F6>V^@^@^@^@&6 ^@^@^@^@^@x<B5><F3>h~^?^@^@H<F2>+$~^?^@^@<E0><F0>+$~^?^@^@h<F2>+$~^?^@^@^D^@^@^@~^?^@^@x<B5<F3>h~^?^@^@(u*$~^?^@^@<B0><C4>+$~^?^@^@(<CA>+$~^?^@^@^A^@^@^@~^?^@^@^G^@^@^@^A^@^@^@^@^@^@^@^@^@^@^@<A0>
<E6>+$~^?^@^@<F0><F2>+$~^?^@^@<D0><E7>+$~^?^@^@&^@^@^@^@^@^@^@<F1>^@^@^@^@^@^@^@^@^@^@^@~^?^@^@^@^@^@^@^@^@
^@^@^P<E6>+$~^?^@^@^P<E6>+$~^?^@^@^@^@^@^@^@^@^@^@H<E6>+$~^?^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@<E6>+$~^?^^@@<E6>+$~^?^@^@^@^@^@^@^@^@^@^@5^@^@^@^@^@^@^@/home/kinder/usr/octave/scripts/pkg^@^@^@^@^@e^@^@^@^@^@^@^@
^A^@^@^@~^?^@^@<E0><E5>+$~^?^@^@`<EE>+$~^?^@^@`<ED>+$~^?^@^@<80>",$~^?^@^@^T^@^@^@^@^@^@^@^T^@^@^@^@^@^@^@
<E8><E6>+$~^?^@^@^A^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@`^@^@^@^@^@^@^@%^@^@^@^@^@^@^@__get_properties__^@^@^@^@^@
^@U^@^@^@^@^@^@^@^@^@^@^@~^?^@^@<F0><E9>+$~^?^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@P<E7>+$~^?^@^@^O^@^@^@^@^@
^@^@unload_packages^@^A^@^@^@~^?^@^@5^@^@^@^@^@^@^@load_packages_and_dependencies^@^@0^@^@^@^@^@^@^@5^@^@^@
^@^@^@^@/home/kinder/usr/octave/scripts/audio^@^@^@U^@^@^@^@^@^@^@^@^@^@^@~^?^@^@<F0><E9>+$~^?^@^@^@^@^@^@^^@^@^@^@^@^@^@^@^@^@^@^@<E8>+$~^?^@^@^K^@^@^@^@^@^@^@write_index^@^@^@^@^@^A^@^@^@~^?^@^@5^@^@^@^@^@^@^@is_rchitecture_dependent^@pkg.m^@0^@^@^@^@^@^@^@%^@^@^@^@^@^@^@__uigetdir_fltk__^@^@^@^@^@^@^@U^@^@^@^@^@^@^@^^@^@^@~^?^@^@^P<C4>+$~^?^@^@P<F1>+$~^?^@^@<C0><E8>+$~^?^@^@<A0><E8>+$~^?^@^@^G^@^@^@^@^@^@^@install^@<C1>^@
^@^@^@^@^@^@^A^@^@^@~^?^@^@U^@^@^@^@^@^@^@^A^@^@^@~^?^@^@p<E8>+$~^?^@^@^@^@^@^@^@^@^@^@^@<F1>+$~^?^@^@^P<E9+$~^?^@^@^R^@^@^@^@^@^@^@^R^@^@^@^@^@^@^@x^@^@$~^?^@^@^A^@^@^@~^?^@^@%^@^@^@^@^@^@^@installed_packages^@^@^^@^@^@E^@^@^@^@^@^@^@/home/kinder/usr/octave/scripts/deprecated^@$~^?^@^@@^@^@^@^@^@^@^@%^@^@^@^@^@^@^@^B^@
^@^@^B^@^@^@^E^@^@^@^A^@^@^@ ^@^@^@^@^@^@^@E^@^@^@^@^@^@^@``k$~^?^@^@``k$~^?^@^@<80><A9>0$~^?^@^@*^@^@^@^@^^@^@*^@^@^@^@^@^@^@ed^@ctory^A^@^@^@^@^@^@^@%^@^@^@^@^@^@^@<F8>^Q,$~^?^@^@^W^@^@^@^B^@^@^@ ^@^@^@^@^@^@^@U^^@^@^@^@^@^@^A^@^@^@~^?^@^@^@<EB>+$~^?^@^@ <E7>+$~^?^@^@<D0><E7>+$~^?^@^@<F0><EB>+$~^?^@^@^P^@^@^@^@^@^@^@^^@^@^@^@^@^@^@A^@^@^@^@^@^@^@^A^@^@^@~^?^@^@5^@^@^@^@^@^@^@is_architecture_dependent^@m^@^@.m^@0^@^@^@^@^@^^@5^@^@^@^@^@^@^@__osmesa_print__.oct^@md_la-colam0^@^@^@^@^@^@^@E^@^@^@^@^@^@^@p<C3>-$~^?^@^@p<C3>-$~^?^@^<C0><8D>/$~^?^@^@$^@^@^@^@^@^@^@$^@^@^@^@^@^@^@ed^@^@^@^@^@^@^A^@^@^@^@^@^@^@%^@^@^@^@^@^@^@^B^@^@^@^B^@^@^^W^@^@^@^A^@^@^@^@^@^@^@^@^@^@^@U^@^@^@^@^@^@^@^@^@^@^@~^?^@^@<A0><EB>+$~^?^@^@<80>!,$~^?^@^@<F0><E9>+$~^?^^@0<EB>+$~^?^@^@        ^@^@^@^@^@^@^@uninstall^@^@^@^@^@^@^@^A^@^@^@~^?^@^@5^@^@^@^@^@^@^@print_package_decription^@x.m^@m^@0^@^@^@^@^@^@^@%^@^@^@^@^@^@^@__uigetfile_fltk__^@^@^@^@^@^@U^@^@^@^@^@^@^@^A^@^@^@~^?^@^`<ED>+$~^?^@^@^P<C1>+$~^?^@^@^@<EB>+$~^?^@^@<D0><EB>+$~^?^@^@
    [526,1] = eor
    [527,1] = 850;2014101323;   7;  38;  11.2;  99.0;eor
    [528,1] = 850;2014101400;   7;  38;  13.1;  97.0;eor
...


Also octave crashes during textscan within another large program with


panic: Speicherzugriffsfehler -- stopping myself...
attempting to save variables to 'octave-workspace'...
save to 'octave-workspace' complete
octave exited with signal 11


Speicherzugriffsfehler = Memory access error
Probably a pointer problem?


(file #36766)

Lars Kindermann <larskindermann>
Sat 26 Mar 2016 07:16:21 AM UTC, comment #2: 

Lars, can you confirm that the attached patch fixes the problem?  Thanks.

(file #36763)

Lachlan Andrew <lachlan>
Sat 26 Mar 2016 06:34:50 AM UTC, comment #1: 

I can confirm this issue.  It only occurs with single-quoted strings.

I'll look into it now.  Meanwhile, could you please post a bug report with a reproducible example of the bug with long files?  If the file is too long to attach to the bug report, you may be able to put it on the web and attach the URL.

Lachlan Andrew <lachlan>
Sat 26 Mar 2016 04:10:50 AM UTC, original submission:  

in current dev (works fine in 4.0)

Save this as textscantest.m:


FID = fopen('textscantest.m', 'r');
f = textscan(FID, '%s', 'delimiter', '\n');
fclose(FID);
f


and execute! Result:


>> textscantest

f =
{
  [1,1] =
  {
    [1,1] = FID = fope
    [2,1] = ('textsca
    [3,1] = test.m', 'r');
    [4,1] = f = textsca
    [5,1] = (FID, '%s', 'delimiter', '
    [6,1] =
    [7,1] = ');
    [8,1] = fclose(FID);
    [9,1] = f
  }
}
>>


This looks like a wrong interpretion of '\n'

But there must be more bugs, when reading long textfiles some binary garbage is appearing in the output...

Lars Kindermann <larskindermann>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #36766:  textscantest.txt added by larskindermann (604KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by lachlan (Posted a comment)
  • -email is unavailable- added by larskindermann (Submitted the item)
  • -email is unavailable- added by larskindermann
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-03-28 mtmiller StatusPatch Submitted Fixed
        Open/ClosedOpen Closed
    2016-03-28 lachlan Attached File- Added bug_47539_textscan_buf_delim.cset, #36784
    2016-03-26 larskindermann Attached File- Added textscantest.txt, #36766
    2016-03-26 lachlan Attached File- Added bug_47539_textscan_escapes.cset, #36763
        StatusConfirmed Patch Submitted
    2016-03-26 lachlan StatusNone Confirmed
        Assigned toNone lachlan
        Summarytextscan is broken textscan misinterprets escape sequences in single quoted strings
    2016-03-26 larskindermann Carbon-Copy- Added larskindermann

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code