bugGNU Octave - Bugs: bug #63383, Error in sscanf in special cases...

 
 

bug #63383: Error in sscanf in special cases when '%f' conversion is used

Submitter:  José Luis García Pallero <jgpallero>
Submitted:  Fri 18 Nov 2022 02:34:46 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  jgpallero Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 02 Mar 2024 10:07:08 PM UTC, comment #22: 

This is working again thanks to a changeset for bug #65390.  Closing this report again.

Rik <rik5>
Group administrator
Fri 01 Mar 2024 07:33:17 PM UTC, comment #21: 

Re-opening bug.

scanf has proven to be really difficult to modify.  I will try to take a look at this on the weekend.

Rik <rik5>
Group administrator
Fri 01 Mar 2024 07:29:00 PM UTC, comment #20: 

The changeset:

changeset:   31542:2f08a53e0a23
parent:      31540:ce5b4a00b022
user:        Rik <rik@octave.org>
date:        Fri Nov 25 08:30:30 2022 -0800
summary:     Fix scanf handling of exceptional values (NA, NaN, Inf) at EOF (bug #63383)


seems to introduce the bug #65390

octave:1> s2 = sprintf("1 2\n")
s2 = 1 2

octave:2> a2 = sscanf(s2, '%f  %f')
a2 =

   1
   2
   2


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 19 Dec 2022 12:01:49 AM UTC, comment #19: 

Thanks for reporting this warning.  It is, in this case, an actual fault in the logic.  There are still two other scanf bugs outstanding and the resolution of those will require fixing this as well.

Rik <rik5>
Group administrator
Sun 18 Dec 2022 12:31:07 AM UTC, comment #18: 

I noticed compiling with clang on aarch64 I get the warning that
I do not get compiling on x86_64:


libtool: compile:  clang++ -DHAVE_CONFIG_H -I. -I.. -DOCTAVE_DLL -DEXTERNAL_DLL -Iliboctave -I../liboctave -I../liboctave/array -Iliboctave/numeric -I../liboctave/numeric -Iliboctave/operators -I../liboctave/operators -I../liboctave/system -I../liboctave/util -I../liboctave/wrappers -fPIC -pthread -fopenmp -Wall -W -Wshadow -Woverloaded-virtual -Wold-style-cast -Wformat -Wpointer-arith -Wwrite-strings -Wcast-align -Wcast-qual -fvisibility=hidden -pipe -O2 -flto=thin -mcpu=native -mtune=native -MT liboctave/util/libutil_la-lo-utils.lo -MD -MP -MF liboctave/util/.deps/libutil_la-lo-utils.Tpo -c ../liboctave/util/lo-utils.cc  -fPIC -DPIC -o liboctave/util/.libs/libutil_la-lo-utils.o
../liboctave/util/lo-utils.cc:244:26: warning: result of comparison of constant -1 with expression of type 'char' is always true [-Wtautological-constant-out-of-range-compare]
                  if (c2 != std::istream::traits_type::eof ())
                      ~~ ^  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.

hg id
67770ec6f0b5 tip @

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 07 Dec 2022 12:45:15 AM UTC, comment #17: 

Thanks, the NA is now correct, but now bug #63467 is causing a duplicate at the end.  Unfortunately, correcting the duplicate is going to be harder and I'm going to have to change function signatures.

Rik <rik5>
Group administrator
Wed 07 Dec 2022 12:06:21 AM UTC, comment #16: 

Same on Ubuntu 22.10/aarch64

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 06 Dec 2022 11:33:10 PM UTC, comment #15: 

OK, on Fedora 37/aarch64:


octave:1> version -hgid
ans = deb553ac2c54
octave:2> [val, count, msg, pos] = sscanf ('2 na 3', '%f');
 assert (val, [2; NA; 3]);

 # This time NA at end
 [val, count, msg, pos] = sscanf ('2 3 na', '%f');
 assert (val, [2; 3; NA]);
error: ASSERT errors for:  assert (val,[2; 3; NA])

  Location  |  Observed  |  Expected  |  Reason
     .          O(2x1)       E(3x1)      Dimensions don't match
octave:6> sscanf ('2 3 na', '%f')
ans =

   2
   3

octave:7>


With the patch


octave:1> version -hgid
ans = deb553ac2c54+
octave:2> [val, count, msg, pos] = sscanf ('2 na 3', '%f');
 assert (val, [2; NA; 3]);

 # This time NA at end
 [val, count, msg, pos] = sscanf ('2 3 na', '%f');
 assert (val, [2; 3; NA]);
error: ASSERT errors for:  assert (val,[2; 3; NA])

  Location  |  Observed  |  Expected  |  Reason
     .          O(4x1)       E(3x1)      Dimensions don't match
octave:6> sscanf ('2 3 na', '%f')
ans =

     2
     3
    NA
    NA


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 06 Dec 2022 10:30:01 PM UTC, comment #14: 

I will, but that will take a while, the computer is slow.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 06 Dec 2022 10:20:17 PM UTC, comment #13: 

Can you try the attached patch: 63383.diff?

(file #54066)

Rik <rik5>
Group administrator
Tue 06 Dec 2022 10:02:11 PM UTC, comment #12: 

Yes:


octave:1> sscanf ('2 3 na', '%f')
ans =

   2
   3

octave:2>


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 06 Dec 2022 09:57:09 PM UTC, comment #11: 

Can you run just


sscanf ('2 3 na', '%f')


so I can see the output.  I expect it is only picking up the "2 3" and ignoring the "na", but always good to confirm.

Rik <rik5>
Group administrator
Tue 06 Dec 2022 09:46:45 PM UTC, comment #10: 

It compiles dev, but with stable (on Fedora 37) I see:


octave:1> version -hgid
ans = 9978ef12aea3
octave:2> [val, count, msg, pos] = sscanf ('2 na 3', '%f');
 assert (val, [2; NA; 3]);

 # This time NA at end
 [val, count, msg, pos] = sscanf ('2 3 na', '%f');
 assert (val, [2; 3; NA]);
error: ASSERT errors for:  assert (val,[2; 3; NA])

  Location  |  Observed  |  Expected  |  Reason
     .          O(2x1)       E(3x1)      Dimensions don't match
octave:6>


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 06 Dec 2022 09:37:30 PM UTC, comment #9: 

It generally handles NAs fine except some weird cases like this one and https://savannah.gnu.org/bugs/?61358

I am rebooting it to Fedora, see if it is is there as well.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 06 Dec 2022 09:31:09 PM UTC, comment #8: 

Does NA value otherwise work on the specific arch64 machine you are using?

Quick test,


 [val, count, msg, pos] = sscanf ('2 na 3', '%f');
 assert (val, [2; NA; 3]);

 # This time NA at end
 [val, count, msg, pos] = sscanf ('2 3 na', '%f');
 assert (val, [2; 3; NA]);


I'm running Kubuntu 22.04, x86_64 and the code above works.  I see you are on Ubuntu 22.10 which might make a difference.

Rik <rik5>
Group administrator
Tue 06 Dec 2022 09:04:59 PM UTC, comment #7: 

I have a couple regressions on aarch64 (Ubuntu 22.10) hgid ab33554f97d2


octave:1> test ../test/io.tst
***** test <*63383>
 [val, count, msg, pos] = sscanf ('2 3 n', '%f');
 assert (val, [2; 3]);
 assert (count, 2);
 assert (msg, 'sscanf: format failed to match');
 assert (pos, 5);
 [val, count, msg, pos] = sscanf ('2 3 na', '%f');
 assert (val, [2; 3; NA]);
 assert (count, 3);
 assert (msg, '');
 assert (pos, 7);
 [val, count, msg, pos] = sscanf ('2 3 nan', '%f');
 assert (val, [2; 3; NaN]);
 assert (count, 3);
 assert (msg, '');
 assert (pos, 8);
!!!!! regression: https://octave.org/testfailure/?63383
ASSERT errors for:  assert (val,[2; 3; NA])

  Location  |  Observed  |  Expected  |  Reason
     .          O(2x1)       E(3x1)      Dimensions don't match
***** test <*63383>
 [val, count, msg, pos] = sscanf ('+NA -NA 1 +NAN -NAN', '%f');
 assert (val, [NA; NA; 1; NaN; NaN]);
 assert (count, 5);
 assert (msg, '');
 assert (pos, 20);
 [val, count, msg, pos] = sscanf ('-NA', '%f');
 assert (val, NA);
 assert (count, 1);
 assert (msg, '');
 assert (pos, 4);
 [val, count, msg, pos] = sscanf ('+NA', '%f');
 assert (val, NA);
 assert (count, 1);
 assert (msg, '');
 assert (pos, 4);
 [val, count, msg, pos] = sscanf ('-NaN', '%f');
 assert (val, NaN);
 assert (count, 1);
 assert (msg, '');
 assert (pos, 5);
 [val, count, msg, pos] = sscanf ('+NaN', '%f');
 assert (val, NaN);
 assert (count, 1);
 assert (msg, '');
 assert (pos, 5);
!!!!! regression: https://octave.org/testfailure/?63383
ASSERT errors for:  assert (val,NA)

  Location  |  Observed  |  Expected  |  Reason
     .          O(0x1)       E(1x1)      Dimensions don't match
warning: test: file /home/uu1/src/octave/test/io.tst leaked global variables: __assert_printf__
PASSES 173 out of 175 tests
octave:2> version -hgid
ans = ab33554f97d2


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 25 Nov 2022 04:58:52 PM UTC, comment #6: 

Thank you very much for your work, Rik

José Luis García Pallero <jgpallero>
Fri 25 Nov 2022 04:53:01 PM UTC, comment #5: 

I finally finished work on the scanf set of function (last changeset: http://hg.savannah.gnu.org/hgweb/octave/rev/2f08a53e0a23).  Everything should be working on the development branch now (Octave 9).  The changes were deemed too uncertain to be placed on the stable branch (Octave 8.1) a mere month ahead of an expected release.  However, with more testing they should be able to be a part of the 8.2 bug fix release.

Marking bug as Fixed and closing report.

Rik <rik5>
Group administrator
Tue 22 Nov 2022 04:46:49 PM UTC, comment #4: 

I just wanted to note that this bug report has spawned hours of work trying to understand, and then correct the behavior of the octave_stream class.  The bad news is that this code is very complicated, spread over multiple files, and because it is so brittle making even small changes breaks other bits of functionality.

The good news is that progress has been made and that a bare '-' character now issues an error for both %d and %f formats with the stream pointer correctly pointing to the source of the error.  See changesets http://hg.savannah.gnu.org/hgweb/octave/rev/d006285b7509 and http://hg.savannah.gnu.org/hgweb/octave/rev/2d38bc5a35db.

Rik <rik5>
Group administrator
Sun 20 Nov 2022 01:33:34 AM UTC, comment #3: 

the scanf parsing is complex and distributed across functions in libinterp and in liboctave.  Using the debugger, it appears that reading a double in liboctave is correctly setting the fail bit on the input stream.  However, back in libinterp the test for the fail condition is specifically disabled if the failure occurs at the EOF.


if (! is.eof () && is.rdstate () & std::ios::failbit)


Removing the eof() test with this patch makes things work for this bug report.


diff --git a/libinterp/corefcn/oct-stream.cc b/libinterp/corefcn/oct-stream.cc
--- a/libinterp/corefcn/oct-stream.cc
+++ b/libinterp/corefcn/oct-stream.cc
@@ -5042,7 +5042,8 @@ namespace octave

                     // If it looks like we have a matching failure, then
                     // reset the failbit in the stream state.
-                    if (! is.eof () && is.rdstate () & std::ios::failbit)
+                    //if (! is.eof () && is.rdstate () & std::ios::failbit)
+                    if (is.rdstate () & std::ios::failbit)
                       {
                         error (who, "format failed to match");
                         is.clear (is.rdstate () & (~std::ios::failbit));


But, this then causes a regression BIST to fail for bug #62723.

Overall, this is probably not too difficult to fix.  But, I think it means re-visiting bug #62723 and finding a different solution there.

Rik <rik5>
Group administrator
Fri 18 Nov 2022 08:02:00 PM UTC, comment #2: 

comentario nº1:

> This problem also happens if you replace 'n' with 'i', and you can also see what I think is a position error with '%d' in the following example:
>


> octave> [val, count, msg, pos] = sscanf ('1 2 3 -', '%d')
> val =
>
>    1
>    2
>    3
>
> count = 3
> msg = sscanf: format failed to match
> pos = 8


>
> What does Matlab do for this case?


Mmmm, in my case Octave returns:

>> [val, count, msg, pos] = sscanf ('1 2 3 -', '%d')
val =

   1
   2
   3

count = 3
msg =
pos = 8

The error message is not set. I'm running Octave 7.3.0 compiled by myself in Debian Sid

On the other hand, Matlab 9.2.0.538062 (R2017a) returns:

>> [val, count, msg, pos] = sscanf ('1 2 3 -', '%d')
val =
     1
     2
     3
count =
     3

msg =
    'Matching failure in format.'

pos =
     7


José Luis García Pallero <jgpallero>
Fri 18 Nov 2022 07:34:43 PM UTC, comment #1: 

This error happens because Octave is trying to parse "nan" and is detecting a conversion error (so only three values are returned) but failing to correctly set the error message.  You'll also see that the position is also not set correctly if you try


[val, count, msg, pos] = sscanf ('1 2 3 n', '%f')


This problem also happens if you replace 'n' with 'i', and you can also see what I think is a position error with '%d' in the following example:


octave> [val, count, msg, pos] = sscanf ('1 2 3 -', '%d')
val =

   1
   2
   3

count = 3
msg = sscanf: format failed to match
pos = 8


What does Matlab do for this case?

John W. Eaton <jwe>
Group administrator
Fri 18 Nov 2022 02:34:46 PM UTC, original submission:  

When I use the text string

>> a = '1 2 3 t';

in sscanf in order to extract the elements using '%f' I obtain obviously an error message due to the last element:

>> [elements,n,msg]=sscanf(a,'%f')
elements =

   1
   2
   3

n = 3
msg = sscanf: format failed to match


But if the last element is the letter 'n':

>> a = '1 2 3 n';
>> [elements,n,msg]=sscanf(a,'%f')
elements =

   1
   2
   3

n = 3
msg =

Only three elements are extracted, as expected, but the error message is not set, so this can not be used to check for errors (Matlab gets the error message). The proplem is present, I think, only with letter 'n'.

If I use '%d' as conversion there is no problem:

>> a = '1 2 3 n';
>> a = '1 2 3 n';
>> [elements,n,msg]=sscanf(a,'%d')
elements =

   1
   2
   3

n = 3
msg = sscanf: format failed to match


José Luis García Pallero <jgpallero>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #54066:  63383.diff added by rik5 (573B - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by jgpallero (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-03-02 rik5 StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2024-03-01 rik5 StatusFixed In Progress
        Open/ClosedClosed Open
    2022-12-06 rik5 Attached File- Added 63383.diff, #54066
    2022-11-25 rik5 StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2022-11-22 rik5 StatusConfirmed In Progress
    2022-11-18 jwe StatusNone Confirmed
        Releaseother dev

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code