bugGNU Octave - Bugs: bug #47741, The scanf family of functions...

 
 

bug #47741: The scanf family of functions cannot parse 64bit hex numbers on Windows and 32bit Linux

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Wed 20 Apr 2016 10:27:06 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 11 May 2016 08:26:29 PM UTC, comment #8: 
Mike Miller <mtmiller>
Group Member
Tue 26 Apr 2016 05:37:55 PM UTC, comment #7: 

I tried an mxe-build with your patch (and your patch for bug #47759) on Windows 10 with the following results:

>> sscanf('3000000000', '%d')
ans =  2147483647
>> sscanf('3000000000', '%ld')
ans =  3000000000
>> sscanf('ffffffffffffffff', '%lx')
ans =   1.84467440737096e+019


So that issue seems to be resolved on Windows, too.

With those two patches the examples from comment #4 give

>> sscanf('1000000000', '%d')
ans =   1.0000e+009
>> class(ans)
ans = double
>> sscanf('3000000000', '%d')
ans =   2.1475e+009
>> class(ans)
ans = double
>> sscanf('6000000000', '%d')
ans =   2.1475e+009
>> class(ans)
ans = double
>> sscanf('6000000000', '%ld')
ans =   6.0000e+009
>> class(ans)
ans = double
>> sscanf('ffff', '%x')
ans =  65535
>> class(ans)
ans = double
>> sscanf('ffff', '%lx')
ans =  65535
>> class(ans)
ans = double
>> sscanf('6000000000 6000000000', '%ld %d')
ans =

  6.0000e+009
  2.1475e+009

>> class(ans)
ans = double
>> sscanf('6000000000 ffff', '%ld %lx')
ans =

  6.0000e+009
  6.5535e+004

>> class(ans)
ans = double


So the only remaining difference to Matlab is the class for conversions with only "%l[oux]" that should be uint64 and for only "%l[di]" that should be int64.

Markus Mützel <mmuetzel>
Group administrator
Fri 22 Apr 2016 03:05:06 AM UTC, comment #6: 

Markus, see also bug #47759 about the integer overflow compatibility issue, I have a Matlab question there if you're willing to run some tests.

Mike Miller <mtmiller>
Group Member
Thu 21 Apr 2016 11:22:49 PM UTC, comment #5: 

I've got a partial solution for the main issue in this bug report, see attached.

Do we want to make %h[dioux] mean exactly a 16-bit integer as Markus recommended in the original post? If we have to make "l" mean a 64-bit integer for Matlab compatibility, then I kind of like Markus' idea of making the "h" modifier mean exactly 16-bit and no modifier exactly 32-bit for symmetry.

There are other related issues that should probably be reported separately, such as scanf should return an array of type int64/uint64 if the only conversion is %l[dioux]. And that Matlab returns a saturated integer when the number read is out of range, while Octave aborts conversion at that point.

(file #36977)

Mike Miller <mtmiller>
Group Member
Thu 21 Apr 2016 08:10:48 AM UTC, comment #4: 

Some examples in Matlab on Windows:

>> sscanf('1000000000', '%d')
ans =
   1.0000e+09
>> class(ans)
ans =
double
>> sscanf('3000000000', '%d')
ans =
   2.1475e+09
>> class(ans)
ans =
double
>> sscanf('6000000000', '%d')
ans =
   2.1475e+09
>> class(ans)
ans =
double
>> sscanf('6000000000', '%ld')
ans =
           6000000000
>> class(ans)
ans =
int64
>> sscanf('ffff', '%x')
ans =
       65535
>> class(ans)
ans =
double
>> sscanf('ffff', '%lx')
ans =
                65535
>> class(ans)
ans =
uint64
>> sscanf('6000000000 6000000000', '%ld %d')
ans =
   1.0e+09 *
    6.0000
    2.1475
>> class(ans)
ans =
double
>> sscanf('6000000000 ffff', '%ld %lx')
ans =
   1.0e+09 *
    6.0000
    0.0001
>> class(ans)
ans =
double


The same in Octave 4.0.1 on Windows:

>> sscanf('1000000000', '%d')
ans =   1.0000e+009
>> class(ans)
ans = double
>> sscanf('3000000000', '%d')
ans = [](0x1)
>> class(ans)
ans = double
>> sscanf('6000000000', '%d')
ans = [](0x1)
>> class(ans)
ans = double
>> sscanf('6000000000', '%ld')
ans = [](0x1)
>> class(ans)
ans = double
>> sscanf('ffff', '%x')
ans =  65535
>> class(ans)
ans = double
>> sscanf('ffff', '%lx')
ans =  65535
>> class(ans)
ans = double
>> sscanf('6000000000 6000000000', '%ld %d')
ans = [](0x1)
>> class(ans)
ans = double
>> sscanf('6000000000 ffff', '%ld %lx')
ans = [](0x1)
>> class(ans)
ans = double


Markus Mützel <mmuetzel>
Group administrator
Wed 20 Apr 2016 06:42:53 PM UTC, comment #3: 

For Matlab compatibility, from the help for fscanf, it seems they do not support any length modifiers other than "l". They list the following two-character conversions as supported: %ld, %li, %lu, %lo, %lx.

They also say that those conversions read 64-bit values, so I guess that is a change we'll have to make.

As for the return value, the help says that if all conversions are signed/unsigned 64-bit integers (meaning all %l+something), then the returned array is of type int64 or uint64.

https://www.mathworks.com/help/matlab/ref/fscanf.html

Beyond that, any other length specifiers we support in addition, like "%hd" or "%jx" or "%lf" is up to us.

I would be curious what Matlab returns for


sscanf('1000000000', '%d')
sscanf('3000000000', '%d')
sscanf('6000000000', '%d')


Mike Miller <mtmiller>
Group Member
Wed 20 Apr 2016 05:55:41 PM UTC, comment #2: 

But what does Matlab do?  If it ignores the size specifiers, then we will probably have to do the same.  Also, what type of value is returned?  Double?  int64?  Does the type depend on the format and/or value (that seems bad, but who knows)?

John W. Eaton <jwe>
Group administrator
Wed 20 Apr 2016 05:37:52 PM UTC, comment #1: 

Good catch about the length modifier actually being used. I read the help text but did not test myself, so that is inconsistent.

My preference would be that these functions map exactly to the way they work in C/C++, which means that the length modifier characters are platform-dependent rather than operating on sized integers.

If we're going to ignore all length modifiers, then I'm fine with that, and we always read a intmax_t/uintmax_t.

If we want to actually use the length modifiers, then I would expect them to work exactly as in the library functions, so "hh" is char, "h" is short int, "l" is long int, "ll" is long long int. And I would like to add support for the standard "j", "t", and "z" modifiers as well, and the GNU extensions "L" and "q" (both are aliases for "ll").

Mike Miller <mtmiller>
Group Member
Wed 20 Apr 2016 10:27:06 AM UTC, original submission:  

As reported by Francesco Potortì in bug #47690 (comment #1 there), Octave cannot read hex strings longer than 8 chars with sscanf. I can confirm that bug.

The underlying problem seems to be in oct-stream.cc where int, short int and long int are used for reading hex numbers. The size of these types are different on different platforms.

As pointed out by Mike Miller (comment #5 there), Octave explicitly says in the section "Input Conversion Syntax" that the type modifier is ignored by the scanf family of functions.

That does not seem to be the case. In Octave 4.0.1:

>> sscanf ("ffffffff", "%x")
ans =   4.2950e+009
>> sscanf ("ffffffff", "%hx")
ans = [](0x1)
>> sscanf ("ffff", "%hx")
ans =  65535


At least some of the length modifiers (namely "h" and "l") are recognized. Either the documentation or the function should be adapted.

The easier (preferred) solution would be to change the function to fit the documentation, strip off the length modifier handling from oct-stream.cc and always read with maximum possible integer size ("j").
However, I do not know which side effects that could have.
Maybe it would be possible (and saver) to pass a parameter to octave_base_stream to disable length modifiers optionally.

On the other hand, the function could be extended to handle all possible length modifiers from C99 (see e.g. [1]).
In that case, the following questions arise:
Do we want native results on all platforms? The results would differ for some integer types on different platforms:
"%lx" or "%x" would result in a 32bit unsigned integer on 32bit Linux and 32/64bit Windows, but in 64bit unsigned integer on 64bit Linux, like it is now.
Or do we want to "fix" each modifier to a certain length? E.g.:
- "l" --> always 64 bits (Is "long long int" always 64bit on all platforms?)
- "h" --> always 16 bits
- no modifier --> always 32 bits
That would give consistent results on different platforms, but would need some conversion of modifiers for certain (or for the sake of simplicity for all?) platforms (convert "l" to "ll" or add "ll").

The same goes for signed as well as unsigned integers.

The latter (expanding octave_base_stream and giving different results on different platforms or messing with modifiers) seems less appealing to me.

[1] http://www.cplusplus.com/reference/cstdio/scanf/

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #36977:  bug47741.diff added by mtmiller (2KiB - text/x-diff)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-05-11 mtmiller StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2016-04-21 mtmiller StatusConfirmed In Progress
    2016-04-21 mtmiller Attached File- Added bug47741.diff, #36977
    2016-04-20 mtmiller StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code