bugGNU Octave - Bugs: bug #36133, num2str fails to switch from...

 
 

bug #36133: num2str fails to switch from integer output to exponential notation for large values

Submitter:  Rik <rik5>
Submitted:  Mon 09 Apr 2012 03:08:33 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 13 Jul 2018 06:08:17 PM UTC, comment #18: 

OK.  I noticed there is a PRECISION input variable.  Some of the same issues might apply there:


>> astr = num2str(a,16)
astr = 1.234567890123457e+19
>> astr = num2str(a,17)
astr = 1.2345678901234567e+19
>> astr = num2str(a,18)
astr = 1.23456789012345672e+19
>> astr = num2str(a,19)
astr = 1.234567890123456717e+19
>> astr = num2str(a,20)
astr = 12345678901234567168


Dan Sebald <sebald>
Fri 13 Jul 2018 05:59:52 PM UTC, comment #17: 

I already changed the default precision that Octave saves data files for exactly this reason.  See


changeset:   23556:31b1ef1ee585
user:        Rik <rik@octave.org>
date:        Wed Jun 07 12:29:23 2017 -0700
summary:     Fix Octave's -text format for saving data to make it lossless for IEEE-754 doubles.


For num2str, I think user's are going to want compatibility so we should leave this at 16 digits.


Rik <rik5>
Group administrator
Fri 13 Jul 2018 05:20:08 PM UTC, comment #16: 

That was easy, good.

I wonder if there should be a test to check the IEEE 754 rule I listed below regarding the conversion to a string and back from a string.  (Comparing against an actual string might already achieve this.)  That is, there should be some mode in which printing out the numbers and then bringing them back into Octave results in the exact same internal representation.  Here's an example:


>> a = 12345678901234567890
a =    1.2346e+19
>> astr = num2str(a)
astr = 1.234567890123457e+19
>> eval(['b = ' astr])
b =    1.2346e+19
>> a == b
ans = 0


Note how b no longer equals a.  If someone were to save their data into a file using num2str in order to make is "portable", s/he might not be happy about the above fact.

The minor capping change from 16 to 17 digits fixes this.  I tried the following:


diff --git a/scripts/general/num2str.m b/scripts/general/num2str.m
--- a/scripts/general/num2str.m
+++ b/scripts/general/num2str.m
@@ -107,7 +107,7 @@ function retval = num2str (x, arg)
         if (ndgt > 15 || any (x(valid) != fix (x(valid))))
           ## Floating point input
           ndgt = max (ndgt + 5, 5);   # Keep at least 5 significant digits
-          ndgt = min (ndgt, 16);      # Cap significant digits at 16
+          ndgt = min (ndgt, 17);      # Cap significant digits at 16
           fmt = sprintf ("%%%d.%dg", ndgt+7, ndgt);
         else
           ## Integer input


and the test above passes:


>> astr = num2str(a)
astr = 1.2345678901234567e+19
>> eval(['b = ' astr])
b =    1.2346e+19
>> a == b
ans = 1


Note the extra digit of resolution.

Is there something to take into account that the number of digits resolution can vary from 15-17?  Is "shimming" still needed when the number of valid resolution digits is 15 for IEEE 754 when typically we'd want to ensure 17 digit resolution can be addressed?

Dan Sebald <sebald>
Fri 13 Jul 2018 04:17:12 PM UTC, comment #15: 

The fix was actually quite simple.  Octave now checks whether there are more than 15 digits to display and if there are then it uses the floating point format rather than integer format.  All of the BIST tests now pass.  I have marked them as passing so if they start failing again it will show up as a regression.  For all of the above, see this cset (https://hg.savannah.gnu.org/hgweb/octave/rev/e55fb9685803).

Marking as fixed and closing report.

Rik <rik5>
Group administrator
Fri 13 Jul 2018 03:55:17 PM UTC, comment #14: 

Matlab behavior has changed since this bug was originally reported.  This is all to the good because sprintf is now compatible without us having to change anything.  Mixed floating point / integer arrays are also displayed correctly.

The remaining difference is integer arrays with large values.  I have re-title the bug report to reflect that.  I'm also changing the classification to "Matlab Compatibility".

I pushed a changeset which updates the BIST tests for num2str to focus on this remaining difference.  See https://hg.savannah.gnu.org/hgweb/octave/rev/8f8d3fef29a8.

Rik <rik5>
Group administrator
Fri 13 Jul 2018 07:05:20 AM UTC, comment #13: 

I noticed this known bug in the buildbot results.  It doesn't seem noted in any of the discussion posts, so I thought I'd mention that the Octave result for Rik's example:


>> num2str ([2.1, 1e23, pi])
ans = 2.1  9.999999999999999e+22      3.141592653589793


is the same as what A.J. has shown for Matlab.  That happens when Octave is attempting to find a reasonable display width and the number magnitudes span a very large range.

Otherwise, Octave is doing something different for narrow range, e.g.,:


>> num2str ([1e22 1e23 1e24])
ans = 10000000000000000000000    99999999999999991611392   999999999999999983222784


So it would seem the same underlying exponent display mechanism is already present, but not necessarily active by default.  And it isn't quite the same as displaying the values:


>> [2.1, 1e23, pi]
ans =

   2.100000000000000e+00   9.999999999999999e+22   3.141592653589793e+00



However, this probably doesn't address the "shim" issue that Rik mentioned in Comment #8.  I don't know what one can consider valid or invalid.  Using this rule from Wikipedia:

"
If a decimal string with at most 15 significant digits is converted to IEEE 754 double-precision representation, and then converted back to a decimal string with the same number of digits, the final result should match the original string. If an IEEE 754 double-precision number is converted to a decimal string with at least 17 significant digits, and then converted back to double-precision representation, the final result must match the original number.[1]
"

The Matlab rounding seems to stay within the rule.  (As does the Octave non-rounding.)  It would seem that those last decimals outside resolution are extraneous.  Perhaps what those decimals turn out to be is compiler dependent, or CPU dependent, or library dependent.  The rounding of the 18th significant digit might be a way to make all platforms have a consistent output for num2str().

I don't see it as too big of an issue.  However, if Octave is left as is, then this can't be the test:


ASSERT errors for:  assert (num2str (1e23),"100000000000000000000000")
  Location  |  Observed  |  Expected  |  Reason
     []      99999999999999991611392 100000000000000000000000   Strings don't match


because the result will never be 100000000000000000000000.  And changing it to


assert (num2str (1e23),"99999999999999991611392")


would likely fail on some systems.  Any test should really only focus on the 15-17 significant digits, whether that be the test itself (by somehow extracting the first 17 characters) or by internally forcing the remaining extraneous digits to something known (i.e., 0).

One last note.  Wikipedia mentions that for some C/C++ compilers there is a possible issue of double rounding with x86 32-bit systems.

Dan Sebald <sebald>
Sun 24 Jun 2018 09:47:07 PM UTC, comment #12: 

Matlab's behavior has changed since R2014b. Here's examples from R2018a:


>> sprintf ('%30.0f', 1e25)
ans =
    '    10000000000000000905969664'



>> num2str(2^512)
ans =
    '1.34078079299426e+154'
>> sprintf ('%.0f', 2^512)
ans =
    '13407807929942597099574024998205846127479365820592393377723561443721764030073546976801874298166903427690031858186486050853753882811946569946433649006084096'


>> for i = 1:25; x = 10^i; disp (num2str (x)); end
10
100
1000
10000
100000
1000000
10000000
100000000
1000000000
10000000000
100000000000
1000000000000
10000000000000
100000000000000
1000000000000000
1e+16
1e+17
1e+18
1e+19
1e+20
1e+21
1e+22
9.999999999999999e+22
1e+24
1e+25



>> num2str ([2.1, 1e23, pi])
ans =
    '2.1  9.999999999999999e+22      3.141592653589793'


Andrew Janke <apjanke>
Wed 25 Apr 2018 10:40:52 PM UTC, comment #11: 

@Joe: Could you run this additional test and report back the result?


x = [2.1, 1e23, pi];
y = num2str (x)



Rik <rik5>
Group administrator
Wed 25 Apr 2018 09:43:54 PM UTC, comment #10: 

Results from Matlab will be interesting, but I doubt that they test that each number to be printed is an exact power of two.  There are 1024 of those numbers compared to 2^53 (9e15) numbers representable in the mantissa.  It really, really fails the 80/20 rule to worry about them.

According to Matlab's documentation for both sprintf and num2str, they are aware of the issue


Note

If you specify a precision operator for floating-point values that exceeds the precision of the input numeric data type, the results might not match the input values to the precision you specified. The result depends on your computer hardware and operating system.


Also, according to the documentation for num2str


s = num2str(A) converts a numeric array into a character array that represents the numbers. The output format depends on the magnitudes of the original values


That's fine, but my guess is that Matlab just uses a form of the %g format for both floating point and integer numbers.  Octave, however, tries to switch format based on whether the input is an integer.  The code is


if (any (x(valid) != fix (x(valid))))
  ## Floating point input
  ndgt = max (ndgt + 5, 5);   # Keep at least 5 significant digits
  ndgt = min (ndgt, 16);      # Cap significant digits at 16
  fmt = sprintf ("%%%d.%dg", ndgt+7, ndgt);
else
  ## Integer input
  ndgt += 3;
  if (any (! valid))
    ndgt = max (ndgt, 5);     # Allow space for Inf/NaN
  endif
  ## FIXME: Integers must be masked to show only 16 significant digits
  ##        See test case for bug #36133 below
  fmt = sprintf ("%%%d.0f", ndgt);
endif


I bet changing the integer case would fix some of this.

Rik <rik5>
Group administrator
Wed 25 Apr 2018 07:44:04 PM UTC, comment #9: 

15-17 decimal digits is true for arbitrary numbers, but powers of 2 can be converted to decimal exactly, up to 2^1023.  So limiting to 15-17 decimal digits doesn't make sense for ALL 64-bit IEEE floating point values.

But yes, we should be switching to exponential display using some kind of rules that produce "pleasing" results.  One problem is deciding what those rules should be.

What does Matlab do for num2str (2^512)?  What about sprintf ('%.0f', 2^512)?

Also, if we "clean up" the output form our printf functions, then  they will differ from what people will get when they use C++ streams or C stdio functions.  Is that really what we want?  If so, then we might want to make our functions easily available for use in .oct files.

A related issue is the max precision problem in bug #53456.

John W. Eaton <jwe>
Group administrator
Wed 25 Apr 2018 04:58:54 PM UTC, comment #8: 

Adding in jwe because the behavior is fascinating.  From Hartmut's tests in comment #3, the Matlab sprintf function returns


sprintf ('%30.0f', 1e25)
ans =
    10000000000000001000000000


where the '1' in the sectond position is very real.

Octave's answer for the same code is


ans =     10000000000000000905969664


If I align the two


    10000000000000001000000000     # Matlab
    10000000000000000905969664     # Octave


it seems pretty clear what Matlab has done.  First, Octave uses the C library routine for sprintf so we get whatever default behavior is implemented when the precision exceeds that of the underlying number.  But Matlab has written a shim layer for sprintf which calls the sprintf library, but then cleans up the result.  However, they haven't been too clever about it.  IEEE-854 doubles have a variable precision of 15-17 decimal digits.  The extra '1' occurs because they are rounding the digit '9' which is the 18 significant figure and not valid.

In any case, once Matlab has printed 17 digits, it pads the rest of the string with zeros.  This is pretty easy to do in C++ with the substr() function or operator[] access and a for loop.

So should Octave do something similar?  And if so, where should it be done?  It could be done just in our sprintf routine, but then other actions like printf would diverge in behavior.  That means it would probably be better to do it in octave::stream.


Rik <rik5>
Group administrator
Wed 25 Apr 2018 04:33:50 PM UTC, comment #7: 

@Joe: Are you proposing that the two assert statements be added as BIST tests to num2str.m?

Also, can you determine when Matlab switches from decimal to exponential notation?

Try running this code:


for i = 1:25
  x = 10^i;
  str = num2str (x);
  disp (str);
end


Octave never switches over to exponential notation


10
100
1000
10000
100000
1000000
10000000
100000000
1000000000
10000000000
100000000000
1000000000000
10000000000000
100000000000000
1000000000000000
10000000000000000
100000000000000000
1000000000000000000
10000000000000000000
100000000000000000000
1000000000000000000000
10000000000000000000000
99999999999999991611392
999999999999999983222784
10000000000000000905969664



Rik <rik5>
Group administrator
Tue 24 Apr 2018 08:14:15 PM UTC, comment #6: 

Oh sorry that's me creating a new one to make sure I don't get confused. I also realized I didn't double check my outputs with Matlab. It my comment should have read:

%! assert (num2str (1e23), "9.999999999999999e+22");
%! assert (num2str (1e25), "1e+25");

Also int2str has the same bug, how can I link that function to this bug?

Joe <spikel46>
Tue 24 Apr 2018 07:42:17 PM UTC, comment #5: 

Your code uses num2strE which is not an Octave core function.  Try 'which num2strE' to find the location.  You will need to report a bug against whoever wrote that function.

Rik <rik5>
Group administrator
Tue 24 Apr 2018 07:21:35 PM UTC, comment #4: 

I think the test should read:

%! assert (num2strE (1e23), "1e+23");
%! assert (num2strE (12345678901234567890), "1.234567890123457e+20");

Also int2str has the same bug, how can I link that function to this bug?

Joe <spikel46>
Mon 21 Nov 2016 01:05:33 PM UTC, comment #3: 

Matlab R2014b:


>> sprintf ('%30.0f', 1e25)
ans =
    10000000000000001000000000

>> sprintf ('%30d', 1e25)
ans =
                  1.000000e+25

>> num2str (1e25)
ans =
1e+25

(The second 1 in the first result is real.)

Hartmut <hardy>
Sun 20 Nov 2016 03:17:53 PM UTC, comment #2: 

For comparison, could someone try the following in Matlab?


sprintf ("%30.0f", 1e25)
sprintf ("%30d", 1e25)
num2str (1e25)


Rik <rik5>
Group administrator
Sat 19 Nov 2016 09:24:37 PM UTC, comment #1: 

This is still present in Octave 4.2.0.

Hartmut <hardy>
Mon 09 Apr 2012 03:08:33 AM UTC, original submission:  

Example code:



num2str (1e25)
ans = 10000000000000000905969664


This should be '1' with 25 zeros after it.

Matlab correctly masks large integer inputs to just 16 significant digits.

For something really wacky try


num2str (realma)



Rik <rik5>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5
  • -email is unavailable- added by spikel46 (Posted a comment)
  • -email is unavailable- added by hardy (Posted a comment)
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by rik5 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-07-13 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2018-07-13 rik5 Item GroupInaccurate Result Matlab Compatibility
        Summarynum2str displays more than 16 significant digits for large integer inputs num2str fails to switch from integer output to exponential notation for large values
    2018-04-25 rik5 Carbon-Copy- Added jwe
    2014-01-19 mtmiller CategoryNone Octave Function
        Operating SystemGNU/Linux Any

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code