bugGNU Octave - Bugs: bug #41359, Output formatting inconsistency...


bug #41359: Output formatting inconsistency for integers > 2^52

Submitter:  None
Submitted:  Mon 27 Jan 2014 02:40:44 AM UTC
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Confirmed Assigned to:  None
Originator Name:  G. Golden Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * dev
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup

Thu 15 Dec 2016 11:09:12 AM UTC, comment #5: 

This issue is still present in Octave 4.2.0.

Hartmut <hardy>
Mon 27 Jan 2014 04:45:42 PM UTC, comment #4: 

I haven't looked into how libc handles this with printf, but sure it could be worth looking into.

And as to the savannah markup, don't worry about it, we are used to it, we've all messed up the markup from time to time. Reposting to fix markup mistakes is fine.

Mike Miller <mtmiller>
Group Member
Mon 27 Jan 2014 02:57:22 PM UTC, comment #3: 

verbose -> verbatim on previous comment, sry.

Which begs the question: Is there a preview feature for submitting comments? Didn't see anything in a brief look thru the Savannah help doc, but perhaps just missed it...

Glenn Golden <grepfor>
Mon 27 Jan 2014 02:44:12 PM UTC, comment #2: 

Interesting, I see what you mean. Need an integer test that's independent of the float EPS.

Fwiw -- perhaps nothing, just an observation -- printf %.16g does get it right:


>   a = 2^52 + (-5:5);
>   printf("a: %.16g\n", a)

a: 4503599627370491
a: 4503599627370492
a: 4503599627370493
a: 4503599627370494
a: 4503599627370495
a: 4503599627370496
a: 4503599627370497
a: 4503599627370498
a: 4503599627370499
a: 4503599627370500
a: 4503599627370501

So might be worth a look at how the integer test is being handled there.

Glenn Golden <grepfor>
Mon 27 Jan 2014 03:15:57 AM UTC, comment #1: 

Yes, you are right that the values should be representable in 16 decimal digits. The basis of the decision whether a number can be displayed as an integer or not is failing for these cases. The output formatter then believes that the value is not strictly an integer and displays what you are seeing.

Here are essentially the computations that the formatter is doing to determine whether the value is an integer or not:

> a = 2^52;
> a == floor (a + .5)
ans =  1
> a-1 == floor (a-1 + .5)
ans =  1
> a+1 == floor (a+1 + .5)
ans = 0
> a+2 == floor (a+2 + .5)
ans =  1

Since we've hit the limit of the significand, adding 0.5 rolls over into the next bit position, floor gives the wrong result, and it appears that a+1 is not an integer according to this formula.

Mike Miller <mtmiller>
Group Member
Mon 27 Jan 2014 02:40:44 AM UTC, original submission:  

In "format long" display mode, with output_precision set to 16, odd integers between 2^52 and 2^53 -- all of which are exactly representable in IEEE-754 double precision, and have 16 or fewer decimal digits -- are nevertheless displayed in E format.


  > format long;
  > output_precision(16);

  > a = 2^52;

  # All values below 2^52 display as expected, i.e. as 16 digit integers:
  > a-3         # ans =  4503599627370493
  > a-2         # ans =  4503599627370494
  > a-1         # ans =  4503599627370495
  > a           # ans =  4503599627370496

  # But odd values between 2^52+1 and 2^53 display alternately in fixed and
  # exponential representation:
  > a+1         # ans =  4.503599627370497e+15
  > a+2         # ans =  4503599627370498
  > a+3         # ans =  4.503599627370499e+15

Note that the displayed values for the odd integers above 2^52 are correct (and exact) but it seems inconsistent that they should be displayed in E format since they can be represented exactly as 16 digit integers.

The same behavior is observed for negative integers in the complementary range below zero.

NOTE: This is not a hardware representation issue, at least under the assumption of IEEE-754 compliant hardware, which was the case for the examples shown above: An IEEE-754 double precision float exactly represents all integers on [-2^53, +2^53], all of which fit into 16 digits.  Approximation error begins at 2^53+1. This can be demonstrated by bumping output_precision to 17:

  > output_precision(17)
  >   2^53
  ans =  9007199254740992       # Error = 0
  >   2^53+1
  ans =  9007199254740992       # Error = 1 LSB of significand, as expected
  >   2^53+2
  ans =  9007199254740994       # Error = 0
  >   2^53+3
  ans =  9007199254740996       # Error = 1 LSB of significand, as expected

and so on, all of which is as expected for IEEE-754 hardware.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by hardy (Posted a comment)
  • -email is unavailable- added by grepfor (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.


    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-01-27 mtmiller StatusNone Confirmed
        Release3.6.4 dev

    Back to the top

    Powered by Savane 3.13-b921.
    Corresponding source code