bugGNU Octave - Bugs: bug #30817, Ouput display: truncation vs....

 
 

bug #30817: Ouput display: truncation vs. rounding

Submitter:  Lukas Reichlin <paramaniac>
Submitted:  Thu 19 Aug 2010 01:12:34 PM UTC
   
 
Category:  None Severity:  2 - Minor
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Wont Fix Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 3.2.4 Operating System:  * Mac OS
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 21 Aug 2010 02:54:06 AM UTC, comment #2: 

Octave inherits its output behavior from the C++ standard libraries.  In this case, C++ does not round when the last digit is exactly 5.  I'm attaching a short program, tst_prec.cpp which demonstrates this behavior. 


g++ tst_prec.cpp
./a.out


This is not something that we can fix in Octave.

(file #21267)

Rik <rik5>
Group administrator
Thu 19 Aug 2010 08:17:34 PM UTC, comment #1: 

What you should realize is that when you type 0.999995, the value is already rounded at that point to nearest (or, in general, a close-by) floating-point number. It has to be; there is no other way. So you may be deceived at the spot:

octave:51> x = 9007154218744718 * 2^-53
x =  0.99999
octave:52> x == 0.999995
ans =  1

now with aid of PARI:

? x = 9007154218744718 * 2^-53
%1 = 4503577109372359/4503599627370496
? x - 99999/10^5 < 1 - x
%2 = 1

so, as you see, Octave's result is mathematically correct. Such are the subtleties of binary floating point values.

Jaroslav Hajek <highegg>
Thu 19 Aug 2010 01:12:34 PM UTC, original submission:  

I consider the display of result 53 as an inconsistency or a (cosmetic) bug because round (0.5) is 1. Rounding up starts at 0.50, not at 0.51.
(Only the first digit after truncation is significant for rounding)

octave:57> 0.999996
ans =  1.00000
octave:53> 0.999995
ans =  0.99999  # this should be 1.00000 like the previous result
octave:54> 0.9999951
ans =  1.00000
octave:55> 0.9999950
ans =  0.99999  # this should be rounded up too
octave:56> 0.9999949
ans =  0.99999


Results from the "dark side":
0.99995

ans =

   1.0000  % this is correct

0.999949

ans =

   0.9999

0.999950

ans =

   1.0000

0.999951

ans =

   1.0000

0.99996

ans =

   1.0000



Cheers,
Lukas

Lukas Reichlin <paramaniac>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #21267:  tst_prec.cpp added by rik5 (429B - text/x-c++src)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by highegg (Posted a comment)
  • -email is unavailable- added by paramaniac (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-08-27 highegg Open/ClosedOpen Closed
    2010-08-21 rik5 Attached File- Added tst_prec.cpp, #21267
        Severity3 - Normal 2 - Minor
        StatusNone Wont Fix
        Summarytruncation vs. rounding Ouput display: truncation vs. rounding

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code