bugGNU Octave - Bugs: bug #53683, numerical output format for float...

 
 

bug #53683: numerical output format for float values with magnitude greater than precision is wrong

Submitter:  Dan Sebald <sebald>
Submitted:  Wed 18 Apr 2018 12:46:11 AM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Regression
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 20 Apr 2018 06:41:02 AM UTC, comment #9: 

This is bigger than I initially thought.  At first I was thinking a one line fix for 4.4, but there is more to it.  Yes, there is one detail that just doesn't make sense with regard to common definitions.  Here is fixed point with 5 fractional digits (and is sort of what what Matlab is doing with its "4 digits after the decimal point"):


       0.00333
       0.03333
       0.33333
       3.33333
      33.33333
     333.33333
    3333.33333
   33333.33333
  333333.33333
 3333333.33333
33333333.33333


The following would be 5 significant digits, something similar to floating point


       0.0033333
       0.033333
       0.33333
       3.3333
      33.333
     333.33
    3333.3
   33333
  333330
 3333300
33333000


Then for the sake of space, at either extreme one might use engineering notation rather than such a long string of numerals. 

Octave is following a floating point progression at the start, then transitions to a fixed point progression at the following point:


ans =  3333.3
ans =  33333.33333


Why should the larger number get more fractional resolution?  That's the one point I disagree with.  Just how many significant digits there are, whether it is fixed point or floating point is all a matter of choice.

I can see where this gets more complicated, if one considers groups of numbers.  For example, the following illustrates that it is the smallest number of the group that governs the overall fractional digits:


octave:7> rand(10,1) * 3e2
ans =

   195.96153
    32.12849
   125.35201
    91.77371
     0.76179
   128.33099
    14.58701
   257.34665
   151.51730
   209.94001

octave:8> rand(10,1) * 3e2
ans =

   259.38
   281.53
   110.50
   278.94
   216.12
   116.73
   173.32
   129.39
   134.56
   288.86

octave:9> rand(10,1) * 3e2
ans =

   233.796
   131.497
   162.690
   100.161
   162.110
    25.142
   122.527
   267.176
    49.537
    79.511


Or more precisely, governed by whatever number has the most fractional digits.  In the following there are numbers greater than that transition point mentioned above; they therefore have five digits to the right of the decimal point):


octave:40> rand(10,1) * 3e4
ans =

    7900.99571
    6466.43122
   18928.80875
   10963.55305
   12949.62312
   19015.19955
   22253.92189
   12301.65306
   20224.36300
   22761.48407


Dan Sebald <sebald>
Thu 19 Apr 2018 09:39:37 PM UTC, comment #8: 

I'm not doing anything more about this for 4.4.

For 5.0, maybe we should decide what behavior we really want to have, define that precisely, then try to make it happen.

John W. Eaton <jwe>
Group administrator
Thu 19 Apr 2018 09:38:25 PM UTC, comment #7: 

I agree that the use of the terms precision or "significant figures" is sloppy, so the documentation could use some work.

But the values shown aren't wrong, are they?  I don't think we are displaying more digits than are valid for the computed values.

John W. Eaton <jwe>
Group administrator
Thu 19 Apr 2018 08:13:32 PM UTC, comment #6: 

Well, I'd say that current behavior is wrong, no matter the interpretation.  I'm not a real stickler for exact compatibility, but the issue here is that Matlab appears to be saying nothing about "precision" or (what Octave states in the documentation) "with 5 significant figures".  To me, "significant figures" can only be interpreted as precision, and Octave is sort of doing that in the first half of the progression below.  Anyway, Matlab just says "4 digits after the decimal point".  Perhaps once that no longer makes reasonable sense, then it switches to exponent notation.

Dan Sebald <sebald>
Thu 19 Apr 2018 07:07:08 PM UTC, comment #5: 

After reverting the changeset mentioned in comment #3, the behavior is


octave:1> for i = -2:8; 10^i/3, endfor
ans =  0.0033333
ans =  0.033333
ans =  0.33333
ans =  3.3333
ans =  33.333
ans =  333.33
ans =  3333.3
ans =  33333.33333
ans =  333333.33333
ans =  3333333.33333
ans =  33333333.33333


so that's not it.

What's weird about the Matlab behavior is that it sometimes shows just a few non-zero digits, but always 4 after the decimal until it would need more than 7.  Whatever.  The "best" behavior here is pretty subjective.

John W. Eaton <jwe>
Group administrator
Wed 18 Apr 2018 10:24:35 AM UTC, comment #4: 

fyi matlab 2017a switches to scientific notation:


>> for i = -2:8; 10^i/3, end
ans =
    0.0033
ans =
    0.0333
ans =
    0.3333
ans =
    3.3333
ans =
   33.3333
ans =
  333.3333
ans =
   3.3333e+03
ans =
   3.3333e+04
ans =
   3.3333e+05
ans =
   3.3333e+06
ans =
   3.3333e+07


Nicholas Jankowski <nrjank>
Group Member
Wed 18 Apr 2018 01:46:58 AM UTC, comment #3: 

Ah, OK, thanks for the clarification.  This then probably goes back to this changeset:


Changeset:
24789 (a4d4ec566fd7) improve formatting of large values in pr-output functions …
http://hg.savannah.gnu.org/hgweb/octave/rev/a4d4ec566fd7

I'll leave this for someone else to investigate.

Dan Sebald <sebald>
Wed 18 Apr 2018 01:21:06 AM UTC, comment #2: 

In previous versions of Octave:


for i = -2:8; 10^i/3, endfor
ans =  0.0033333
ans =  0.033333
ans =  0.33333
ans =  3.3333
ans =  33.333
ans =  333.33
ans =  3333.3
ans =    3.3333e+04
ans =    3.3333e+05
ans =    3.3333e+06
ans =    3.3333e+07


I believe Matlab also switches to scientific notation when the range of the value exceeds the output precision setting.

Mike Miller <mtmiller>
Group Member
Wed 18 Apr 2018 01:09:27 AM UTC, comment #1: 

Here's a changeset that will address this bug, but the fix may not be 100% accurate as to what the intended behavior is.  It procudes


octave:1> for i=-2:8
>   10^i / 3
> endfor
ans =  0.0033333
ans =  0.033333
ans =  0.33333
ans =  3.3333
ans =  33.333
ans =  333.33
ans =  3333.3
ans =   33333
ans =  333333
ans = 3333333
ans = 33333333


which shows how the right side digits is eventually discarded.  Note in the changeset that I actually chose the ld and rd to reflect the scientific or engineering understanding of "precision", not the computer/C interpretation.  Here's the formula change:


       if (digits > 0)
         {
-          ld = digits;
-          rd = (prec > digits ? prec - digits : prec);
+          ld = (prec > digits ? digits : prec);
+          rd = (prec > digits ? prec - digits : 0);
         }


where my expectation was that ld limited to prec (i.e., 5) would create

ans = 33333000

rather than

ans = 33333333

as an example; but obviously it doesn't do that.  In some sense, what I intended could be a desirable behavior for those who want to present data in a paper or something and limit the precision to, say, three digits without having to do the rounding manually.  Do something manually always opens the door for mistakes so why not let the computer do the rounding?

Nonetheless, I suspect the that ld precision is interpreted in a C sense at the following point in the code:


template <typename T>
std::ostream&
operator << (std::ostream& os, const pr_formatted_float<T>& pff)
{
  octave::preserve_stream_state stream_state (os);

  float_format real_fmt = pff.m_ff;

  if (real_fmt.fw >= 0)
    os << std::setw (real_fmt.fw);

  if (real_fmt.prec >= 0)
    os << std::setprecision (real_fmt.prec);

  os.flags (static_cast<std::ios::fmtflags>
            (real_fmt.fmt | real_fmt.up | real_fmt.sp));

  os << pff.m_val;

  return os;
}


So, there are a few questions, the most primary being what the output should do in terms of "precision".  It might be good to create a set of tests with perhaps fifty examples of all the various formats...in string format, then just output the result to a string and compare.  If the attached change is small enough and fixes an obvious bug, then apply as it is, but in the long run this may need a more extensive review.

(file #43963)

Dan Sebald <sebald>
Wed 18 Apr 2018 12:46:11 AM UTC, original submission:  

It caught my attention that the output format is displaying too much precision for numbers that have magnitude greater than the number of digits of precision, stated imprecisely.  That is, notice the progression:


octave:1> for i=-2:8
>   10^i / 3
> endfor
ans =  0.0033333
ans =  0.033333
ans =  0.33333
ans =  3.3333
ans =  33.333
ans =  333.33
ans =  3333.3
ans =  33333.33333
ans =  333333.33333
ans =  3333333.33333
ans =  33333333.33333


that once the left digits hits 5 the right digits jumps back up to five.

Here are some related bug reports, but they aren't quite the same issue:

https://savannah.gnu.org/bugs/?func=detailitem&item_id=53456
https://savannah.gnu.org/bugs/?func=detailitem&item_id=41359

Dan Sebald <sebald>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by cbm
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by sebald (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-04-18 cbm Carbon-Copy- Added cbm
    2018-04-18 mtmiller Item GroupIncorrect Result Regression
    2018-04-18 sebald Attached File- Added octave-precision_right_side_digits-djs2018apr17.patch, #43963

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code