bugPSPP - Bugs: bug #51235, Wrong rounding of decimals in...


bug #51235: Wrong rounding of decimals in FACTOR output?

Submitted by:  Jaap Verhage <jaapv>
Submitted on:  Tue 13 Jun 2017 07:57:26 PM UTC  
Category:  Numerical Errors Severity:  5 - Average
Status:  Invalid Assigned to:  None
Open/Closed:  Closed Release:  None
Effort:  0.00

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Fri 16 Jun 2017 04:28:25 PM UTC, comment #6: 


So both you and Jaap agree that the current behaviour is correct.

We can close this item then.

John Darrington <jmd>
Project Administrator
Fri 16 Jun 2017 03:28:28 PM UTC, comment #5: 

> I think the problem is in data-out.c - the function should_round_up is a bit too naive:

I think that's a misunderstanding.  First, my notes and the comments in data-out.c say that SPSS follows the unusual policy that ties are always rounded away from zero, so that the basic idea that '5' rounds up is correct.

But more importantly, should_round_up() is supposed to be looking at a formatted number that has as many decimals as necessary to make a decision.  The formatting is done by rounder_init(), which has logic that is supposed to supply extra digits when an ambiguous case comes up.

Ben Pfaff <blp>
Project Administrator
Fri 16 Jun 2017 07:37:44 AM UTC, comment #4: 

Hello John and Ben,

if the number is, in fact, 0.9349999999999994981791929, that would mean that displaying it as .93 would be correct, wouldn't it? Sorry if I raised a false alarm.

Regards, Jaap.

Jaap Verhage <jaapv>
Thu 15 Jun 2017 11:33:46 AM UTC, comment #3: 

I did a bit of investigation into this, based on some data Jaap sent me offline.

The number Jaap is dealing with is in fact 0.9349999999999994981791929 (at least that's how GSL calculates it and printf ("%0.22f") displays it.)

I think the problem is in data-out.c - the function should_round_up is a bit too naive:

/* Returns true if the magnitude represented by R should be
   rounded up when chopped off at DECIMALS decimal places, false
   if it should be rounded down. */
static bool
should_round_up (const struct rounder *r, int decimals)
  int digit = r->string[r->integer_digits + decimals + 1];
  assert (digit >= '0' && digit <= '9');
  return digit >= '5';

It simply looks as the next number after the desired decimal places - which in this case is '4' so concludes that rounding down is appropriate.

I suggest we change the implementation of should_round_up to iterate from the right hand side and carry the previous digit.  Something like:

static bool
should_round_up (const struct rounder *r, int decimals)
  int x;
  int digit = 0;
  int carry_digit = '0';
  for (x = strlen (r->string) - 1;
       x >= r->integer_digits + decimals + 1;
      digit = r->string[x];
      assert (digit >= '0' && digit <= '9');
      if (carry_digit >= '5')
      carry_digit = digit;
  return digit >= '5';

The comment for this function would need to be adjusted too, as would numerous test results.

John Darrington <jmd>
Project Administrator
Wed 14 Jun 2017 09:06:42 AM UTC, comment #2: 

Hello Ben,

thanks for replying so soon. I just tried out your suggestion; it's  .935000000000000 .

Regards, Jaap.

Jaap Verhage <jaapv>
Wed 14 Jun 2017 03:00:33 AM UTC, comment #1: 

Maybe.  Internally, calculations are done to about 15 digits.  If you display 15 decimals, what value is displayed?

Ben Pfaff <blp>
Project Administrator
Tue 13 Jun 2017 07:57:26 PM UTC, original submission:  

In the "Communalities table" in the output of a FACTOR command, PSPP reports a value of .93. When instructed to display three decimals via the SET FORMAT command and repeating the FACTOR command, this displays as .935. Shoudn't that be .94 instead of .93 with two decimals? (It's .935000 when instructed to display six decimals.)

Regards, Jaap.

Jaap Verhage <jaapv>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by jmd (Posted a comment)
  • -email is unavailable- added by blp (Posted a comment)
  • -email is unavailable- added by jaapv (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-06-16 jmd StatusNone => Invalid
        Open/ClosedOpen => Closed

    Back to the top

    Powered by Savane 3.6