bugGNU Octave - Bugs: bug #53456, output_precision: Cannot set...


bug #53456: output_precision: Cannot set precision higher than 16

Submitter:  Oliver Heimlich <oheim>
Submitted:  Sat 24 Mar 2018 11:24:22 PM UTC
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Regression
Status:  None Assigned to:  None
Originator Name:  Oliver Heimlich Open/Closed:  * Open
Release:  * dev Operating System:  * Any
* Mandatory Fields

Add a New Comment (Rich Markup)

Comment Type & Canned Response:


( Jump to the original submission )

Sun 01 Jul 2018 08:54:47 PM UTC, comment #9: 

Just wanted to say that version 3.2.0 of the interval package includes printf methods, which allows one to set the desired precision with string templates. I couldn't make the printf method compatible with other data types (for the reasons described in comment #3), so only one interval object is allowed as the second input argument.

With this alternative at hand, it is no longer a priority for me to increase the output precision with this particular function beyond 16. Like Mike said in comment #2, the other functions are far better for this use case.

Oliver Heimlich <oheim>
Tue 10 Apr 2018 02:37:27 AM UTC, comment #8: 

I think if someone is going to look at adding more options to output_precision as in comment #7, that's an Octave 5 development task, not something that will be done for 4.4.

So for Octave 4.4, the output_precision remains as it is, with an error if the user tries to set the value larger than 16. Do we want to do anything about that for Octave 4.4, like change the error into a warning, allow the user to set a higher limit but have no effect on how doubles and singles are displayed?

Mike Miller <mtmiller>
Project Administrator
Sun 01 Apr 2018 09:45:35 PM UTC, comment #7: 

I agree that a limit on output_precision is of little use if it is used for both single and double precision numbers: If it is forbidden (or discouraged) to set precision higher than 16 for double, then 7 digits should be handled equally for single precision. OTOH, if output_precision is used for built-in floating-point formats only and output precision will be capped anyway (starting with Octave 4.4), it is of no use to set a value higher than 16.

I assume that the only reason for capping the output precisions of both number formats has been for Matlab compatibility of the “format long” command.

Thus, I'd like to suggest the following improvement (which would resolve the regression): Internally, there should be two output precisions, one for single and one for double. The output_precision function should have the following syntaxes:

 · output_precision () or output_precision ("double") return double output precision
 · output_precision ("single") return single output precision
 · output_precision (p, "single") set single output precision
 · output_precision (p, "double") set double output precision
 · output_precision (p) set both single and double output precision (like in Octave 4.2 and earlier)

The number of digits displayed should never be capped (to 7 or 16 digits respectively). Instead, for Matlab compatibility the “format long” command should set each internal precision accordingly. “format short” (the default format) should set both precisions to 5.

Oliver Heimlich <oheim>
Wed 28 Mar 2018 09:09:30 AM UTC, comment #6: 

OK, I remember now that it was easier to just make the hard limit 16 digits instead of providing limits for both real and complex values.


I'm not sure what to do unless we introduce a separate limit for single precision values.

John W. Eaton <jwe>
Project Administrator
Wed 28 Mar 2018 02:14:23 AM UTC, comment #5: 

Yes a warning would be good. So if you remove the limit, Octave singles will still display with 8 digits and doubles with 16 digits even if the limit is set to 20 or 100?

I can see how having the option of a larger limit could still be useful to other user types and functions.

Mike Miller <mtmiller>
Project Administrator
Wed 28 Mar 2018 02:06:30 AM UTC, comment #4: 

I'm not opposed to removing the limit.  Maybe we should just have an (optional) warning if it is set to more than 16.  The warning could say that generally, double precision numbers have about 16 meaningful decimal digits (and 7 for single).  Would that be OK?

John W. Eaton <jwe>
Project Administrator
Tue 27 Mar 2018 10:48:19 PM UTC, comment #3: 

Regarding floating-point numbers, num2str and printf are a valid workaround. Although, I also found it comfortable to simply raise the number of digits with output_precision for number output. IMO this is a valid feature for numerical analysis tools. With the limitation to max. 16 digits, the function becomes pretty useless for accuracy explorations.

Another use case of mine is to control output for the interval package, where the output_precision function has been used to set the number of desired digits. With directed (outward) rounding of floating-point numbers it becomes interesting to set the precision to 17 to see inclusion properties of intervals. Without any alternative, I probably have to come up with a different solution now.

The num2str function is not suitable for the interval package, but I'd like to implement printf support for the interval type. Unfortunately, the built-in printf functions do not allow to be extended with custom types (m-file classes). I can't even make use of the printf_format_list class, because it is not part of the public API. So, it probably is not a good idea to override the printf methods.

Oliver Heimlich <oheim>
Mon 26 Mar 2018 07:16:21 PM UTC, comment #2: 

Workarounds include using the printf functions or num2str with a precision argument.

Mike Miller <mtmiller>
Project Administrator
Mon 26 Mar 2018 07:15:14 PM UTC, comment #1: 

If the limit were removed, you would still only get 16 digits for double-precision values and 8 digits for single-precision values.

The new behavior is that output_precision() is a cap on the default output width. But it will never increase the default output width from 8 or 16 for single and double values respectively. If you set output_precision to 50, you would still only get 16 digits. So that's the meaning of the new error message.

Mike Miller <mtmiller>
Project Administrator
Sat 24 Mar 2018 11:24:22 PM UTC, original submission:  

This is probably related to bug #53167.

>> output_precision (17)
error: output_precision: arg must be less than or equal to 16

The current documentation for this function suggests that you can set a higher value, but that is not possible anymore.

Also, I don't understand why the limit has been introduced. Double precision numbers can have up to 751 significant digits, see pow2 (-1074).

Oliver Heimlich <oheim>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by oheim (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only project members can vote.





    No changes have been made to this item

    Back to the top

    Powered by Savane 3.9