bugGNU Octave - Bugs: bug #59830, Architecture-dependent results for...

 
 

bug #59830: Architecture-dependent results for arithmetic operations involving NA

Submitter:  Rafael Laboissière <rlaboiss>
Submitted:  Tue 05 Jan 2021 10:10:12 PM UTC
   
 
Category:  Interpreter Severity:  2 - Minor
Priority:  1 - Later Item Group:  Incorrect Result
Status:  In Progress Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * 6.1.0 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 27 May 2021 10:25:47 PM UTC, comment #21: 

The base fix works.  I'm leaving the bug report open in case anyone wants to work on the small remaining issues.  But I've downgraded the priority to 1.

Rik <rik5>
Group administrator
Sun 14 Feb 2021 06:31:11 PM UTC, comment #20: 

I don't think it is worth the effort.  Cross-compiling is used when we have a hard time (as with Windows) getting the necessary toolchain and libraries in place.

For a MIPs processor it should be easy enough to have a Linux distribution and do a native build in which case the configure script will work correctly.

I took the easy path and added a default value of 'no' (don't use MIPS definition of NaN) for cross-compiling to the AC_RUN_IFELSE macro.  See http://hg.savannah.gnu.org/hgweb/octave/rev/8e549734531e.


Rik <rik5>
Group administrator
Sun 14 Feb 2021 08:33:16 AM UTC, comment #19: 

The new configure test breaks when cross-compiling for a host for which the build platform cannot run executables.

Maybe we could check the value of `$host_cpu` first and run a specific test only for MIPS hosts (with appropriate guesses when cross-compiling).
Or as a more minimal change, we could just add a 4th argument with guess values to `AC_RUN_IFELSE` in the new configure test.

Markus Mützel <mmuetzel>
Group administrator
Sat 13 Feb 2021 07:24:41 PM UTC, comment #18: 

I pushed a change to the stable branch that isn't perfect (http://hg.savannah.gnu.org/hgweb/octave/rev/e9b7ac7031fb), but does allow NA to work by default on MIPS architectures.

Ideally, the configure test OCTAVE_MIPS_NAN would check the actual value of the high word in an IEEE quiet NaN and see whether it is the non-standard value.  Right now, it just assumes that if your target is MIPS then you are using the non-standard option.  This is the default, but someone might build with the '-mnan=2008' option at some point.

The issue of binary compatibility of files saved with NA values could be addressed by changing the __lo_ieee_replace_old_NA() function in liboctave/utils/lo-ieee.cc.  Right now that function is called when loading double values from a binary file and updates any old values of NA to the new value of NA.  It could be extended to also search for any MIPS NA value and replace with the architecture-specific value of NA.

These are definitely edge cases, and I don't use MIPS so I don't have an incentive to work on this.  As a workaround, you could save data using Octave's -text format along with the '-z' flag to compress the resulting file.

Rik <rik5>
Group administrator
Wed 20 Jan 2021 08:08:35 AM UTC, comment #17: 

I tried it (using the Debian package for octave) and the configuration fails with the following error message:


configure:6770: checking whether the C compiler works
configure:6792: gcc -g -O2 -fdebug-prefix-map=/home/rafael/octave-6.1.1~hg.2020.12.27=. -fstack-protector-strong -Wformat -Werror=format-security -mnan=2008 -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-z,relro conftest.c  >&5
/usr/bin/ld: /tmp/cc8LGpyx.o: linking -mnan=2008 module with previous -mnan=legacy modules
/usr/bin/ld: failed to merge target specific data of file /tmp/cc8LGpyx.o
collect2: error: ld returned 1 exit status
configure:6796: $? = 1
configure:6834: result: no
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "GNU Octave"
| #define PACKAGE_TARNAME "octave"
| #define PACKAGE_VERSION "6.1.1~hg.2020.12.27-1"
| #define PACKAGE_STRING "GNU Octave 6.1.1~hg.2020.12.27-1"
| #define PACKAGE_BUGREPORT "https://octave.org/bugs.html"
| #define PACKAGE_URL "https://www.gnu.org/software/octave/"
| #define PACKAGE "octave"
| #define VERSION "6.1.1~hg.2020.12.27-1"
| #define OCTAVE_SOURCE 1
| /* end confdefs.h.  */
|
| int
| main ()
| {
|
|   ;
|   return 0;
| }
configure:6839: error: in `/home/rafael/octave-6.1.1~hg.2020.12.27':
configure:6841: error: C compiler cannot create executables


Rafael Laboissière <rlaboiss>
Tue 19 Jan 2021 05:36:25 PM UTC, comment #16: 

Hmm, it could get to be time consuming to support ARM if there are a lot of instances where this is going to diverge from all the other processor architectures.

What happens if you undo my patch, add '-mnan=2008' to your CXXFLAGS, CFLAGS, and FFLAGS variables, and re-compile Octave from scratch?  Do operations with NA work correctly?  Does 'make check' pass?

Rik <rik5>
Group administrator
Tue 19 Jan 2021 08:54:22 AM UTC, comment #15: 


comment #13:

> If so, this isn't that hard a fix.  There will need to be a configure test written to determine whether Octave is compiling on a MIPS architecture and then there will need to be an #ifdef in lo-ieee.h to choose between two different definitions of LO_IEEE_NA_HW.


We should also ensure the interoperability between the different architectures.  For instance, if we run the following code on a amd64 system:


octave:1> x = NA;
octave:2> save -binary x.dat x


I get now the following on mips64el with the patched Octave, when trying to load the file saved on amd64:


octave:1> load x.dat
octave:2> x
x = NaN


Rafael Laboissière <rlaboiss>
Tue 19 Jan 2021 07:31:06 AM UTC, comment #14: 


comment #13:

> It seems that re-defining Octave NA to be a quiet signaling NaN on MIPS architectures will resolve this issue.  To be absolutely certain, could you use this custom-built version of Octave and try running the BIST tests for the level-set package and see if they pass?


I did the required test. I compiled Octave on a mips64el system with your LO_IEEE_NA_HW patch and used that version to run the 18 BISTs in function so_step_armijo of the level-set package, all of which passed.

Rafael Laboissière <rlaboiss>
Tue 12 Jan 2021 04:58:24 PM UTC, comment #13: 

Apparently my last posting (comment #12) was truncated.

It seems that re-defining Octave NA to be a quiet signaling NaN on MIPS architectures will resolve this issue.  To be absolutely certain, could you use this custom-built version of Octave and try running the BIST tests for the level-set package and see if they pass?

If so, this isn't that hard a fix.  There will need to be a configure test written to determine whether Octave is compiling on a MIPS architecture and then there will need to be an #ifdef in lo-ieee.h to choose between two different definitions of LO_IEEE_NA_HW.

Rik <rik5>
Group administrator
Sun 10 Jan 2021 04:27:18 PM UTC, comment #12: 

The last test seems to have worked which means this might be a way forward.  According to your post, you tried for verification


y = x / NA
+verbatim+

I actually wanted to verify

+verbatim+
y = x / 4


Does that also result in isna (y) being true?

And if it does, can you use this custom version of Octave to run the BIST tests in the level-set package and see if they pass now?

Rik <rik5>
Group administrator
Sun 10 Jan 2021 09:19:49 AM UTC, comment #11: 


comment #8:

> Instead, you might try re-defining the high word of Octave NA so that the quiet signaling bit is '0' to match MIPS.  As another test, I would ask you to try changing liboctave/util/lo-ieee.h.
>


> diff -r e4c152e827aa liboctave/util/lo-ieee.h
> --- a/liboctave/util/lo-ieee.h        Thu Jan 07 18:34:18 2021 +0100
> +++ b/liboctave/util/lo-ieee.h        Fri Jan 08 14:09:31 2021 -0800
> @@ -69,7 +69,7 @@ typedef union
>
>  #define LO_IEEE_NA_HW_OLD 0x7ff00000
>  #define LO_IEEE_NA_LW_OLD 1954
> -#define LO_IEEE_NA_HW 0x7FF840F4
> +#define LO_IEEE_NA_HW 0x7FF040F4
>  #define LO_IEEE_NA_LW 0x40000000
>  #define LO_IEEE_NA_FLOAT   0x7FC207A2
>


>
> Apply the patch with 'patch -p1 < mypatch' and then re-compile.
>
> Try
>


> format native-bit
> x = NA
> y = x / 4
> isna (y)


I recompiled Octave on a Debian mips64el system with Rik's patch above.  It yields this:


octave:1> format native-bit
octave:2> x = NA
x = 0000000000000000000000000000001000101111000000100000111111111110
octave:3> y = x / NA
y = 0000000000000000000000000000001000101111000000100000111111111110
octave:4> isna (y)
ans = 10000000


Just for the record, this is what happens without the patch:


octave:1> format native-bit
octave:2> x = NA
x = 0000000000000000000000000000001000101111000000100001111111111110
octave:3> y = x / NA
y = 1111111111111111111111111111111111111111111111111110111111111110
octave:4> isna (y)
ans = 00000000


Rafael Laboissière <rlaboiss>
Sat 09 Jan 2021 06:11:41 PM UTC, comment #10: 

The specific value of NA was chosen to be exactly the same as the corresponding value in R so that saving and loading data in binary format would preserve the value when transferring data between Octave and R.  I had initially hoped we would maybe modify functions in Octave to actually respect NA values when appropriate, but that hasn't ever happened since NA was added to Octave nearly 20 years ago.  Maybe it would make more sense to just deprecate and drop NA from Octave?

John W. Eaton <jwe>
Group administrator
Sat 09 Jan 2021 03:50:39 PM UTC, comment #9: 

On mips64el:


octave:1> format native-bit
octave:2> x = NA
x = 0000000000000000000000000000001000101111000000100001111111111110
octave:3> x / 4
ans = 1111111111111111111111111111111111111111111111111110111111111110


Rafael Laboissière <rlaboiss>
Fri 08 Jan 2021 10:12:32 PM UTC, comment #8: 

Can you try this?


format native-bit
x = NA
x / 4


Also, one thing I notice is that the silent signaling bit for NaN in IEEE is turned on in Octave according to the IEEE 2008 format.  However, as I noted in comment #6 and is shown in your comment #7, MIPS systems seem to use a value of '0' to indicate quiet signaling by default.

Trying to reverse that convention would require using the '-mnan=2008' option not just for Octave, but for all the libraries it links to.  That sounds hard.

Instead, you might try re-defining the high word of Octave NA so that the quiet signaling bit is '0' to match MIPS.  As another test, I would ask you to try changing liboctave/util/lo-ieee.h.


diff -r e4c152e827aa liboctave/util/lo-ieee.h
--- a/liboctave/util/lo-ieee.h        Thu Jan 07 18:34:18 2021 +0100
+++ b/liboctave/util/lo-ieee.h        Fri Jan 08 14:09:31 2021 -0800
@@ -69,7 +69,7 @@ typedef union

 #define LO_IEEE_NA_HW_OLD 0x7ff00000
 #define LO_IEEE_NA_LW_OLD 1954
-#define LO_IEEE_NA_HW 0x7FF840F4
+#define LO_IEEE_NA_HW 0x7FF040F4
 #define LO_IEEE_NA_LW 0x40000000
 #define LO_IEEE_NA_FLOAT   0x7FC207A2


Apply the patch with 'patch -p1 < mypatch' and then re-compile.

Try


format native-bit
x = NA
y = x / 4
isna (y)


Rik <rik5>
Group administrator
Fri 08 Jan 2021 08:32:01 PM UTC, comment #7: 

Just in case this can be of any interest, here is what I get on a mips64el Debian system:


octave:1> format native-bit
octave:2> NA
ans = 0000000000000000000000000000001000101111000000100001111111111110
octave:3> NaN
ans = 1111111111111111111111111111111111111111111111111110111111111110


and this is what I get on a amd64 Debian system:


octave:1> format native-bit
octave:2> NA
ans = 0000000000000000000000000000001000101111000000100001111111111110
octave:3> NaN
ans = 0000000000000000000000000000000000000000000000000001111111111110


Rafael Laboissière <rlaboiss>
Fri 08 Jan 2021 07:05:00 PM UTC, comment #6: 

I suspect there isn't going to be a good solution.

See the file liboctave/utils/lo-ieee.h.

Octave defines an NA value to be composed of two 32-bit values.


#define LO_IEEE_NA_HW 0x7FF840F4
#define LO_IEEE_NA_LW 0x40000000


The routine that decides whether something is NA versus NaN is


int
__lo_ieee_is_NA (double x)
{
  lo_ieee_double t;
  t.value = x;
  return (__lo_ieee_isnan (x) && t.word[lo_ieee_hw] == LO_IEEE_NA_HW
          && t.word[lo_ieee_lw] == LO_IEEE_NA_LW) ? 1 : 0;
}


The IEEE standard defines NaN as having the sign bit cleared, maximum possible exponent, and a mantissa which is non-zero.  Thus, the high word begins with 0x7FF.  Most mathematical libraries detect NaN and then don't further process the number.  So when you operate on an NA value in an IEEE-compliant library the value just passes through.  However, this is implementation dependent.  My guess is that the ARM software library, or hardware implementation, detects that the value is a NaN and then substitutes a constant NaN value such as 0x7FF80000.  This value is also NaN according to IEEE standards, but it is no longer an Octave NA because the values have changed.  If you can compile with a debugger you could check whether this is correct.

There may be special switches you can throw during compilation to determine how IEEE operations are implemented (such as not using 80bit precision, etc.).  Maybe there is something that might be useful for MIPS?  In gcc, for example, I see '-mhard-float' to require use of HW math co-processor and '-msoft-float' to use software libraries for floating point.

This option also looks promising


-mnan=2008
       -mnan=legacy
           These options control the encoding of the special not-a-number (NaN) IEEE 754 floating-point
           data.

           The -mnan=legacy option selects the legacy encoding.  In this case quiet NaNs (qNaNs) are
           denoted by the first bit of their trailing significand field being 0, whereas signaling NaNs
           (sNaNs) are denoted by the first bit of their trailing significand field being 1.

           The -mnan=2008 option selects the IEEE 754-2008 encoding.  In this case qNaNs are denoted by
           the first bit of their trailing significand field being 1, whereas sNaNs are denoted by the
           first bit of their trailing significand field being 0.


If you can verify with a debugger first I think that would be the best first step.

Rik <rik5>
Group administrator
Thu 07 Jan 2021 07:52:13 PM UTC, comment #5: 

comment #4:

> Okay, at least defining an NA value is possible.
>
> Next step is to isolate the problem and see which mathematical operations preserve NA and which convert it to NaN.  Just guessing right now, it is more likely that a hardware accelerated multiplication/division is going to change things.
>
> Tests:
>
> x = NA;
> x + 1
> x - 1
> x * 1
> x / 1


Here we are:


octave:1> x = NA;
octave:2> x + 1
ans = NaN
octave:3> x - 1
ans = NaN
octave:4> x * 1
ans = NaN
octave:5> x / 1
ans = NaN


Rafael Laboissière <rlaboiss>
Thu 07 Jan 2021 02:43:06 PM UTC, comment #4: 

Okay, at least defining an NA value is possible.

Next step is to isolate the problem and see which mathematical operations preserve NA and which convert it to NaN.  Just guessing right now, it is more likely that a hardware accelerated multiplication/division is going to change things.

Tests:


x = NA;
x + 1
x - 1
x * 1
x / 1


Rik <rik5>
Group administrator
Thu 07 Jan 2021 11:50:13 AM UTC, comment #3: 

I get this on a mips46el system with Octave 6.1.0 (actually, with Debian package octave version 6.1.1~hg.2020.12.27-2):


octave:1> x = NA
x =  NA
octave:2> isna (x)
ans = 1
octave:3> y = ~x
error: invalid conversion from NaN to logical


Rafael Laboissière <rlaboiss>
Thu 07 Jan 2021 02:20:05 AM UTC, comment #2: 

Very strange.

On the mips64el system, is every NA translated to NaN?  For example, does this print NA and 1?


x = NA
isna (x)


What happens when you try to perform the logical operation ~x?


y = ~x


I get an error message on Linux.

Rik <rik5>
Group administrator
Wed 06 Jan 2021 08:28:26 AM UTC, comment #1: 

The BIST in the level-set package that exposes the problem is the one in so_step_armijo.m, as can be seen in the build log of the Debian package octave-level-set on mips64el:


>>>>> /<<PKGBUILDDIR>>/inst/so_step_armijo.m
***** function s = updateState (phi, data)
  inside = ls_inside (phi);
  vol = data.g.h * length (find (inside));
  cost = 1 / vol;

  % Throw if the volume is too large.  This is used
  % to check the error handling code.
  if (vol > 15)
    error ("volume too large");
  endif

  s = struct ("phi", phi, "cost", cost);
***** endfunction
***** shared n, phi, baseData, d, f, dJ
  n = 100;
  x = linspace (-10, 10, n);
  h = x(2) - x(1);
  a = 0;
  b = 5;
  phi = ls_genbasic (x, "box", a, b);

  baseData = struct ();
  baseData.p = so_init_params (false);
  baseData.g = struct ("x", x, "h", h);
  baseData.cb = struct ("update_state", @updateState);
  baseData.s = updateState (phi, baseData);
  baseData.s.phi = phi;

  f = ones (size (phi));
  d = ls_solve_stationary (phi, f, h);

  curVol = b - a;
  dJ = -2 / curVol^2;
!!!!! test failed
assert (all (isna (d0 (:)) | !isnan (d0 (:)))) failed
shared variables   scalar structure containing the fields:

    n = [](0x0)
    phi = [](0x0)
    baseData = [](0x0)
    d = [](0x0)
    f = [](0x0)
    dJ = [](0x0)


These are the lines in ls_solve_stationary.m that cause the bug in mips64el:


  d0 = nb;
  d0(~fz) ./= fabs(~fz);


Rafael Laboissière <rlaboiss>
Tue 05 Jan 2021 10:10:12 PM UTC, original submission:  

Octave is producing different results for arithmetic operations involving NA in different architectures of Debian GNU/Linux.  On a amd64 system, I get:


$ echo "NA / 1" | octave-cli
ans = NA


while on a mips64el system, I get:


$ echo "NA / 1" | octave-cli
ans = NaN


This is preventing a BIST in the level-set package to succeed on a mips64el system.

Does anyone know the cause for this disparity?



Rafael Laboissière <rlaboiss>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by rlaboiss (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-05-27 rik5 Severity3 - Normal 2 - Minor
        Priority5 - Normal 1 - Later
        StatusReady For Test In Progress
    2021-02-13 rik5 StatusNeed Info Ready For Test
    2021-01-07 rik5 StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code