bugGNU Octave - Bugs: bug #61358, dlmread fails to read NA values on...

 
 

bug #61358: dlmread fails to read NA values on arm64

Submitter:  pengyueqing <pyq>
Submitted:  Wed 20 Oct 2021 11:55:15 AM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Need Info Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 12 Nov 2021 06:54:09 AM UTC, comment #17: 

Reading/writing binary NA seems to work:


octave:1> fid = fopen ("foo.dat", "wb");
octave:2> fwrite (fid, [1, -1] * NA, "double");
octave:3> fclose (fid);
octave:4> fid = fopen ("foo.dat", "rb")
fid = 9
octave:5> x = fread (fid, "double");
octave:6> x
x =

    NA
    NA

octave:7>


The sign got lost but that seems to be a known feature
https://savannah.gnu.org/bugs/?40668

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 28 Oct 2021 02:26:55 AM UTC, comment #16: 

I also executed the script, and the result was similar to that of Dmitri.

 pengyueqing <pyq>
Thu 28 Oct 2021 01:14:12 AM UTC, comment #15: 

octave:1> format long
octave:2> cosint(pi)
ans = 7.366791204642578e-02


Dmitri A. Sergatskov <dasergatskov>
Thu 28 Oct 2021 01:08:40 AM UTC, comment #14: 

BTW, cosint(pi) is still failing. The tolerance should be 4e-15.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 27 Oct 2021 11:37:29 PM UTC, comment #13: 

Okay, so it's definitely in the read routine rather than the assignment.  I'll have to dig into the code a bit, but probably not today.

 Rik <rik5>
Group administrator
Wed 27 Oct 2021 11:30:15 PM UTC, comment #12: 


octave:1> x = dlmread ('NA.empty.csv', 'emptyvalue', NA)
x =

     1    NA     3
     4   NaN     6

octave:2> system("uname -m")
aarch64
ans = 0
octave:3> __octave_config_info__ ("hg_id")
ans = 774d6edb4cae
octave:4> system ("cat NA.empty.csv")
1,,3
4,NaN,6
ans = 0


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 27 Oct 2021 11:13:37 PM UTC, comment #11: 

I'm attaching a simple file with an empty-value which hopefully will help localize where the assignment is going wrong.

Can you download NA.empty.csv and then in Octave try


x = dlmread ('NA.empty.csv', 'emptyvalue', NA)



 Rik <rik5>
Group administrator
Wed 27 Oct 2021 10:12:40 PM UTC, comment #10: 

Looks the same here:


octave:1> format hex
NaN
NA
ans = 7ff8000000000000
ans = 7ff840f440000000
octave:4> version
ans = 7.0.0
octave:5> __octave_config_info__ ("hg_id")
ans = 774d6edb4cae
octave:6> system("uname -m")
aarch64


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 27 Oct 2021 10:01:00 PM UTC, comment #9: 

Can someone on ARM64 try this code in Octave?


format hex
NaN
NA


Results for an Intel x68 Linux system are shown below


octave:1> format hex
octave:2> NaN
ans = 7ff8000000000000
octave:3> NA
ans = 7ff840f440000000



 Rik <rik5>
Group administrator
Wed 27 Oct 2021 03:44:25 PM UTC, comment #8: 

The "NA" is in the file, it is just is not being  read.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 27 Oct 2021 03:41:53 PM UTC, comment #7: 

I am not the OP, but I have Fedora 35 on RaspberryPi
(cortexa72p)


 ./run-octave
GNU Octave, version 7.0.0
Copyright (C) 2020 The Octave Project Developers.
This is free software; see the source code for copying conditions.
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  For details, type 'warranty'.

Octave was configured for "aarch64-unknown-linux-gnu".

Additional information about Octave is available at https://www.octave.org.

Please contribute if you find this software useful.
For more information, visit https://www.octave.org/get-involved.html

Read https://www.octave.org/bugs.html to learn how to submit bug reports.
For information about changes from previous versions, type 'news'.

octave:1> fid = fopen ("NA.csv", "wt");
fdisp (fid, "1,NA,3");
fdisp (fid, "4,NaN,6");
fclose (fid);

data = dlmread ("NA.csv", ',')
data =

     1     0     3
     4   NaN     6

octave:6> __octave_config_info__ ("hg_id")
ans = 3da14aa5dbac


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 27 Oct 2021 02:34:49 PM UTC, comment #6: 

Octave gets very little testing on ARM platforms (32-bit or 64-bit).  There is also an issue that ARM processors can switch their endianness on the fly.

1) Are you building Octave from scratch on your machine?
2) What is the 'hg id' of the code you are building?
3) What endianness are you using (big or little)?

Can you run this test code (also attached as tst_NA.m) and report what it prints?


fid = fopen ("NA.csv", "wt");
fdisp (fid, "1,NA,3");
fdisp (fid, "4,NaN,6");
fclose (fid);

data = dlmread ("NA.csv", ',')




(file #52161)

 Rik <rik5>
Group administrator
Fri 22 Oct 2021 10:05:10 AM UTC, comment #5: 

I pushed a change to the default branch that increases the tolerance of the test in camlookat.m here:
http://hg.savannah.gnu.org/hgweb/octave/rev/789d0e220350

It looks like the tolerance for that BIST in cosint.m has already been increased before and it should be passing once a version with the updated tolerance has been released.

Re-titling for the remaining issue.

 Markus Mützel <mmuetzel>
Group administrator
Thu 21 Oct 2021 11:46:30 AM UTC, comment #4: 

the un-modified tarball can compile correctly on arm.
`a = NA` also worked in arm.

octave:1> a = NA
a =  NA
octave:2> a + 1
ans =  NA

pengyueqing <pyq>
Thu 21 Oct 2021 07:47:45 AM UTC, comment #3: 

Aaah. That is on ARM64 hardware. Interesting!

We don't have any CI on arm64 afaict. So I'm happy Octave seems to be compiling fine and seems to be working mostly without issues.
Did you need to apply any local patches or did the un-modified tarball compile correctly?

The tests related to trigonometric functions seem to be exceeding the tolerances only very slightly. It is probably ok to just increase the tolerance in the affected tests slightly to have them pass on that platform.


There might be an issue with our definition of NA on that platform. (I misread the code flow and thought we were entirely relying on the standard libraries. But that is not the case for floating point values.)
Afaict, that definition is done in liboctave/util/lo-ieee.cc.
Tbh, I'm not very familiar with that. CC'ing Rik in the hopes he might have a better insight in that part of the code.

Does working with NA work at all with Octave on that platform. E.g., does a simple assignment like `a = NA` work?


Markus Mützel <mmuetzel>
Group administrator
Thu 21 Oct 2021 03:05:42 AM UTC, comment #2: 

[root@simon28li spack-src]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

[root@simon28li spack-src]# uname -a
Linux simon28li.localdomain 4.18.0-193.el8.aarch64 #1 SMP Fri May 8 11:05:12 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

[root@simon28li octave-6.2.0-e55tupk2hrxvusr6xrvs7vvamxu5tsp6]# ldd bin/octave
        linux-vdso.so.1 (0x0000ffff96850000)
        libiconv.so.2 => /home/spack/opt/spack/linux-centos8-aarch64/gcc-8.3.1/libiconv-1.16-dx2c372kkymixiwrw6jtba3gruhddu2d/lib/libiconv.so.2 (0x0000ffff96720000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000ffff96550000)
        libm.so.6 => /lib64/libm.so.6 (0x0000ffff96480000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x0000ffff96420000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000ffff963e0000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000ffff963a0000)
        libc.so.6 => /lib64/libc.so.6 (0x0000ffff96220000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffff96860000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000ffff961f0000)

[root@simon28li spack-src]# rpm -qa | grep c++
libstdc++-8.3.1-5.el8.0.2.aarch64
libstdc++-devel-8.3.1-5.el8.0.2.aarch64
gmp-c++-6.1.2-10.el8.aarch64
gcc-c++-8.3.1-5.el8.0.2.aarch64
ncurses-c++-libs-6.1-7.20180224.el8.aarch64

pengyueqing <pyq>
Wed 20 Oct 2021 12:46:33 PM UTC, comment #1: 

The errors in dlmread.cc-tst look like an issue in the implementation of the C++ standard libraries to me.
What Octave is essentially doing is using `std::istringstream::operator>>(double)` to get the double values from the stream created with the file content. It looks like the implementation of the C++ standard libraries you are using cannot deal correctly with "NA" values.
Can you test outside Octave if that function is working correctly for you?

The errors in camlookat.m and cosint.m are probably caused by slightly higher inaccuracies in some math functions in your standard libraries. (Probably some trigonometric or inverse trigonometric functions.)
We already saw in the past that some of these functions are implemented with lower precision on Windows and macOS.
Which OS is this and which libraries are you using?

Markus Mützel <mmuetzel>
Group administrator
Wed 20 Oct 2021 11:55:15 AM UTC, original submission:  


>>>>> processing /home/spack-stage/root/spack-stage-octave-6.2.0-e55tupk2hrxvusr6xrvs7vvamxu5tsp6/spack-src/libinterp/corefcn/dlmread.cc-tst

*** test
 file = tempname ();
 unwind_protect
   fid = fopen (file, "wt");
   fwrite (fid, "1,NA,3");
   fclose (fid);

   assert (dlmread (file), [1, NA, 3]);
 unwind_protect_cleanup
   unlink (file);
 end_unwind_protect
!!!!! test failed
ASSERT errors for:  assert (dlmread (file),[1, NA, 3])

  Location  |  Observed  |  Expected  |  Reason
    (2)           0            NA        'NA' mismatch
*** test <*54029>
 file = tempname ();
 unwind_protect
   fid = fopen (file, "wt");
   fwrite (fid, "NaNe,bNa,Name,c\n1,NaN,3,Inftest\n-Inf,6,NA,8");
   fclose (fid);

   assert (dlmread (file), [0, 0, 0, 0; 1, NaN, 3, 0; -Inf, 6, NA, 8]);
 unwind_protect_cleanup
   unlink (file);
 end_unwind_protect
!!!!! regression: https://octave.org/testfailure/?54029
ASSERT errors for:  assert (dlmread (file),[0, 0, 0, 0; 1, NaN, 3, 0; -Inf, 6, NA, 8])

  Location  |  Observed  |  Expected  |  Reason
   (3,3)          0            NA        'NA' mismatch


>>>>> processing /home/spack-stage/root/spack-stage-octave-6.2.0-e55tupk2hrxvusr6xrvs7vvamxu5tsp6/spack-src/scripts/plot/appearance/camlookat.m

*** test
 hf = figure ("visible", "off");
 unwind_protect
   [x y z] = sphere ();
   h1 = surf (x + 1, y, z);
   hold on
   h2 = surf (x - 1, y + 2, z + 1);
   dir = camtarget () - campos ();
   dir /= norm (dir);
   camlookat (h1);
   dir2 = camtarget () - campos ();
   dir2 /= norm (dir2);
   assert (dir, dir2, -2*eps);
   camlookat (h2);
   dir2 = camtarget () - campos ();
   dir2 /= norm (dir2);
   assert (dir, dir2, -2*eps)
   camlookat ([h1 h2]);
   dir2 = camtarget () - campos ();
   dir2 /= norm (dir2);
   assert (dir, dir2, -2*eps);
 unwind_protect_cleanup
   close (hf);
 end_unwind_protect
!!!!! test failed
ASSERT errors for:  assert (dir,dir2,-2 * eps)

  Location  |  Observed  |  Expected  |  Reason
    (2)        0.72803      0.72803      Rel err 4.5749e-16 exceeds tol 4.4409e-16 by 1e-17


>>>>> processing /home/spack-stage/root/spack-stage-octave-6.2.0-e55tupk2hrxvusr6xrvs7vvamxu5tsp6/spack-src/scripts/specfun/cosint.m

*** assert (cosint (pi), 0.0736679120464254860, -2e-15)
!!!!! test failed
ASSERT errors for:  assert (cosint (pi),0.0736679120464254860,-2e-15)

  Location  |  Observed  |  Expected  |  Reason
     ()        0.073668     0.073668     Rel err 3.956e-15 exceeds tol 2e-15 by 2e-15


pengyueqing <pyq>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #52164:  NA.empty.csv added by rik5 (13B - text/csv)
file #52161:  tst_NA.m added by rik5 (123B - text/x-matlab)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by pyq (Submitted the item)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-10-27 rik5 Attached File- Added NA.empty.csv, #52164
    2021-10-27 rik5 Attached File- Added tst_NA.m, #52161
        StatusNone Need Info
    2021-10-22 mmuetzel CategoryTest Suite Interpreter
        StatusNeed Info None
        Release6.2.0 dev
        Summarymake check failed in dlmread.cc-tst camlookat.m cosint.m on arm64 dlmread fails to read NA values on arm64
    2021-10-21 mmuetzel Item GroupNone Incorrect Result
        Summarymake check failed in dlmread.cc-tst camlookat.m cosint.m make check failed in dlmread.cc-tst camlookat.m cosint.m on arm64
        Carbon-Copy- Added rik5
    2021-10-20 mmuetzel StatusNone Need Info

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.13-8a8d.
    Corresponding source code