bugGNU Octave - Bugs: bug #52627, Failing BIST tests on Mac OS X

 
 

bug #52627: Failing BIST tests on Mac OS X

Submitted by:  Rik <rik5>
Submitted on:  Sat 09 Dec 2017 12:11:24 AM UTC  
 
Category: Test SuiteSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: FixedAssigned to: Rik <rik5>
Originator Name: Open/Closed: Closed
Release: devOperating System: Mac OS

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 06 Jan 2018 03:16:21 PM UTC, comment #23:

I changed some tolerances to as much as 5 eps and now the acos BIST tests pass. There are real problems with the hyperbolic inverse functions. They are provide by whatever math library Apple is using, but they are pretty innacurate when compared to gcc's libm or to Mathematica's variable precision. Since this is a known issue, and not really Octave's problem, I marked the failing tests as xtests only for the Mac platform (http://hg.savannah.gnu.org/hgweb/octave/rev/a6753fc10731).

Rik <rik5>
Project AdministratorIn charge of this item.
Wed 27 Dec 2017 11:02:51 PM UTC, comment #22:

I changed the errors related to bug #47413 (a bad libc++ implementation of iostream library) into xtests on the Mac platform. The buildbot confirms this (http://buildbot.octave.org:8010/builders/clang-osx/builds/793/steps/test/logs/stdio).

The remaining errors are due to differences in the inverse trigonometric functions. Any ideas on how to resolve those? Do we run the results in a variable precision calculator and determine what the true results should be and whose implementation is off? Do we just relax the tolerances?

Rik <rik5>
Project AdministratorIn charge of this item.
Fri 22 Dec 2017 12:05:12 AM UTC, comment #21:

I fixed the errors with sorting of complex numbers on Mac platforms here (http://hg.savannah.gnu.org/hgweb/octave/rev/138c2380521c). It is confirmed by the buildbot results: http://buildbot.octave.org:8010/builders/clang-osx/builds/770/steps/test/logs/stdio which now have 7/7 passing for complex.tst.

Rik <rik5>
Project AdministratorIn charge of this item.
Wed 20 Dec 2017 08:08:04 PM UTC, comment #20:

@Ben: No, I'm leaving their libraries alone. I've interacted with some Mac zealots before and my life is too short to want to do that again. They believe they are right and won't be convinced otherwise.

My plan was to add a separate code template just for floats and just for the Mac platform using #if defined. In that template I would calculate the value of PI according to their library. That value would be calculated just once and stored in a static variable. Thereafter I would use that new value for the PI constant in all comparisons.

Rik <rik5>
Project AdministratorIn charge of this item.
Wed 20 Dec 2017 07:31:37 PM UTC, comment #19:

@Rik, silly question, but .... you're thinking about patching atan2 on MacOS? rather than redefining PI to match the atan2 result?

Ben Abbott <bpabbott>
Project Member
Wed 20 Dec 2017 05:12:08 PM UTC, comment #18:

Thank you Ben! And just as expected, everything is mostly good except for std::arg of -1 + 0i.

Mac: 3.1415925025939941
Linux: 3.1415927410125732

Clearly they are both approximately PI to 7 digits, but the math lib constant for PI does not match the atan2 result for PI which leads to problems.

I think I can code around that by using a static variable on Macs to hold the value of whatever they think PI is.

Rik <rik5>
Project AdministratorIn charge of this item.
Wed 20 Dec 2017 03:21:51 PM UTC, comment #17:

Thanks Dimitri, I should have been able to figure that our ;-)

MacOS results

Ben Abbott <bpabbott>
Project Member
Wed 20 Dec 2017 03:16:39 PM UTC, comment #16:

You probably need to compile it with "clang++"
Try

and

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 20 Dec 2017 03:05:58 PM UTC, comment #15:

@Rik, I get errors on MacOS (10.13.2)

Ben Abbott <bpabbott>
Project Member
Fri 15 Dec 2017 01:07:30 AM UTC, comment #14:

Does anyone have access to a mac to test the C++ programe in comment #13? Should I ask on the maintainer's list?

Rik <rik5>
Project AdministratorIn charge of this item.
Mon 11 Dec 2017 07:19:45 PM UTC, comment #13:

@Mike: I was working from your list. I knocked off the issue in fileattrib.m and the issue in camzoom.m. I'm not sure I want to tackle the issues in the math libraries.

For the sorting issue, I'm pretty sure that the problem is a cast to float on the constant M_PI. Could someone with access to a Mac try this C++ program and report the results?

The file is attached as tst_arg.cc. One should be able to compile it with

The results on my Linux machine are

If it turns out to be the problem with casting then we could always override the generic templated code in liboctave/util/oct-cmplx.h with a specialization for floats on Mac platforms.

Rik <rik5>
Project AdministratorIn charge of this item.
Sun 10 Dec 2017 07:28:39 PM UTC, comment #12:

I see the same problems with trig and hyperbolic functions
on linux. E.g.:

(clang/libc++)

I do NOT see sort problems on linux.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 10 Dec 2017 06:49:14 PM UTC, comment #11:

I summarized all of the macOS test failures earlier: https://lists.gnu.org/archive/html/octave-maintainers/2017-11/msg00071.html

The complex sorting is a distinct issue from the LLVM numeric string parsing, I don't know what the cause is, may be a macOS system math library issue.

Mike Miller <mtmiller>
Project Administrator
Sat 09 Dec 2017 03:55:50 PM UTC, comment #10:

@Dmitri: that probably indicates that std::arg for floats isn't returning the same values as everybody else. I'm not surprised because their atan2 function for floats is different.

Rik <rik5>
Project AdministratorIn charge of this item.
Sat 09 Dec 2017 04:44:02 AM UTC, comment #9:

FWIW it does pass on Fedora/clang/libc++

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 09 Dec 2017 04:38:32 AM UTC, comment #8:

The sort code for this special case is in liboctave/util/oct-cmplx.h. I suspect what is going on is that the arg() function for floats is computed slightly differently with clang's libc++. And static_cast<float> (M_PI) probably doesn't match this value.

Example code:

Rik <rik5>
Project AdministratorIn charge of this item.
Sat 09 Dec 2017 04:19:04 AM UTC, comment #7:

@Mike: Since you're quite familiar with the tests, do you know quickly why this one in complex.tst is failing?

It's obvious that the results aren't sorted according to the special sequence that Octave uses of magnitude followed by phase angle, but we ship our own sort routine as far as I know.

Rik <rik5>
Project AdministratorIn charge of this item.
Sat 09 Dec 2017 03:50:46 AM UTC, comment #6:

I compiled dev on Fedora with clang and libc++.
I had to disable arpack and magick

Summary:

PASS 15744
FAIL 13
REGRESSION 5
XFAIL (expected failure) 5
XFAIL (reported bug) 34
SKIPPED (feature) 239

If there is an interest we could add a fedora-clang-libcxx
buildbot slave to separate MacOS's problems from other issues.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 09 Dec 2017 12:56:18 AM UTC, comment #5:

It seems like those defaults are bound to persist at an operating system level--Linux systems will continue to link against GCC's libstdc++ and macOS wil link against clang's libc++.

I think that is enough of a positive correlation to place guard's around these tests. Even if the clang library is changed, the future of when that change is very uncertain. The last comment in the link you sent is from February, 2017. 10 months and no mention of forward progress are not confidence inspiring.

Rik <rik5>
Project AdministratorIn charge of this item.
Sat 09 Dec 2017 12:39:03 AM UTC, comment #4:

Sorry, it's actually a bug with the LLVM project's implementation of the C++ runtime, not the compiler itself. The clang compiler can be configured to build with either GCC's libstdc++ or with LLVM's own libc++ library.

On GNU/Linux systems the default is to use GCC's libstdc++, and the error does not occur. On macOS the default is to use clang's own libc++.

One can reproduce this on any operating system by building with "-stdlib=libc++" added to CXXFLAGS, assuming you have the clang libc++ library and headers installed.

Mike Miller <mtmiller>
Project Administrator
Sat 09 Dec 2017 12:35:19 AM UTC, comment #3:

For the record -- on Fedora buildbot clang (at the moment) is:
lang -v
clang version 4.0.1 (tags/RELEASE_401/final)

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 09 Dec 2017 12:30:40 AM UTC, comment #2:

But isn't it the case that it is only the clang version that is shipped with Mac OS X that actually has the problem. Looking at the buildbots, clang4 and clang5 on debian, and clang on Fedora, all pass this test.

If this is reliably correlated with operating system, can we use "! ismac ()" as a guard around the failing tests?

Rik <rik5>
Project AdministratorIn charge of this item.
Sat 09 Dec 2017 12:19:45 AM UTC, comment #1:

See bug #47413.

This is a clang compiler bug, not a macOS operating system bug.

The clang compiler does not handle numeric parsing correctly.

See the open clang bug https://bugs.llvm.org/show_bug.cgi?id=17782, in which the devs seem to be open to fixing this behavior, pending some clarification in the C++ specifications.

Mike Miller <mtmiller>
Project Administrator
Sat 09 Dec 2017 12:11:24 AM UTC, original submission:

There is an automatic buildbot which compiles the latest Octave development version and then runs the test suite. See http://buildbot.octave.org:8010/waterfall.

I've resolved two of the errors by inspecting the logfiles. But for the others I need some help because I don't have a Mac computer.

I'm hoping someone who can build on Mac OS X can be a partner for me on this bug report and run tests as I need them so we can reduce the number of failing tests from 15, to potentially 0.

The first function I want to try and fix is str2double which I think is causing several other failures.

Could someone run the following code on a Mac with a newly compiled version of the Octave development code?

Upload the diary file "diary1.txt" to this bug report and I'll investigate.

Rik <rik5>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by bpabbott (Posted a comment)
  • -unavailable- added by dasergatskov (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by rik5 (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 06 Jan 2018 03:16:21 PM UTCrik5StatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1