bugGNU Octave - Bugs: bug #52627, Failing BIST tests on Mac OS X

 
 

bug #52627: Failing BIST tests on Mac OS X

Submitted by:  Rik <rik5>
Submitted on:  Sat 09 Dec 2017 12:11:24 AM UTC  
 
Category: Test SuiteSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: FixedAssigned to: Rik <rik5>
Originator Name: Open/Closed: Closed
Release: devOperating System: Mac OS

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Fri 02 Feb 2018 05:07:19 AM UTC, comment #34:

I checked in a change here which lookt at the returned Java version and splits it into new or old style before processing. I also marked the bug with Integer.compareTo with report number 51804. See http://hg.savannah.gnu.org/hgweb/octave/rev/5aa9c885ea18.

Rik <rik5>
Project AdministratorIn charge of this item.
Thu 01 Feb 2018 11:04:48 PM UTC, comment #33:

Sorry, I missed report #51804. However, could you mark this test as a known bug?

Bye
Sebastian

Sebastian <sschoeps>
Thu 01 Feb 2018 11:00:48 PM UTC, comment #32:

I agree with Mike. The compareTo is something deep in the Octave<->Java interface and beyond my understanding.

I don't mind making the change on the version. I think that was there mostly to make sure the interface is working at all.

Rik <rik5>
Project AdministratorIn charge of this item.
Thu 01 Feb 2018 10:35:15 PM UTC, comment #31:

The compareTo bug is already reported, see bug #51804. Needs someone with a deeper understanding of the Java API and maybe a rewrite for how Octave inspects and searches for matching method overloads.

Mike Miller <mtmiller>
Project Administrator
Thu 01 Feb 2018 10:30:52 PM UTC, comment #30:

Thanks Rik! Your fix for TMP works... the last issues that I see are in ov-java.cc-tst:

This is due to Java 9 calling itself "9" instead of "1.9". We could do

Furthermore

fails but

would work. Not sure why this was no issue before.

Sebastian <sschoeps>
Thu 01 Feb 2018 01:26:20 AM UTC, comment #29:

I checked in a change to save, clear, and then restore the environment variables TMPDIR and TMP when running certain BIST tests (http://hg.savannah.gnu.org/hgweb/octave/rev/276eca875721).

Rik <rik5>
Project AdministratorIn charge of this item.
Wed 31 Jan 2018 06:42:24 PM UTC, comment #28:

I guess the test was written under the assumption that no one would set TMP on any GNU or Unix system because it doesn't normally do anything. But Octave does use TMP as an override for the P_tmpdir compile-time constant if it is set regardless of OS.

Mike Miller <mtmiller>
Project Administrator
Wed 31 Jan 2018 05:31:24 PM UTC, comment #27:

Arrgh. Even Linux pays attention to both TMP and TMPDIR. I think that particular test needs to be re-coded. Re-opening this report and marking as "In Progress".

Rik <rik5>
Project AdministratorIn charge of this item.
Wed 31 Jan 2018 05:02:11 PM UTC, comment #26:

@Rik: It's quite complicated to understand what is going on because this problem occurs mainly when using homebrew as a package manager. Homebrew calls configure and make from its own environment (using ruby). The code

if (ispc ())
envname = "TMP";
else
envname = "TMPDIR";
endif
envdir = getenv (envname);
unsetenv (envname);
## Strip trailing file separators from P_tmpdir
def_tmpdir = P_tmpdir;
while (length (def_tmpdir) > 2 && strfind (filesep ("all"), def_tmpdir(end)))
def_tmpdir(end) = [];
endwhile
## Test 0-argument form
fname = tempname ();
[tmpdir, tmpfname] = fileparts (fname);
assert (tmpdir, def_tmpdir);

fails with

error: ASSERT errors for: assert (tmpdir,def_tmpdir)

Location | Observed | Expected | Reason
[] /tmp /var/tmp Strings don't match

and experiments in that environment show

octave:1> fname = tempname ()
fname = /var/tmp/oct-ynlTG9
octave:2> setenv("TMPDIR","/tmp")
octave:3> fname = tempname ()
fname = /var/tmp/oct-LmECe5
octave:4> setenv("TMP","/tmp")
octave:5> fname = tempname ()
fname = /tmp/oct-zFVL0A

and in my user environment

octave:1> fname = tempname ()
fname = /var/folders/yr/0kxqs6ns5_ncvr7v_sg568ww0000gp/T/oct-vpF8Ub
octave:2> setenv("TMPDIR","/tmp")
octave:3> fname = tempname ()
fname = /var/tmp/oct-e4Moqt
octave:4> setenv("TMP","/tmp")
octave:5> fname = tempname ()
fname = /tmp/oct-yIW2Zw

However, in either case "TMP" seems to have preference over "TMPDIR".

Bye
Sebastian

Sebastian <sschoeps>
Wed 31 Jan 2018 12:27:30 AM UTC, comment #25:

@Sebastian: Could you post the failing test to this bug report so I can take a look? According to multiple Google hits, TMPDIR is supposed to be the environment variable that sets the temporary directory, not TMP. It may be that Mac OSX has a hierarchy of values with TMPDIR first, TMP second, etc.

Rik <rik5>
Project AdministratorIn charge of this item.
Tue 30 Jan 2018 10:13:25 PM UTC, comment #24:

I observe a test failure in "file-io.cc-tst" on macOS if the environment variable "TMP" is set. It's easily fixed by replacing "if (ispc ())" by "if (ispc () || ismac())" which makes sense since the tmp-environment-variable on macOS is indeed "TMP". However, if "tempname()" should be independent of the environment variables then this might also be a bug.

Sebastian <sschoeps>
Sat 06 Jan 2018 03:16:21 PM UTC, comment #23:

I changed some tolerances to as much as 5 eps and now the acos BIST tests pass. There are real problems with the hyperbolic inverse functions. They are provide by whatever math library Apple is using, but they are pretty innacurate when compared to gcc's libm or to Mathematica's variable precision. Since this is a known issue, and not really Octave's problem, I marked the failing tests as xtests only for the Mac platform (http://hg.savannah.gnu.org/hgweb/octave/rev/a6753fc10731).

Rik <rik5>
Project AdministratorIn charge of this item.
Wed 27 Dec 2017 11:02:51 PM UTC, comment #22:

I changed the errors related to bug #47413 (a bad libc++ implementation of iostream library) into xtests on the Mac platform. The buildbot confirms this (http://buildbot.octave.org:8010/builders/clang-osx/builds/793/steps/test/logs/stdio).

The remaining errors are due to differences in the inverse trigonometric functions. Any ideas on how to resolve those? Do we run the results in a variable precision calculator and determine what the true results should be and whose implementation is off? Do we just relax the tolerances?

Rik <rik5>
Project AdministratorIn charge of this item.
Fri 22 Dec 2017 12:05:12 AM UTC, comment #21:

I fixed the errors with sorting of complex numbers on Mac platforms here (http://hg.savannah.gnu.org/hgweb/octave/rev/138c2380521c). It is confirmed by the buildbot results: http://buildbot.octave.org:8010/builders/clang-osx/builds/770/steps/test/logs/stdio which now have 7/7 passing for complex.tst.

Rik <rik5>
Project AdministratorIn charge of this item.
Wed 20 Dec 2017 08:08:04 PM UTC, comment #20:

@Ben: No, I'm leaving their libraries alone. I've interacted with some Mac zealots before and my life is too short to want to do that again. They believe they are right and won't be convinced otherwise.

My plan was to add a separate code template just for floats and just for the Mac platform using #if defined. In that template I would calculate the value of PI according to their library. That value would be calculated just once and stored in a static variable. Thereafter I would use that new value for the PI constant in all comparisons.

Rik <rik5>
Project AdministratorIn charge of this item.
Wed 20 Dec 2017 07:31:37 PM UTC, comment #19:

@Rik, silly question, but .... you're thinking about patching atan2 on MacOS? rather than redefining PI to match the atan2 result?

Ben Abbott <bpabbott>
Project Member
Wed 20 Dec 2017 05:12:08 PM UTC, comment #18:

Thank you Ben! And just as expected, everything is mostly good except for std::arg of -1 + 0i.

Mac: 3.1415925025939941
Linux: 3.1415927410125732

Clearly they are both approximately PI to 7 digits, but the math lib constant for PI does not match the atan2 result for PI which leads to problems.

I think I can code around that by using a static variable on Macs to hold the value of whatever they think PI is.

Rik <rik5>
Project AdministratorIn charge of this item.
Wed 20 Dec 2017 03:21:51 PM UTC, comment #17:

Thanks Dimitri, I should have been able to figure that our ;-)

MacOS results

Ben Abbott <bpabbott>
Project Member
Wed 20 Dec 2017 03:16:39 PM UTC, comment #16:

You probably need to compile it with "clang++"
Try

and

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 20 Dec 2017 03:05:58 PM UTC, comment #15:

@Rik, I get errors on MacOS (10.13.2)

Ben Abbott <bpabbott>
Project Member
Fri 15 Dec 2017 01:07:30 AM UTC, comment #14:

Does anyone have access to a mac to test the C++ programe in comment #13? Should I ask on the maintainer's list?

Rik <rik5>
Project AdministratorIn charge of this item.
Mon 11 Dec 2017 07:19:45 PM UTC, comment #13:

@Mike: I was working from your list. I knocked off the issue in fileattrib.m and the issue in camzoom.m. I'm not sure I want to tackle the issues in the math libraries.

For the sorting issue, I'm pretty sure that the problem is a cast to float on the constant M_PI. Could someone with access to a Mac try this C++ program and report the results?

The file is attached as tst_arg.cc. One should be able to compile it with

The results on my Linux machine are

If it turns out to be the problem with casting then we could always override the generic templated code in liboctave/util/oct-cmplx.h with a specialization for floats on Mac platforms.

Rik <rik5>
Project AdministratorIn charge of this item.
Sun 10 Dec 2017 07:28:39 PM UTC, comment #12:

I see the same problems with trig and hyperbolic functions
on linux. E.g.:

(clang/libc++)

I do NOT see sort problems on linux.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 10 Dec 2017 06:49:14 PM UTC, comment #11:

I summarized all of the macOS test failures earlier: https://lists.gnu.org/archive/html/octave-maintainers/2017-11/msg00071.html

The complex sorting is a distinct issue from the LLVM numeric string parsing, I don't know what the cause is, may be a macOS system math library issue.

Mike Miller <mtmiller>
Project Administrator
Sat 09 Dec 2017 03:55:50 PM UTC, comment #10:

@Dmitri: that probably indicates that std::arg for floats isn't returning the same values as everybody else. I'm not surprised because their atan2 function for floats is different.

Rik <rik5>
Project AdministratorIn charge of this item.
Sat 09 Dec 2017 04:44:02 AM UTC, comment #9:

FWIW it does pass on Fedora/clang/libc++

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 09 Dec 2017 04:38:32 AM UTC, comment #8:

The sort code for this special case is in liboctave/util/oct-cmplx.h. I suspect what is going on is that the arg() function for floats is computed slightly differently with clang's libc++. And static_cast<float> (M_PI) probably doesn't match this value.

Example code:

Rik <rik5>
Project AdministratorIn charge of this item.
Sat 09 Dec 2017 04:19:04 AM UTC, comment #7:

@Mike: Since you're quite familiar with the tests, do you know quickly why this one in complex.tst is failing?

It's obvious that the results aren't sorted according to the special sequence that Octave uses of magnitude followed by phase angle, but we ship our own sort routine as far as I know.

Rik <rik5>
Project AdministratorIn charge of this item.
Sat 09 Dec 2017 03:50:46 AM UTC, comment #6:

I compiled dev on Fedora with clang and libc++.
I had to disable arpack and magick

Summary:

PASS 15744
FAIL 13
REGRESSION 5
XFAIL (expected failure) 5
XFAIL (reported bug) 34
SKIPPED (feature) 239

If there is an interest we could add a fedora-clang-libcxx
buildbot slave to separate MacOS's problems from other issues.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 09 Dec 2017 12:56:18 AM UTC, comment #5:

It seems like those defaults are bound to persist at an operating system level--Linux systems will continue to link against GCC's libstdc++ and macOS wil link against clang's libc++.

I think that is enough of a positive correlation to place guard's around these tests. Even if the clang library is changed, the future of when that change is very uncertain. The last comment in the link you sent is from February, 2017. 10 months and no mention of forward progress are not confidence inspiring.

Rik <rik5>
Project AdministratorIn charge of this item.
Sat 09 Dec 2017 12:39:03 AM UTC, comment #4:

Sorry, it's actually a bug with the LLVM project's implementation of the C++ runtime, not the compiler itself. The clang compiler can be configured to build with either GCC's libstdc++ or with LLVM's own libc++ library.

On GNU/Linux systems the default is to use GCC's libstdc++, and the error does not occur. On macOS the default is to use clang's own libc++.

One can reproduce this on any operating system by building with "-stdlib=libc++" added to CXXFLAGS, assuming you have the clang libc++ library and headers installed.

Mike Miller <mtmiller>
Project Administrator
Sat 09 Dec 2017 12:35:19 AM UTC, comment #3:

For the record -- on Fedora buildbot clang (at the moment) is:
lang -v
clang version 4.0.1 (tags/RELEASE_401/final)

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 09 Dec 2017 12:30:40 AM UTC, comment #2:

But isn't it the case that it is only the clang version that is shipped with Mac OS X that actually has the problem. Looking at the buildbots, clang4 and clang5 on debian, and clang on Fedora, all pass this test.

If this is reliably correlated with operating system, can we use "! ismac ()" as a guard around the failing tests?

Rik <rik5>
Project AdministratorIn charge of this item.
Sat 09 Dec 2017 12:19:45 AM UTC, comment #1:

See bug #47413.

This is a clang compiler bug, not a macOS operating system bug.

The clang compiler does not handle numeric parsing correctly.

See the open clang bug https://bugs.llvm.org/show_bug.cgi?id=17782, in which the devs seem to be open to fixing this behavior, pending some clarification in the C++ specifications.

Mike Miller <mtmiller>
Project Administrator
Sat 09 Dec 2017 12:11:24 AM UTC, original submission:

There is an automatic buildbot which compiles the latest Octave development version and then runs the test suite. See http://buildbot.octave.org:8010/waterfall.

I've resolved two of the errors by inspecting the logfiles. But for the others I need some help because I don't have a Mac computer.

I'm hoping someone who can build on Mac OS X can be a partner for me on this bug report and run tests as I need them so we can reduce the number of failing tests from 15, to potentially 0.

The first function I want to try and fix is str2double which I think is causing several other failures.

Could someone run the following code on a Mac with a newly compiled version of the Octave development code?

Upload the diary file "diary1.txt" to this bug report and I'll investigate.

Rik <rik5>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by sschoeps (Posted a comment)
  • -unavailable- added by bpabbott (Posted a comment)
  • -unavailable- added by dasergatskov (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by rik5 (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 01 Feb 2018 01:26:20 AM UTCrik5StatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed
    Wed 31 Jan 2018 05:31:24 PM UTCrik5StatusFixed=>In Progress
      Open/ClosedClosed=>Open
    Sat 06 Jan 2018 03:16:21 PM UTCrik5StatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1