bugGNU Octave - Bugs: bug #53438, tests: rand.cc: known fixed state...

 
 

bug #53438: tests: rand.cc: known fixed state tests of randn and randg generators inaccurate on i686

Submitted by:  Mike Miller <mtmiller>
Submitted on:  Fri 23 Mar 2018 07:18:17 AM UTC  
 
Category:  Test Suite Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Inaccurate Result
Status:  Postponed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  GNU/Linux

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sat 24 Mar 2018 02:03:03 AM UTC, comment #8: 

For future reference, this can be reproduced on x86_64 easily by building Octave with "-mfpmath=387" added to CFLAGS and CXXFLAGS as mentioned in comment #2.

The generators that are affected are randn and randg. The values produced by randg are derived from randn values, and both involve floating point multiplications. With a given fixed state, a user should get the same exact uniform random number sequence on any system, and those tests do pass.

Mike Miller <mtmiller>
Project Administrator
Sat 24 Mar 2018 12:52:07 AM UTC, comment #7: 

I pushed a change on stable to allow these tests to pass for the upcoming release

https://hg.savannah.gnu.org/hgweb/octave/rev/15d2f32db174

Leaving the bug open for further investigation after the release dust settles.

Mike Miller <mtmiller>
Project Administrator
Fri 23 Mar 2018 06:56:15 PM UTC, comment #6: 

Thanks. I do get the same test failures as you on x86_64 with -mfpath=387. I now have full test logs from a full i686 VM, an i686 container on an x86_64 host, and x86_64 with -mfpmath=387, enough to work out the superset of tests that need looser tolerances.

Mike Miller <mtmiller>
Project Administrator
Fri 23 Mar 2018 05:28:04 PM UTC, comment #5: 

I guess the point I was trying to make is that those are all
the x87 vs sse issues.

For betaincinv:

***** test
x = linspace (0.1, 0.9, 11);
a = [0.1:0.1:1];
[x,a,b] = ndgrid (x,a,a);
xx = betaincinv (betainc (x, a, b, "upper"), a, b, "upper");
assert (xx, x, 3e-15);
!!!!! test failed
ASSERT errors for:  assert (xx,x,3e-15)

  Location  |  Observed  |  Expected  |  Reason
  (6,1,1)        0.5          0.5        Abs err 4.3299e-15 exceeds tol 3e-15 by 1e-15

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 23 Mar 2018 05:25:15 PM UTC, comment #4: 

I kept all of my test failures in separate reports since I know different developers are working on each of the function domains.

See bug #53435 for the "cam*" functions, and bug #53437 for the gammainc and betainc functions. I didn't get a test failure in betaincinv, could you add details about that to the appropriate bug report so we can loosen the tolerance?

Mike Miller <mtmiller>
Project Administrator
Fri 23 Mar 2018 05:18:05 PM UTC, comment #3: 

I missed

scripts/specfun/betaincinv.m ................................ PASS     25/27 
                                                                  FAIL    2

Dmitri A. Sergatskov <dasergatskov>
Fri 23 Mar 2018 05:10:36 PM UTC, comment #2: 

I compiled (8b50eab9689e (stable) tip) on x86_64 with
-mfpmath=387 and  got 15 FAILs in make check:

gcc_opt/libinterp/corefcn/rand.cc-tst ....................... PASS     60/65
                                                                  FAIL    5
...
scripts/plot/appearance/camorbit.m .......................... PASS     10/11
                                                                  FAIL    1
...
  scripts/plot/appearance/camroll.m ........................... PASS      5/6
                                                                  FAIL    1
...
scripts/specfun/gammainc.m .................................. PASS     79/84
                                                                  FAIL    4
                                               (expected failure) XFAIL   1
...
Summary:

  PASS                            14915
  FAIL                               13
  XFAIL (reported bug)               32
  XFAIL (expected failure)            6
  SKIP (missing feature)             39
  SKIP (run-time condition)          12

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 23 Mar 2018 03:33:49 PM UTC, comment #1: 

Interesting.  The short term expedient is to relax the tolerances.  However, I'm not exactly pleased with that course of action.

Octave code should be portable between platforms.  If I happen to write on a Mac or i586 machine, and fix a random seed because I want reproducible results, I should be able to transfer this to an i686 machine without issue.

My guess is that this has to do with preserving extra accuracy, 80 bits versus 64 bits, in the Intel math co-processor.

For the release, go ahead and apply your patch.  But let's leave the bug open and try to resolve it.

Rik <rik5>
Project Administrator
Fri 23 Mar 2018 07:18:17 AM UTC, original submission:  

In testing Octave built for i686, I am seeing 5 test failures in rand.cc. The 5 tests that are failing are reproducible because they are labeled "Test a known fixed state". I believe these test failures were introduced with changeset 8bb0251fcfde (https://hg.savannah.gnu.org/hgweb/octave/rev/8bb0251fcfde), which tightened the tolerances on many of the tests.

The attached patch fixes all test failures for me on i686 by loosening the tolerances again, not as loose as they used to be.

Cc Rik who last editing these tests, does it make sense to you to bump the tolerances up a bit to allow for what I'm seeing on i686?

I've cut out the important lines from the test log below:

ASSERT errors for:  assert (randn (1, 6),[-2.666521678978671, -0.7381719971724564, 1.507903992673601, 0.6019427189162239, -0.450661261143348, -0.7054431351574116],eps)
    (2)        -0.73817     -0.73817     Abs err 8.8818e-16 exceeds tol 2.2204e-16 by 7e-16
    (3)         1.5079       1.5079      Abs err 4.4409e-16 exceeds tol 2.2204e-16 by 2e-16
    (4)        0.60194      0.60194      Abs err 2.9976e-15 exceeds tol 2.2204e-16 by 3e-15
    (6)        -0.70544     -0.70544     Abs err 1.3323e-15 exceeds tol 2.2204e-16 by 1e-15
ASSERT errors for:  assert (randg (0.95, 1, 6),[3.099382433255327, 0.3974529788871218, 0.644367450750855, 1.143261091802246, 1.964111762696822, 0.04011915547957939],2 * eps)
    (1)         3.0994       3.0994      Abs err 8.8818e-16 exceeds tol 4.4409e-16 by 4e-16
    (3)        0.64437      0.64437      Abs err 2.5535e-15 exceeds tol 4.4409e-16 by 2e-15
    (4)         1.1433       1.1433      Abs err 1.1102e-15 exceeds tol 4.4409e-16 by 7e-16
ASSERT errors for:  assert (randg (1, 1, 6),[0.2273389379645993, 1.288822625058359, 0.2406335209340746, 1.218869553370733, 1.024649860162554, 0.09631230343599533],4 * eps)
    (2)         1.2888       1.2888      Abs err 3.9968e-15 exceeds tol 8.8818e-16 by 3e-15
    (4)         1.2189       1.2189      Abs err 8.8818e-15 exceeds tol 8.8818e-16 by 8e-15
ASSERT errors for:  assert (randg (10, 1, 6),[3.520369644331133, 15.15369864472106, 8.332112081991205, 8.406211067432674, 11.81193475187611, 10.88792728177059],16 * eps)
    (2)        15.1537      15.1537      Abs err 7.1054e-15 exceeds tol 3.5527e-15 by 4e-15
    (4)        8.40621      8.40621      Abs err 1.2434e-14 exceeds tol 3.5527e-15 by 9e-15
    (5)        11.8119      11.8119      Abs err 5.3291e-15 exceeds tol 3.5527e-15 by 2e-15
ASSERT errors for:  assert (randg (100, 1, 6),[75.34570255262264, 115.4911985594699, 95.23493031356388, 95.48926019250911, 106.2397448229803, 103.4813150404118],132 * eps)
    (4)        95.48926     95.48926     Abs err 5.6843e-14 exceeds tol 2.931e-14 by 3e-14
    (6)        103.4813     103.4813     Abs err 4.2633e-14 exceeds tol 2.931e-14 by 1e-14

Mike Miller <mtmiller>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #43638:  rand-tests.diff added by mtmiller (2KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mtmiller
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-02-26 mtmiller Carbon-CopyRemoved 80942 => -
    2018-03-24 mtmiller StatusPatch Submitted => Postponed
        Summarytests: rand.cc: 5 failing tests on i686 due to tolerance changes => tests: rand.cc: known fixed state tests of randn and randg generators inaccurate on i686
    2018-03-23 mtmiller Attached File- => Added rand-tests.diff, #43638
    2018-03-23 mtmiller Carbon-Copy- => Added rik5

    Back to the top


    Powered by Savane 3.5