bugGNU Octave - Bugs: bug #49166, "system.tst" fails under...

 
 

bug #49166: "system.tst" fails under Windows

Submitter:  Hartmut <hardy>
Submitted:  Fri 23 Sep 2016 11:47:59 AM UTC
   
 
Category:  Test Suite Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Inaccurate Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 4.2.0-rc2 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 28 Sep 2016 03:59:01 PM UTC, comment #6: 

It's my opinion as well that we don't want to do open heart surgery on Octave to find all these instances.  It's a shame there isn't some compiler option to just guarantee conformance to the IEEE specification for doubles.

The failures reported in this bug report are all


1*eps (t)


To be safe, I added a margin of 2 eps (t) in this cset http://hg.savannah.gnu.org/hgweb/octave/rev/29dab5b8332a.


Rik <rik5>
Group administrator
Wed 28 Sep 2016 03:00:12 PM UTC, comment #5: 

That's a funny test.

I don't know what we should do about extended precision.  We probably don't want to declare every floating point variable with volatile.  Or track down every place where it might matter.  Is there any place where these results could vary by more than eps or 2*eps (relative to the value of the variables)?

John W. Eaton <jwe>
Group administrator
Wed 28 Sep 2016 05:17:31 AM UTC, comment #4: 

Well, it looks like I know what is happening; But, I don't like the results.  I re-compiled a version of Octave via MXE where I truncated the intermediate results in Fcputime in data.cc.

I changed


  double usr = cpu_tm.user ();
  double sys = cpu_tm.system ();


to


  OCTAVE_FLOAT_TRUNCATE double usr = cpu_tm.user ();
  OCTAVE_FLOAT_TRUNCATE double sys = cpu_tm.system ();


The script which used to fail on the first or second iteration now passes 200 iterations without problem.

I'm adding jwe to the CC list for the bug.  This is a fairly large architectural question.  Do we want to find all places in the code where there is a potential for extra precision to be maintained and use OCTAVE_FLOAT_TRUNCATE?  Or do we want to just add a tolerance to the BIST tests whenever we encounter these issues?  I'm tempted to mark this as Postponed since I don't think the answer will be clear before the 4.2.0 release is made.

Rik <rik5>
Group administrator
Tue 27 Sep 2016 11:06:02 PM UTC, comment #3: 

This is repeatable on a Windows XP virtual machine that I keep around for testing.  There are some weird correlations.  If I remove the 'cd ...' line from the sample script in comment #2--which means Octave is running in the same directory as the script itself--then the failure does not happen so quickly.  If I cd to a directory that is not quite so far away from the test script then the script will fail, but at perhaps the 140th iteration.  If I keep the original 'cd ...' line then the script fails on the first or second iteration.

This may be due to extended precision, or lack thereof, when running on Windows platforms.  If you look at the C++ code for Fcputime in data.cc it is quite simple.


  if (args.length () != 0)
    print_usage ();

  octave::sys::cpu_time cpu_tm;

  double usr = cpu_tm.user ();
  double sys = cpu_tm.system ();

  return ovl (usr + sys, usr, sys);


From this, you can see that the m-file code


[t1, u1, s1] = cputime ();
assert (t1, u1 + s1);


really should pass.

In the first case, it is C++ double variables and the C++ '+' operator.

In the second case, it is Octave double variables and the Octave '+' operator.

After a failure, using format long, I can see that the variables look correct.


error: ASSERT errors for:  assert (t1,u1 + s1)

  Location  |  Observed  |  Expected  |  Reason
     ()         1.6409       1.6409      Abs err 2.2204e-016 exceeds tol 0>> format long
>> t1
t1 =  1.64092200000000
>> u1
u1 =  1.32046100000000
>> s1
s1 =  0.320461000000000
>> u1 + s1
ans =  1.64092200000000


And if I do some arbitrary arithmetic, I can get the assert statement to pass.


>> t = t1 + eps
t =  1.64092200000000
>> u = u1 + eps
u =  1.32046100000000
>> s = s1 + eps
s =  0.320461000000000
>> assert (t, u + s)


I'm going to try declaring the double variables in data.cc Fcputime using the OCTAVE_FLOAT_TRUNCATE option which should truncate intermediate results and see if that makes a difference.  That test will take several hours as I have to rebuild a Windows installer with MXE.


Rik <rik5>
Group administrator
Sat 24 Sep 2016 08:37:56 PM UTC, comment #2: 

Here is an easier way to maybe reproduce this behavior.

The following script basically does "test system.tst" in a loop. It NEVER makes it to the end on my Win-7 machine with Octave-4.2.0-rc2. But this very same script ALWAYS finishes just fine on the same PC, if I use the Octave-4.0.3 release instead.


more off
cd "C:/Octave/Octave-4.2.0-rc2/share/octave/4.2.0-rc2/etc/tests/fixed"
for t = 1:200
  disp("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX");
  disp(num2str(t));
  disp("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX");

  #result = test("system.tst", "verbose");
  #if (result != true)
  #  disp("BUG OCCURED!");
  #  break
  #endif
  #disp("Test PASSED");

  ### use code from test directly, to speed things up:
  [t1, u1, s1] = cputime ();
  for i = 1:200
    sin (i);
  endfor
  [t2, u2, s2] = cputime ();
  assert (t1, u1 + s1);
  assert (t2 == u2 + s2);
  assert (t2 >= t1);
  assert (u2 >= u2);
  assert (s2 >= s2);
endfor


I have observed the following discrepancies on my PC so far (observed time, absolute error):

  • 13.1105 , 1.7764e-15
  • 25.4753 , 3.5527e-15
  • (some others as well)


Hartmut <hardy>
Fri 23 Sep 2016 04:14:10 PM UTC, comment #1: 

To test just system.tst you need to copy the file in to a directory that is in your PATH variable.  Use the File Explorer in Windows to locate system.tst which should be at


C:\Octave\Octave-4.2.0-rc2\share\octave\4.2.0-rc2\etc\tests\fixed\system.tst


unless you chose a different installation directory than the default.

Copy that file somewhere convenient, like your present working directory in Octave.


pwd


Then you can run


test system.tst



Rik <rik5>
Group administrator
Fri 23 Sep 2016 11:47:59 AM UTC, original submission:  

When I do "__run_test_suite__" within Octave, installed via the official Windows installer from the alpha.gnu.org website (version 4.2.0-rc2) on a Win7 machine, then I get 1 FAIL-ing test result from "system.tst".

Here is the corresponding text from fntests.log:


>>>>> processing system.tst
***** test
 [t1, u1, s1] = cputime ();
 for i = 1:200
   sin (i);
 endfor
 [t2, u2, s2] = cputime ();
 assert (t1, u1 + s1);
 assert (t2 == u2 + s2);
 assert (t2 >= t1);
 assert (u2 >= u2);
 assert (s2 >= s2);
!!!!! test failed
ASSERT errors for:  assert (t1,u1 + s1)

  Location  |  Observed  |  Expected  |  Reason
     ()        412.9534     412.9534     Abs err 5.6843e-014 exceeds tol 0
***** testif HAVE_GETPWUID
 x = getpwuid (getuid ());
 assert (x.dir, tilde_expand ("~"));
 assert (x.dir, tilde_expand (sprintf ("~%s", x.name)));
 assert ("foobar", tilde_expand ("foobar"));
----- skipped test

***** testif HAVE_GETPGRP
 assert (getpgrp () > 0);
----- skipped test

***** testif HAVE_GETPPID
 assert (getppid () > 0);
----- skipped test

***** testif HAVE_GETEUID
 assert (geteuid () >= 0)
----- skipped test

***** testif HAVE_GETUID
 assert (getuid () >= 0)
----- skipped test

***** testif HAVE_GETEGID
 assert (getegid () >= 0)
----- skipped test

***** testif HAVE_GETGID
 assert (getgid () >= 0)
----- skipped test

***** testif HAVE_GETPWENT
 s = getpwent ();
 endpwent ();
 assert (isstruct (s)
         && isfield (s, "name")
         && isfield (s, "passwd")
         && isfield (s, "uid")
         && isfield (s, "gid")
         && isfield (s, "gecos")
         && isfield (s, "dir")
         && isfield (s, "shell"));
----- skipped test

***** testif HAVE_GETPWUID
 x = getpwent ();
 y = getpwuid (x.uid);
 endpwent ();
 assert (strcmp (x.name, y.name) && x.uid == y.uid && x.gid == y.gid);
----- skipped test

***** testif HAVE_GETPWNAM
 x = getpwent ();
 y = getpwnam (x.name);
 endpwent ();
 assert (strcmp (x.name, y.name) && x.uid == y.uid && x.gid == y.gid);
----- skipped test

***** testif HAVE_SETPWENT
 x = getpwent ();
 setpwent ();
 y = getpwent ();
 endpwent ();
 assert (strcmp (x.name, y.name) && x.uid == y.uid && x.gid == y.gid);
----- skipped test

***** testif HAVE_GETGRENT
 x = getgrent ();
 endgrent ();
 assert (isstruct (x)
         && isfield (x, "name")
         && isfield (x, "passwd")
         && isfield (x, "gid")
         && isfield (x, "mem"));
----- skipped test

***** testif HAVE_GETGRGID
 x = getgrent ();
 y = getgrgid (x.gid);
 endgrent ();
 assert (strcmp (x.name, y.name) && x.gid == y.gid);
----- skipped test

***** testif HAVE_GETGRNAM
 x = getgrent ();
 y = getgrnam (x.name);
 endgrent ();
 assert (strcmp (x.name, y.name) && x.gid == y.gid);
----- skipped test

***** testif HAVE_SETGRENT
 x = getgrent ();
 setgrent ();
 y = getgrent ();
 endgrent ();
 assert (strcmp (x.name, y.name) && x.gid == y.gid);
----- skipped test


Additional notes:

  • When I try "test system.tst" this does not seem to call the test at all. How do I call this single test?
  • When I paste only the relevant test code


 [t1, u1, s1] = cputime ();
 for i = 1:200
   sin (i);
 endfor
 [t2, u2, s2] = cputime ();
 assert (t1, u1 + s1);
 assert (t2 == u2 + s2);
 assert (t2 >= t1);
 assert (u2 >= u2);
 assert (s2 >= s2);

into an m-file, then this runs totally fine. No errors in this case.

  • I have not tried to run the full test suite for a second time, because it takes so horrendously long.



Hartmut <hardy>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by hardy (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-09-28 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2016-09-28 rik5 Carbon-Copy- Added jwe
    2016-09-27 rik5 StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code