bugGNU Octave - Bugs: bug #57460, Tolerance exceeded in test for...

 
 

bug #57460: Tolerance exceeded in test for "svds" on Windows

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Sun 22 Dec 2019 10:10:10 AM UTC
   
 
Category:  Test Suite Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Inaccurate Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 24 Dec 2019 10:20:20 AM UTC, comment #7: 

The matrix A that is factorized in the tests isn't random. Probably some other input that in used in svds is.

Anyway, I went ahead and increased the tolerance for the tests in this changeset:
http://hg.savannah.gnu.org/hgweb/octave/rev/6cc2ddcdc2c4

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Mon 23 Dec 2019 04:56:16 PM UTC, comment #6: 

ARPACK and the BIST tests have been a huge saga.  The initialization value of 42 was set after a search on Linux, Windows, and Mac machines for a value that would work everywhere.  That was a difficult process which just isn't worth repeating.

I think we should just relax the tolerance of the test.  In fact, given a matrix of random values, it is not clear to me why we should expect the 7 largest of those to be within any particular zone of convergence.

Rik <rik5>
Group administrator
Mon 23 Dec 2019 04:28:23 PM UTC, comment #5: 

The results for your test:

>> stats_tst_svds
Fraction passing = 0 / 100


If I comment the lines in "tst_svds.m" setting the rng state, I see the following:

>> stats_tst_svds
Fraction passing = 97 / 100


So another "fix" would be to change the seed.

Markus Mützel <mmuetzel>
Group administrator
Mon 23 Dec 2019 04:10:40 PM UTC, comment #4: 

Interestingly, the failing test does not use the random number generator to initialize the starting search point for ARPACK.  To do that, Octave has to pass the opts structure.  The first test that is failing is just calling "svds (A, k)".

In this case, ARPACK chooses its own random starting point.  That means, it may be just random bad luck that sometimes it fails and sometimes it doesn't.

Could you run the following benchmark which I have attached in two files (tst_svds.m and stats_tst_svds.m)?  Download the files, put them in the same directory, start octave, and then run "stats_tst_svds"

(file #48110, file #48111)

Rik <rik5>
Group administrator
Mon 23 Dec 2019 07:51:56 AM UTC, comment #3: 

I checked with an older Octave (hg id 56d94f86a659), I still had lying around.
In that version, the test still passed with the given tolerance.
This was before the changes to the rng.

The difference is small. Changing the tolerance to "tol = 12 eps () norm (s2, 1);" is enough to make the test pass.
IIUC, a factor of n/k  (~ 14.3) would still be expected if all singular values are of approximately the same magnitude: "tol = n/k eps () norm (s2, 1);"

Please, correct me if that is wrong.

Markus Mützel <mmuetzel>
Group administrator
Mon 23 Dec 2019 05:45:43 AM UTC, comment #2: 

Afaict this was after your changes to the rng.

Markus Mützel <mmuetzel>
Group administrator
Mon 23 Dec 2019 05:41:51 AM UTC, comment #1: 

Was this before or after cset 797be8d10c22?

There were changes to the rand generator for doubles which I reverted in 797be8d10c22 that could influence this test.

Rik <rik5>
Group administrator
Sun 22 Dec 2019 10:10:10 AM UTC, original submission:  

Running the test suite on Windows with Octave hg id 1a75fca6ad5d and MXE Octave hg id 5a6318b5b71d, I see the following error for svds.m:

>>>>> processing D:\Repositories\Octave\test\octave-2019-12-22-10-10-w64_clean\octave-2019-12-22-10-10-w64\mingw64\share\octave\6.0.0\m\sparse\svds.m
***** testif HAVE_ARPACK
 [u2,s2,v2,flag] = svds (A,k);
 s2 = diag (s2);
 assert (flag, ! 1);
 tol = 10 * eps() * norm(s2, 1);
 assert (s2, s(end:-1:end-k+1), tol);
!!!!! test failed
ASSERT errors for:  assert (s2,s (end:-1:end - k + 1),tol)

  Location  |  Observed  |  Expected  |  Reason
    (4)        41.9848      41.9848      Abs err 7.6739e-13 exceeds tol 6.5244e-13 by 1e-13


The shared variables used for this test are:

n = 100;
k = 7;
A = sparse ([3:n,1:n,1:(n-2)],[1:(n-2),1:n,3:n],[ones(1,n-2),0.4*n*ones(1,n),ones(1,n-2)]);
[u,s,v] = svd (full (A));
s = diag (s);
[~, idx] = sort (abs (s));
s = s(idx);


Maybe it is good enough to just increase the tolerance.
But it would be better if someone more experienced in numerical mathematics could comment.

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #48110:  tst_svds.m added by rik5 (666B - text/x-matlab)
file #48111:  stats_tst_svds.m added by rik5 (144B - text/x-matlab)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-24 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2019-12-24 mmuetzel StatusNeed Info Ready For Test
    2019-12-23 rik5 Attached File- Added tst_svds.m, #48110
        Attached File- Added stats_tst_svds.m, #48111
    2019-12-23 rik5 StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code