bugGNU Octave - Bugs: bug #57435, Test suite crashes on Windows on...

 
 

bug #57435: Test suite crashes on Windows on sparse.tst

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Tue 17 Dec 2019 06:54:48 PM UTC
   
 
Category:  Test Suite Severity:  5 - Blocker
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 23 Dec 2019 04:18:55 PM UTC, comment #24: 

I think there is a question for jwe here.  How many of the functions in dldfcn really need to be there now? 

The functions which are not truly internal, i.e, the ones not starting with two underscores, still have to provide a documentation string.  So the code looks like


DOCSTRING
#if defined (HAVE_AMD)

 ... code ...

#else

  octave_unused_parameter (args);
  octave_unused_parameter (nargout);

  err_disabled_feature ("amd", "AMD");

#endif


Using amd as an example, Octave is always going to provide that function if it has the libraries available.  Therefore, there really isn't a need to make it dynamically loadable.

Rik <rik5>
Group administrator
Sun 22 Dec 2019 08:51:08 AM UTC, comment #23: 

I pushed the patch here:
http://hg.savannah.gnu.org/hgweb/octave/rev/1a75fca6ad5d

And opened bug #57459 for moving those functions to core.

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Sun 22 Dec 2019 12:06:36 AM UTC, comment #22: 

I cross-compiled with the attached patch (and SuiteSparse 4.5.6).
The test suite now completes without Octave closing unexpectedly.

Wrt Rik's comment #21: I believe this is what I did with the last 2 verbatim blocks in comment #18. Before "clear functions", the pointers in the SuiteSparse_config struct are valid and resolve to the functions as set in amd.cc. After "clear functions", the pointers still have the same values. But they point to invalid memory.

I'll try if I can get a newer version of SuiteSparse to cross-compile. But I think that the problem is caused by Octave setting function pointers in the global structure that can get out of scope.
The mlock is necessary. Or the functions shouldn't be (un-)loadable and moved to Octave core like jwe suggested.

(file #48099)

Markus Mützel <mmuetzel>
Group administrator
Sat 21 Dec 2019 10:52:21 PM UTC, comment #21: 

Also, to narrow it down, could you verify that the pointers in the Suitesparse_config struct are valid before the call to clear functions, and then invalid afterwards.  For testing, I would suggest a modification of your small example to


amd (ones (30, 30));
inv (sparse ([1,1;1,1+i]))  # check with debugger that pointers are valid
clear functions
inv (sparse ([1,1;1,1+i]))  # check with debugger that pointers are invalid



Rik <rik5>
Group administrator
Sat 21 Dec 2019 10:45:07 PM UTC, comment #20: 

Regarding sparse matrix representation, Octave uses a compressed column format (not simple row, column, value triplets).  There is some information about it in section 22.1.1 "Storage of Sparse Matrices" in the Octave manual.

In this case, the values you show for the column indices look correct to me for Octave's internal representation.

I had also noticed the fact that we are using SUITESPARSE_ASSIGN_FPTR to assign function handles to variables within SuiteSparse.  The code in symbfact.cc is


  cholmod_common Common;
  cholmod_common *cm = &Common;
  CHOLMOD_NAME(start) (cm);

  double spu = octave_sparse_params::get_key ("spumoni");
  if (spu == 0.0)
    {
      cm->print = -1;
      SUITESPARSE_ASSIGN_FPTR (printf_func, cm->print_function, nullptr);
    }
  else
    {
      cm->print = static_cast<int> (spu) + 2;
      SUITESPARSE_ASSIGN_FPTR (printf_func, cm->print_function, &SparseCholPrint);
    }

  cm->error_handler = &SparseCholError;
  SUITESPARSE_ASSIGN_FPTR2 (divcomplex_func, cm->complex_divide, divcomplex);
  SUITESPARSE_ASSIGN_FPTR2 (hypot_func, cm->hypotenuse, hypot);


When I looked at this I assumed that because cm was a local variable any changes to the options structure would be confined to this function and disappear when the variable Common went out of scope.

But, you quoted the expansion of the macros


#  if (SUITESPARSE_VERSION >= SUITESPARSE_VER_CODE (4, 3))
#    define SUITESPARSE_NAME(name) SuiteSparse_ ## name
#    define SUITESPARSE_ASSIGN_FPTR(f_name, f_var, f_assign) (SuiteSparse_config.f_name = f_assign)
#    define SUITESPARSE_ASSIGN_FPTR2(f_name, f_var, f_assign) (SuiteSparse_config.f_name = SUITESPARSE_NAME (f_assign))
#  else
#    define SUITESPARSE_ASSIGN_FPTR(f_name, f_var, f_assign) (f_var = f_assign)
#    define SUITESPARSE_ASSIGN_FPTR2(f_name, f_var, f_assign) (f_var = CHOLMOD_NAME (f_assign))
#  endif


which shows that the middle argument isn't even used when SuiteSparse >= 4.3.

This could still be an error in SuiteSparse.  I just checked the version I am using in Linux and it is 5.1.2.  Can you try moving all the way up to a 5.X series of SuiteSparse and see if the problem still persists?


Rik <rik5>
Group administrator
Sat 21 Dec 2019 05:31:30 PM UTC, comment #19: 

If locking amd.oct avoids the problem, then maybe we should just move that function to corefcn instead of defining it as a .oct file in dldfcn.  The suitesparse libraries are always linked with liboctave if they are available so there is no real point in dynamically loading functions that depend on that library.

If we always load suitesparse libraries if they are available, then we could also refactor the way that suitesparse config is handled as the comments in SuiteSparse_config.h recommend.  Instead of attempting to initialize those function pointers on demand, we could do it once at startup.

John W. Eaton <jwe>
Group administrator
Sat 21 Dec 2019 05:05:24 PM UTC, comment #18: 

Another way to trigger the bug is:

amd (ones (30, 30));
clear functions
inv (sparse ([1,1;1,1+i]))


Looking at the code for both functions (amd and symbfact), the thing they have in common is that they both use functions from suitesparse.
However, I can't trigger the function using "chol" in the first step.

Another common feature that is not used in "chol" is the SUITESPARSE_ASSIGN_FPTR macro.
The macro definition is different for the newer suitesparse version:

#  if (SUITESPARSE_VERSION >= SUITESPARSE_VER_CODE (4, 3))
#    define SUITESPARSE_NAME(name) SuiteSparse_ ## name
#    define SUITESPARSE_ASSIGN_FPTR(f_name, f_var, f_assign) (SuiteSparse_config.f_name = f_assign)
#    define SUITESPARSE_ASSIGN_FPTR2(f_name, f_var, f_assign) (SuiteSparse_config.f_name = SUITESPARSE_NAME (f_assign))
#  else
#    define SUITESPARSE_ASSIGN_FPTR(f_name, f_var, f_assign) (f_var = f_assign)
#    define SUITESPARSE_ASSIGN_FPTR2(f_name, f_var, f_assign) (f_var = CHOLMOD_NAME (f_assign))
#  endif


SuiteSparse_config is declared as "extern" [1].

Before the "clear functions" at a break point at sparse-lu.cc:325

(gdb) p SuiteSparse_config
$1 = {malloc_func = 0x62549778 <malloc>, calloc_func = 0x625497a8 <calloc>, realloc_func = 0x62549758 <realloc>, free_func = 0x62549798 <free>,
  printf_func = 0x62549b70 <printf(char const*, ...)>, hypot_func = 0x781470 <SuiteSparse_hypot>, divcomplex_func = 0x7813d0 <SuiteSparse_divcomplex>}


After the "clear functions", calling "inv" with a sparse matrix causes the segfault. At a nearby location:

(gdb) p SuiteSparse_config
$2 = {malloc_func = 0x62549778, calloc_func = 0x625497a8, realloc_func = 0x62549758, free_func = 0x62549798, printf_func = 0x62549b70,
  hypot_func = 0x781470 <SuiteSparse_hypot>, divcomplex_func = 0x7813d0 <SuiteSparse_divcomplex>}
(gdb) p SuiteSparse_config.malloc_func
$3 = (void *(*)(size_t)) 0x62549778
(gdb) p *SuiteSparse_config.malloc_func
Cannot access memory at address 0x62549778


One possible fix might be to mlock those functions that use SUITESPARSE_ASSIGN_FPTR or SUITESPARSE_ASSIGN_FPTR2. I'll try that next.

[1]: https://github.com/DrTimothyAldenDavis/SuiteSparse/blob/75753f94f45b3bcfaf59229832c33bbd1bccf744/SuiteSparse_config/SuiteSparse_config.h#L96

Markus Mützel <mmuetzel>
Group administrator
Sat 21 Dec 2019 01:30:57 PM UTC, comment #17: 

I'm not sure what the internal representation of sparse matrices is in Octave. Without much research, I thought they would be triplets of row index, column index and value. But looking at the constructors of SparseRep, it does not seem to work that way:

    SparseRep (octave_idx_type nr, octave_idx_type nc, octave_idx_type nz = 1)
      : d (nz > 0 ? new T [nz] : new T [1]),
        r (nz > 0 ? new octave_idx_type [nz] {} : new octave_idx_type [1] {}),
        c (new octave_idx_type [nc+1] {}),
        nzmx (nz > 0 ? nz : 1), nrows (nr), ncols (nc), count (1)
    { }


So please ignore my previous comment.

Markus Mützel <mmuetzel>
Group administrator
Sat 21 Dec 2019 01:07:22 PM UTC, comment #16: 

I set a break at liboctave/numeric/sparse-lu.cc:325 and inspected the data just before calling the UMFPACK function:

(gdb) p *(Ap)
$1 = 0
(gdb) p *(Ap+1)
$2 = 2
(gdb) p *(Ap+2)
$3 = 4
(gdb) p *(Ap+3)
$4 = -8646640802109656046
(gdb) p *(Ai)
$5 = 0
(gdb) p *(Ai+1)
$6 = 1
(gdb) p *(Ai+2)
$7 = 0
(gdb) p *(Ai+3)
$8 = 1
(gdb) p *(Ai+4)
$9 = 241020416
(gdb) p *(Az)
$10 = {_M_value = 1 + 0 * I}
(gdb) p *(Az+1)
$11 = {_M_value = 1 + 0 * I}
(gdb) p *(Az+2)
$12 = {_M_value = 1 + 0 * I}
(gdb) p *(Az+3)
$13 = {_M_value = 1 + 1 * I}


The values at Ap (the column indices) don't seem to be correct.

Fwiw, they are already incorrect in SparseComplexMatrix::inverse (CSparse.cc:1040):

(gdb) up 2
#2  0x0000000068fafd7f in SparseComplexMatrix::inverse (this=this@entry=0x5b8fb820, mattype=..., info=@0x5b8fb6f0: 1900551, rcond=@0x5b8fb6f8: 0,
    calc_cond=calc_cond@entry=true) at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/array/MArray.h:70
70      /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/array/MArray.h: No such file or directory.
(gdb) p this->cidx()
$15 = (octave_idx_type *) 0x592be8e0
(gdb) p *(this->cidx())
$16 = 0
(gdb) p *(this->cidx()+1)
$17 = 2
(gdb) p *(this->cidx()+2)
$18 = 4
(gdb) p *(this->cidx()+3)
$19 = -8646686981603003266


And they seem to be wrong from the get-go in Finv at libinterp/corefcn/inv.cc:176:

(gdb) display-sparse-array m
sparse object: {<MSparse<std::complex<double> >> = {<Sparse<std::complex<double> >> = {_vptr.Sparse = 0xeb11740 <vtable for SparseComplexMatrix+16>,
      rep = 0x5d8cc1e0, dimensions = {rep = 0x5f15e890}}, <No data fields>}, <No data fields>}
dimensions.rep[0] = ndims: 2
dims: {2, 2}

dimensions.rep[1] = 2
rep = {d = 0x5b608a50, r = 0x5f15e4c0, c = 0x592be8e0, nzmx = 4, nrows = 2, ncols = 2, count = {m_count = {<std::__atomic_base<long long>> = {
        static _S_alignment = 8, _M_i = 1}, <No data fields>}}}
rep.d = {{_M_value = 1 + 0 * I}, {_M_value = 1 + 0 * I}, {_M_value = 1 + 0 * I}, {_M_value = 1 + 1 * I}}
rep.r = {0, 1, 0, 1}
rep.c = {0, 2, 4}


So it is probably indeed an error on Octave's side.

Markus Mützel <mmuetzel>
Group administrator
Fri 20 Dec 2019 10:31:56 PM UTC, comment #15: 

The backtrace is slightly different for that sequence of commands:

(gdb) bt
#0  0x0000000044ea1ff7 in umfzl_scale ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\libumfpack.dll
#1  0x0000000044e97861 in umfzl_scale_column ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\libumfpack.dll
#2  0x0000000044e90b12 in umfzl_kernel ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\libumfpack.dll
#3  0x0000000044ea5beb in umfpack_zl_numeric ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\libumfpack.dll
#4  0x0000000069ab1f70 in octave::math::umfpack_numeric<std::complex<double> > (Info=0x597a5ad0, Control=0x446e0000, Numeric=0x51240000,
    Symbolic=0x5934f860, Az=0x5b9ab2b0, Ai=<optimized out>, Ap=0x5dca6010)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/numeric/sparse-lu.cc:325
#5  octave::math::sparse_lu<SparseComplexMatrix>::sparse_lu (this=0x0, a=..., Qinit=..., piv_thres=..., scale=32, FixedQ=false,
    droptol=6.9525070095186182e-310, milu=false, udiag=false)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/numeric/sparse-lu.cc:725
#6  0x0000000069c80530 in vtable for MArray<double> ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\liboctave-7.dll
#7  0x0000000068fafd7f in SparseComplexMatrix::inverse (this=0x5b9ab860, this@entry=0x5b9ab820, mattype=..., info=@0x5b9ab860: 1774714144,
    rcond=@0x5b9ab6f8: 0, calc_cond=calc_cond@entry=true)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/array/MArray.h:70
#8  0x00000000017afd49 in Finv (args=..., nargout=0)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/corefcn/inv.cc:176
#9  0x000000000139c9e5 in octave_builtin::call (this=0x59425d20, tw=..., nargout=0, args=...)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/octave-value/ov-builtin.cc:62
#10 0x00000000014de8c1 in octave::tree_index_expression::evaluate_n (this=0x5920ab20, tw=..., nargout=0)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/array/dim-vector.h:158
#11 0x0000000001b0fc5f in octave::tree_index_expression::evaluate (this=<optimized out>, tw=..., nargout=<optimized out>)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-idx.h:106
#12 0x00000000014cea83 in octave::tree_evaluator::visit_statement (this=0x59f6e840, stmt=...)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-eval.cc:2720
#13 0x00000000014bfeec in octave::tree_statement::accept (tw=..., this=0x53820bf0)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-stmt.h:121
#14 octave::tree_evaluator::visit_statement_list (this=0x59f6e840, lst=...)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-eval.cc:2779
#15 0x00000000014c045a in octave::tree_statement_list::accept (tw=..., this=<optimized out>)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-stmt.h:198
#16 octave::tree_evaluator::eval (this=0x59f6e840, stmt_list=..., interactive=<optimized out>)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-eval.cc:399
#17 0x00000000017b4243 in octave::interpreter::main_loop (this=this@entry=0x59f6dae0)


I copied the gdbinit file to the Octave directory (where I started the shell). Instead it had to be in the User directory. Now the macros are available. Thanks for the hint.

Markus Mützel <mmuetzel>
Group administrator
Fri 20 Dec 2019 10:20:13 PM UTC, comment #14: 

Do the gdb macros not work or were they simply not loaded?  The gdbinit file needs to be in the directory from which you start gdb.  Alternatively, you can use "source .gdbinit" from within the debugger to load any set off commands.

Rik <rik5>
Group administrator
Fri 20 Dec 2019 10:10:40 PM UTC, comment #13: 

I could limit the tests that are necessary to trigger the bug to the following:
- libinterp/dldfcn/symbfact.cc-tst
- fixed/bug-35881/bug-35881.tst
- fixed/sparse.tst

If I remove any of those, Octave doesn't crash.
Might be another "clear all" bug.

The minimum set of commands seems to be:

A = sparse (magic (3));
[count, h, parent, post, r] = symbfact (A);
clear all
inv (sparse ([1,1;1,1+i]))


The gdb macros don't seem to work on Windows.

Markus Mützel <mmuetzel>
Group administrator
Fri 20 Dec 2019 09:49:15 PM UTC, comment #12: 

I agree that it would be better if we had an idea about why this is happening.  Is it our code or theirs?

I made an m-file to help you test.  It is meant to be run from the directory that contains run-octave.


./run-octave -f -q bisect_tst_files.m


The file contains these lines which set the lower and upper bound of the files that are tested.


## Change these to do a binary search of tests
l = 1;
u = 132;  # 132 is max


132 is nearly 128 so it will take you about 8 runs to do a binary search to find out which test is the problem.  I ran it on Linux and found a bunch of warnings that don't show up with 'make check' that should be fixed.  Also, libinterp contains Java tests and the JVM has been known to cause memory issues.  Anyways, try it out.




(file #48098)

Rik <rik5>
Group administrator
Fri 20 Dec 2019 08:19:27 PM UTC, comment #11: 

Maybe the bug is in suitesparse?  Or something that we are doing wrong is exposed by the new version?  It might be nice to know what the cause of the problem is.

Looking at the _run_test_suite_ script, it operates recursively on directories, so there doesn't seem to be a simple way to execute a linear list of test files.  Maybe you could generate a cell array list of all the .tst files that are installed and then do something like


list_of_test_files = { ... };
for i = 1:numel (list_of_test_files)
  f = list_of_test_files{i};
  disp (f);
  test (f);
end
test ("/path/to/sparse.tst")


and then bisect on the list of files until you can find the one that needs to be executed before sparse.tst to cause the failure?


John W. Eaton <jwe>
Group administrator
Fri 20 Dec 2019 08:02:38 PM UTC, comment #10: 

Running the fixed tests alone is not enough to trigger the bug.
It's necessary to run the libinterp tests first.

Should we look deeper into this? Or just revert to the suitesparse version we used previously.

Markus Mützel <mmuetzel>
Group administrator
Fri 20 Dec 2019 07:03:47 PM UTC, comment #9: 

Cross-compilation with suitesparse 4.2.1 finished.
Octave does NOT close unexpectedly whilst running the test suite with that version.
It might be related to this(?) change in suitesparse:
https://github.com/DrTimothyAldenDavis/SuiteSparse/releases/tag/v4.3.0

Also when looking at the back trace (#3) with the newer suitesparse (v4.5.6): "droptol" should default to -1.0 not to a very small positive number. And scale is supposed to be false but it is 240. The address of this doesn't seem to be correct either.
Something seems to wreak havoc on the stack...

I'll try to remove the tests as suggested by jwe in comment #8.

Markus Mützel <mmuetzel>
Group administrator
Fri 20 Dec 2019 06:09:42 PM UTC, comment #8: 

If the crash occurs in sparse.tst when running the full test suite but not when running sparse.tst by itself in a fresh Octave session, then it is possible that the real problem is in some earlier test that doesn't immediately cause a crash but corrupts some memory that causes the later crash in sparse.tst

Does the crash occur if you make something like the following change to remove the directories for the "integrated test scripts" and run the test suite again and see if it still crashes?  You can continue bisecting by adding and removing tests until you find the test directory, file, and then hopefully the individual test that must be executed before the sparse.tst file to cause the crash.  Then maybe we can understand exactly what is causing the crash.


diff --git a/scripts/testfun/__run_test_suite__.m b/scripts/testfun/__run_test_suite__.m
--- a/scripts/testfun/__run_test_suite__.m
+++ b/scripts/testfun/__run_test_suite__.m
@@ -30,7 +30,11 @@ function [pass, fail, xfail, xbug, skip,
   fixedtestdir = fullfile (testsdir, "fixed");
   fcnfiledir = __octave_config_info__ ("fcnfiledir");
   if (nargin == 0)
-    fcndirs = { liboctavetestdir, libinterptestdir, fcnfiledir };
+    fcndirs = { ...
+##                liboctavetestdir, ...
+##                libinterptestdir, ...
+##                fcnfiledir
+              };
     fixedtestdirs = { fixedtestdir };
   endif
   files_with_no_tests = {};


I did this same kind of thing recently to find why publish.tst was suddenly failing when run as part of the test suite but not when running the test by itself (bug #57424 and bug #57439).  It took some time but finally uncovered a problem with function handles caused by a "clear functions" command in the file test/bug-35881/bug35881.m.

John W. Eaton <jwe>
Group administrator
Fri 20 Dec 2019 05:36:25 PM UTC, comment #7: 

The backtrace does show the code going through SparseComplexMatrix::inverse at level #5.  The code for this is in liboctave/array/CSparse.cc.  The bit that looks relevant is


  else
    {
      if (mattype.ishermitian ())
        {
          MatrixType tmp_typ (MatrixType::Upper);
          octave::math::sparse_chol<SparseComplexMatrix> fact (*this, info, false);
          rcond = fact.rcond ();
          if (info == 0)
            {
              double rcond2;
              SparseMatrix Q = fact.Q ();
              SparseComplexMatrix InvL = fact.L ().transpose ().
                                         tinverse (tmp_typ, info, rcond2,
                                                   true, false);
              ret = Q * InvL.hermitian () * InvL * Q.transpose ();
            }
          else
            {
              // Matrix is either singular or not positive definite
              mattype.mark_as_unsymmetric ();
            }
        }

      if (! mattype.ishermitian ())
        {
          octave_idx_type n = rows ();
          ColumnVector Qinit(n);
          for (octave_idx_type i = 0; i < n; i++)
            Qinit(i) = i;

          MatrixType tmp_typ (MatrixType::Upper);
          octave::math::sparse_lu<SparseComplexMatrix> fact (*this,
                                                             Qinit, Matrix (),
                                                             false, false);
          rcond = fact.rcond ();
          if (rcond == 0.0)
            {
              // Return all Inf matrix with sparsity pattern of input.
              octave_idx_type nz = nnz ();
              ret = SparseComplexMatrix (rows (), cols (), nz);
              std::fill (ret.xdata (), ret.xdata () + nz,
                         octave::numeric_limits<double>::Inf ());
              std::copy_n (ridx (), nz, ret.xridx ());
              std::copy_n (cidx (), cols () + 1, ret.xcidx ());

              return ret;
            }
          double rcond2;
          SparseComplexMatrix InvL = fact.L ().transpose ().
                                     tinverse (tmp_typ, info, rcond2,
                                               true, false);
          SparseComplexMatrix InvU = fact.U ().
                                     tinverse (tmp_typ, info, rcond2,
                                               true, false).transpose ();
          ret = fact.Pc ().transpose () * InvU * InvL * fact.Pr ();
        }
    }


Rik <rik5>
Group administrator
Fri 20 Dec 2019 05:14:54 PM UTC, comment #6: 

I don't think the culprit is the first test.  That test just calls sparse(), and the UMFPACK library is called for inverse.

If you want to be certain, I would try the show-octave-dbstack macro to see where in the m-file code you were.

Alternatively, flush the stdout stream after the call to disp ('Test 1') with the additional line


fflush (stdout);


Rik <rik5>
Group administrator
Fri 20 Dec 2019 05:11:19 PM UTC, comment #5: 

@Markus: You might want to take a look at the macros available in .gdbinit which are specific to Octave-debugging.



Usage: display-dims DIM_VECTOR
Display the contents of an Octave dimension vector.



Usage: display-dense-array ARRAY
Display the contents of an ordinary, i.e., dense Octave array.

See also [display-sparse-array] for showing the contents
of sparse Octave arrays.



Usage: display-sparse-array SPARSE_ARRAY
Display the contents of a sparse Octave array.

See also [display-dense-array] for showing the contents
of ordinary Octave arrays.



Usage: show-octave-dbstack
Display the contents of the current Octave debugging stack.

This is the function stack that the Octave interpreter is processing
and will be different from the C++ stack being debugged with gdb.


The last one is useful for checking where in an m-file Octave was when the debugger started.  "display-dims" is easier than looking at each dim component separately.

Rik <rik5>
Group administrator
Fri 20 Dec 2019 03:11:06 PM UTC, comment #4: 

In the meantime, I sprinkled a few "disp"s into sparse.tst to narrow down which of the tests causes the error. It looks like, it is the very first one:

%!test # segfault test from edd@debian.org
%! disp ('Test 1')
%! n = 510;
%! sparse (kron ((1:n)', ones (n,1)), kron (ones (n,1), (1:n)'), ones (n));


At least, "Test 1" appears at the command prompt and "Test 2" of the following test does not:

%% segfault tests from Fabian@isas-berlin.de
%% Note that the last four do not fail, but rather give a warning
%% of a singular matrix, which is consistent with the full matrix
%% behavior.  They are therefore disabled.
%!testif HAVE_UMFPACK
%! disp ('Test 2')
%! assert (inv (sparse ([1,1;1,1+i])), sparse ([1-1i,1i;1i,-1i]), 10*eps);
%#!error inv ( sparse ([1,1;1,1]  ) );
%#!error inv ( sparse ([0,0;0,1]  ) );
%#!error inv ( sparse ([0,0;0,1+i]) );
%#!error inv ( sparse ([0,0;0,0]  ) );


Might still be that some buffering is happening and it is actually a later test that is triggering the segmentation fault.

This time n_row has a different value:

Thread 15 received signal SIGSEGV, Segmentation fault.
0x0000000062549778 in ?? ()
(gdb) bt
#0  0x0000000062549778 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) set $pc = *(void **)$rsp
(gdb) bt
#0  0x00000000433e6cc5 in symbolic_analysis ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\libumfpack.dll
#1  0x0000000000000002 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) set $pc = *(void **)$rsp
(gdb) bt
#0  0x00000000433e6cc5 in symbolic_analysis ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\libumfpack.dll
#1  0x00000000433eb6e9 in umfpack_zl_qsymbolic ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\libumfpack.dll
#2  0x000000005beb7d60 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) set $pc = *(void **)$rsp
(gdb) bt
#0  0x00000000433e6cc5 in symbolic_analysis ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\libumfpack.dll
#1  0x00000000433eb6e9 in umfpack_zl_qsymbolic ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\libumfpack.dll
#2  0x0000000069ab1edc in octave::math::umfpack_qsymbolic<std::complex<double> > (Info=0xb2644490, Control=0x53410000, Symbolic=0x5beb7d98,
    Qinit=0x59e2baa0, Az=0x5beb7a90, Ai=0x5beb7d60, Ap=0x59e2baa0, n_col=2, n_row=1508030048)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/numeric/sparse-lu.cc:339
#3  octave::math::sparse_lu<SparseComplexMatrix>::sparse_lu (this=0x2d0, a=..., Qinit=..., piv_thres=..., scale=240, FixedQ=false,
    droptol=6.9525070095186182e-310, milu=false, udiag=false)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/numeric/sparse-lu.cc:704
#4  0x0000000069c80530 in vtable for MArray<double> ()
   from D:\Repositories\Octave\test\octave-2019-12-19-17-17-w64_gtk2\octave-2019-12-19-17-17-w64\mingw64\bin\liboctave-7.dll
#5  0x0000000068fafd7f in SparseComplexMatrix::inverse (this=0x5beb8040, this@entry=0x5beb8000, mattype=..., info=@0x5beb8040: 1774714144,
    rcond=@0x5beb7ed8: 0, calc_cond=calc_cond@entry=true)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/array/MArray.h:70
#6  0x000000000e4ffd49 in Finv (args=..., nargout=1)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/corefcn/inv.cc:176
#7  0x000000000e0ec9e5 in octave_builtin::call (this=0x5832ba20, tw=..., nargout=1, args=...)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/octave-value/ov-builtin.cc:62
#8  0x000000000e22e8c1 in octave::tree_index_expression::evaluate_n (this=0x5dc82d90, tw=..., nargout=1)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/array/dim-vector.h:158
#9  0x000000000e85fc5f in octave::tree_index_expression::evaluate (this=<optimized out>, tw=..., nargout=<optimized out>)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-idx.h:106
#10 0x000000000e215ef0 in octave::tree_evaluator::convert_to_const_vector (this=this@entry=0x5a0596b0, arg_list=arg_list@entry=0xb278d570,
    object=object@entry=0x0)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-eval.cc:1399
#11 0x000000000e22e06c in octave::tree_index_expression::evaluate_n (this=0x5dc83fb0, tw=..., nargout=0)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-idx.cc:492
#12 0x000000000e85fc5f in octave::tree_index_expression::evaluate (this=<optimized out>, tw=..., nargout=<optimized out>)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-idx.h:106
#13 0x000000000e21ea83 in octave::tree_evaluator::visit_statement (this=0x5a0596b0, stmt=...)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-eval.cc:2720
#14 0x000000000e20feec in octave::tree_statement::accept (tw=..., this=0xa7ea8a20)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-stmt.h:121
#15 octave::tree_evaluator::visit_statement_list (this=0x5a0596b0, lst=...)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-eval.cc:2779
#16 0x000000000e2201bd in octave::tree_statement_list::accept (tw=..., this=0xb278dc60)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/parse-tree/pt-stmt.h:198
--Type <RET> for more, q to quit, c to continue without paging--


So maybe something that isn't correctly initialized?

If I change to "Finv" and inspect the size of "m", it still seems to be ok:

(gdb) up 6
#6  0x000000000e4ffd49 in Finv (args=..., nargout=1)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/corefcn/inv.cc:176
176     /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/corefcn/inv.cc: No such file or directory.
(gdb) p m
$2 = {<MSparse<std::complex<double> >> = {<Sparse<std::complex<double> >> = {_vptr.Sparse = 0xea11740 <vtable for SparseComplexMatrix+16>,
      rep = 0xa7ea84a0, dimensions = {rep = 0xb278f7a0}}, <No data fields>}, <No data fields>}
(gdb) p m.dimensions
$3 = {rep = 0xb278f7a0}
(gdb) p m.dimensions->rep
$4 = (octave_idx_type *) 0xb278f7a0
(gdb) p *m.dimensions->rep
$5 = 2
(gdb) p *(m.dimensions->rep+1)
$7 = 2


Tbh, I don't understand the stack yet. I would have expected "SparseComplexMatrix::inverse" to reference CSparse.cc as its source. But maybe there is only so much one could expect from inspecting a stack that was corrupt from the start.
Interestingly, "this" changes in that function (not sure why). This is also where the dimensions seem to be lost:

(gdb) p *(this@entry->rep)
$22 = {d = 0xb2644490, r = 0xb278eda0, c = 0x59e2b660, nzmx = 4, nrows = 2, ncols = 2, count = {m_count = {<std::__atomic_base<long long>> = {
        static _S_alignment = 8, _M_i = 2}, <No data fields>}}}
(gdb) p *(this->rep)
$23 = {d = 0x59ffcf00, r = 0x54d22a10, c = 0x5a3a8ed0, nzmx = 1, nrows = 0, ncols = 0, count = {m_count = {<std::__atomic_base<long long>> = {
        static _S_alignment = 8, _M_i = 5}, <No data fields>}}}


Markus Mützel <mmuetzel>
Group administrator
Fri 20 Dec 2019 11:44:14 AM UTC, comment #3: 

"test sparse.tst" doesn't trigger an error. All 1024 test pass.
But I see a bunch of warnings, I didn't notice when running the test suite on Ubuntu:

warning: lu: function may fail when called with less than 4 output arguments and a sparse input
warning: called from
    __test__ at line 3 column 10
    test at line 660 column 11


Also "clear all" followed by running the test again does not fail.

I'm reverted to the suitesparse version we used until a few weeks ago ([1] and a few follow up changes). Completing the clean cross-build will take some time. I'll report back if this makes the issue disappear.

[1]: https://hg.octave.org/mxe-octave/rev/af2966900462

Markus Mützel <mmuetzel>
Group administrator
Thu 19 Dec 2019 09:56:10 PM UTC, comment #2: 

Looking at the last bit of the backtrace which is still within Octave before calling out to libumfpack I see


#2  0x0000000001a3eebc in octave::math::umfpack_qsymbolic<std::complex<double> > (Info=0x779b5ad0,
    Control=0x61fb0000, Symbolic=0x67d27d78, Qinit=0x77904170, Az=0x67d27a70, Ai=0x67d27d40, Ap=0x77904170, n_col=2,
    n_row=2005942032)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/numeric/sparse-lu.cc:339


The number of rows, n_row=2005942032, looks highly suspect given that the number of columns is only 2.

If you copy sparse.tst in to a local directory and then run "test sparse.tst" does it still segfault?  Or is it only after the test suite has been running?

Rik <rik5>
Group administrator
Tue 17 Dec 2019 08:17:03 PM UTC, comment #1: 

Here a hopefully a little more useful backtrace:

[Thread 17324.0x374 exited with code 20115]
[Thread 17324.0x15a4 exited with code 0]

Thread 15 received signal SIGSEGV, Segmentation fault.
0x000000006f3a9728 in ?? ()
(gdb) bt
#0  0x000000006f3a9728 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) set $pc = *(void **)$rsp
(gdb) bt
#0  0x000000006a6a6cc5 in symbolic_analysis ()
   from D:\Repositories\Octave\test\octave-2019-12-17-20-10-w64_no_strip\octave-2019-12-17-20-10-w64\mingw64\bin\libumfpack.dll
#1  0x0000000000000002 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) set $pc = *(void **)$rsp
(gdb) bt
#0  0x000000006a6a6cc5 in symbolic_analysis ()
   from D:\Repositories\Octave\test\octave-2019-12-17-20-10-w64_no_strip\octave-2019-12-17-20-10-w64\mingw64\bin\libumfpack.dll
#1  0x000000006a6ab6e9 in umfpack_zl_qsymbolic ()
   from D:\Repositories\Octave\test\octave-2019-12-17-20-10-w64_no_strip\octave-2019-12-17-20-10-w64\mingw64\bin\libumfpack.dll
#2  0x0000000067d27d40 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) set $pc = *(void **)$rsp
(gdb) bt
#0  0x000000006a6a6cc5 in symbolic_analysis ()
   from D:\Repositories\Octave\test\octave-2019-12-17-20-10-w64_no_strip\octave-2019-12-17-20-10-w64\mingw64\bin\libumfpack.dll
#1  0x000000006a6ab6e9 in umfpack_zl_qsymbolic ()
   from D:\Repositories\Octave\test\octave-2019-12-17-20-10-w64_no_strip\octave-2019-12-17-20-10-w64\mingw64\bin\libumfpack.dll
#2  0x0000000001a3eebc in octave::math::umfpack_qsymbolic<std::complex<double> > (Info=0x779b5ad0,
    Control=0x61fb0000, Symbolic=0x67d27d78, Qinit=0x77904170, Az=0x67d27a70, Ai=0x67d27d40, Ap=0x77904170, n_col=2,
    n_row=2005942032)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/numeric/sparse-lu.cc:339
#3  octave::math::sparse_lu<SparseComplexMatrix>::sparse_lu (this=0x1c09360 <vtable for Array<double>+16>, a=...,
    Qinit=..., piv_thres=..., scale=32, FixedQ=false, droptol=6.9525070095186182e-310, milu=false, udiag=false)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/numeric/sparse-lu.cc:704
#4  0x0000000001c097f0 in vtable for MArray<double> ()
   from D:\Repositories\Octave\test\octave-2019-12-17-20-10-w64_no_strip\octave-2019-12-17-20-10-w64\mingw64\bin\liboctave-7.dll
#5  0x0000000000f7e98f in SparseComplexMatrix::inverse (this=0x67d28020, this@entry=0x67d27fe0, mattype=...,
    info=@0x67d28020: 29396960, rcond=@0x67d27eb8: 0, calc_cond=calc_cond@entry=true)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/array/MArray.h:70
#6  0x000000001f2eb539 in Finv (args=..., nargout=1)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/corefcn/inv.cc:176
#7  0x000000001eed9e35 in octave_builtin::call (this=0x73a10b50, tw=..., nargout=1, args=...)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/libinterp/octave-value/ov-builtin.cc:62
#8  0x000000001f01aca1 in octave::tree_index_expression::evaluate_n (this=0xb23c4f40, tw=..., nargout=1)
    at /home/osboxes/Repositories/Octave/mxe-octave-1/tmp-default-octave/octave-6.0.0/liboctave/array/dim-vector.h:158
#9  0x000000001f6488bf in octave::tree_index_expression::evaluate (this=<optimized out>, tw=...,
--Type <RET> for more, q to quit, c to continue without paging--


Markus Mützel <mmuetzel>
Group administrator
Tue 17 Dec 2019 06:54:48 PM UTC, original submission:  

When running the test suite on Windows with hg id 56d94f86a659, Octave exits unexpectedly on sparse.tst.
Unfortunately, the backtrace isn't very helpful:

Thread 12 received signal SIGSEGV, Segmentation fault.
0x0000000062549728 in ?? ()
(gdb) bt
#0  0x0000000062549728 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)


I forgot to disable stripping the files. Would that have helped?
Is there anything I could do to make this more helpful?

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #48099:  bug57435_suitesparse_mlock.patch added by mmuetzel (2KiB - application/octet-stream)
file #48098:  bisect_tst_files.m added by rik5 (5KiB - text/x-matlab)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-27 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2019-12-22 mmuetzel StatusPatch Submitted Ready For Test
    2019-12-22 mmuetzel Attached File- Added bug57435_suitesparse_mlock.patch, #48099
        StatusNone Patch Submitted
    2019-12-20 rik5 Attached File- Added bisect_tst_files.m, #48098

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code