It was some time since the last time the test suite crashed at the test for gmres.m. But it happened again here on the clang-debian builder:
http://buildbot.octave.org:8010/#/builders/32/builds/1365/steps/7/logs/stdio

Another crash at the same tests:
http://buildbot.octave.org:8010/#/builders/36/builds/287/steps/7/logs/stdio

Another crash during the tests of gmres:
http://buildbot.octave.org:8010/#/builders/36/builds/285/steps/7/logs/stdio

Looks like that wasn't it either...

Assuming the crash occurred during the tests for gmres, I can't see where that would plot anything. But I might be missing something...

It looks like this still occurs occasionally on jwe's new buildbots:
http://buildbot.octave.org:8010/#/builders/32/builds/960/steps/7/logs/stdio

  sparse/etreeplot.m ............................................. pass    2/2
fatal: caught signal Segmentation fault -- stopping myself...
/bin/bash: line 1: 78967 Segmentation fault      /bin/bash ../run-octave --no-init-file --silent --no-history -p /scratch/buildbot/workers/jwe-debian-x86_64-1/clang-debian/build/test/mex /scratch/buildbot/workers/jwe-debian-x86_64-1/clang-debian/build/../src/test/fntests.m /scratch/buildbot/workers/jwe-debian-x86_64-1/clang-debian/build/../src/test
make[3]: *** [Makefile:32234: check-local] Error 139
  sparse/gmres.m .................................................make[3]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-1/clang-debian/build'
make[2]: *** [Makefile:28334: check-am] Error 2
make[2]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-1/clang-debian/build'
make[1]: *** [Makefile:28036: check-recursive] Error 1
make[1]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-1/clang-debian/build'
make: *** [Makefile:28336: check] Error 2
program finished with exit code 2

Here is what I see for blas:

devnull> ldd /usr/local/octave/dev/libexec/octave/7.0.0/exec/x86_64-pc-linux-gnu/octave-gui | grep blas
        libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007fda3f660000)
        libopenblas.so.0 => /usr/lib/x86_64-linux-gnu/libopenblas.so.0 (0x00007fda3bd40000)

devnull> ls -l /usr/lib/x86_64-linux-gnu/libblas.so.3
lrwxrwxrwx 1 root root 47 Jan 11  2019 /usr/lib/x86_64-linux-gnu/libblas.so.3 -> /etc/alternatives/libblas.so.3-x86_64-linux-gnu

devnull> ls -l /etc/alternatives/libblas.so.3-x86_64-linux-gnu
lrwxrwxrwx 1 root root 54 May 20  2020 /etc/alternatives/libblas.so.3-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas-serial/libblas.so.3

devnull> ls -l /usr/lib/x86_64-linux-gnu/openblas-serial/libblas.so.3
-rw-r--r-- 1 root root 317712 Apr 18 04:36 /usr/lib/x86_64-linux-gnu/openblas-serial/libblas.so.3

devnull> ls -l /usr/lib/x86_64-linux-gnu/libopenblas.so.0
lrwxrwxrwx 1 root root 51 Nov  4  2019 /usr/lib/x86_64-linux-gnu/libopenblas.so.0 -> /etc/alternatives/libopenblas.so.0-x86_64-linux-gnu

devnull> ls -l /etc/alternatives/libopenblas.so.0-x86_64-linux-gnu
lrwxrwxrwx 1 root root 59 Nov  4  2019 /etc/alternatives/libopenblas.so.0-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblas.so.0

devnull> ls -l /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblas.so.0
lrwxrwxrwx 1 root root 23 Apr 18 04:36 /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblas.so.0 -> libopenblasp-r0.3.13.so

so there do appear to be two different BLAS libraries involved.  I've been configuring with --with-blas=-lblas.  I thought that would do the right thing with the Debian alternatives mechanism.

For lapack I see

devnull> ldd /usr/local/octave/dev/libexec/octave/7.0.0/exec/x86_64-pc-linux-gnu/octave-gui | grep lapack
        liblapack.so.3 => /usr/lib/x86_64-linux-gnu/liblapack.so.3 (0x00007fdd8c978000)

devnull> ls -l /usr/lib/x86_64-linux-gnu/liblapack.so.3
lrwxrwxrwx 1 root root 49 Jan 11  2019 /usr/lib/x86_64-linux-gnu/liblapack.so.3 -> /etc/alternatives/liblapack.so.3-x86_64-linux-gnu

devnull> ls -l /etc/alternatives/liblapack.so.3-x86_64-linux-gnu
lrwxrwxrwx 1 root root 56 May 20  2020 /etc/alternatives/liblapack.so.3-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/openblas-serial/liblapack.so.3

devnull> ls -l /usr/lib/x86_64-linux-gnu/openblas-serial/liblapack.so.3
-rw-r--r-- 1 root root 6920304 Apr 18 04:36 /usr/lib/x86_64-linux-gnu/openblas-serial/liblapack.so.3

I'm reopening at least temporarily because after all this time I finally encountered a crash in the gmres.m tests while running "make check" from the command line and I happened to have core files enabled:

  ...
  sparse/gmres.m .................................................fatal: caught signal Segmentation fault -- stopping myself...
/bin/bash: line 1: 2189756 Segmentation fault      (core dumped) /bin/bash ../run-octave --no-init-file --silent --no-history -p /home/jwe/build/octave/test/mex /home/jwe/src/octave/test/fntests.m /home/jwe/src/octave/test
make[3]: *** [Makefile:32312: check-local] Error 139
make[3]: Leaving directory '/net/devnull/scratch/jwe/build/octave'
make[2]: *** [Makefile:28396: check-am] Error 2
make[2]: Leaving directory '/net/devnull/scratch/jwe/build/octave'
make[1]: *** [Makefile:28098: check-recursive] Error 1
make[1]: Leaving directory '/net/devnull/scratch/jwe/build/octave'
make: *** [Makefile:28398: check] Error 2

If this were a problem with the Linux memory manager I would expect the process to be killed, not to stop with a segfault.

I have the following OpenBLAS packages installed on the system where the crash happened:

ii  libopenblas-dev:amd64         0.3.13+ds-3  amd64        Optimized BLAS (linear algebra) library (dev, meta)
ii  libopenblas-pthread-dev:amd64 0.3.13+ds-3  amd64        Optimized BLAS (linear algebra) library (dev, pthread)
ii  libopenblas0:amd64            0.3.13+ds-3  amd64        Optimized BLAS (linear algebra) library (meta)
ii  libopenblas0-pthread:amd64    0.3.13+ds-3  amd64        Optimized BLAS (linear algebra) library (shared lib, pthread)
ii  libopenblas0-serial:amd64     0.3.13+ds-3  amd64        Optimized BLAS (linear algebra) library (shared lib, serial)

These should be the same on all of my buildbot systems.

Here is the top of the call stack at the point of the crash:

(gdb) where
#0  0x00007fd339cf804c in dgemm_incopy_PILEDRIVER () at /usr/lib/x86_64-linux-gnu/libopenblas.so.0
#1  0x00007fd33ac45830 in gotoblas () at /usr/lib/x86_64-linux-gnu/libopenblas.so.0
#2  0x0000000000000001 in  ()
#3  0x00007fd338b7b345 in dgemm_tn () at /usr/lib/x86_64-linux-gnu/libopenblas.so.0
#4  0x00007fd33c2c5908 in dgemm_ () at /usr/lib/x86_64-linux-gnu/libblas.so.3
#5  0x00007fd33cb99f1c in dlarfb_ () at /usr/lib/x86_64-linux-gnu/liblapack.so.3
#6  0x00007fd33cbd7909 in dormqr_ () at /usr/lib/x86_64-linux-gnu/liblapack.so.3
#7  0x00007fd33cbd560a in dormbr_ () at /usr/lib/x86_64-linux-gnu/liblapack.so.3
#8  0x00007fd33cb1e778 in dgelsd_ () at /usr/lib/x86_64-linux-gnu/liblapack.so.3
#9  0x00007fd33ff37b7e in Matrix::lssolve(Matrix const&, long&, long&, double&) const
    (this=0x7fd2f3fe1fb0, b=<optimized out>, info=@0x7fd2f3fe1f30: 0, rank=@0x7fd2f3fe1e68: 0, rcon=@0x7fd2f3fe1f38: -1)
    at /home/jwe/src/octave/liboctave/array/dMatrix.cc:2051
#10 0x00007fd33ff3829b in Matrix::solve(MatrixType&, Matrix const&, long&, double&, void (*)(double), bool, blas_trans_type) const
    (this=this@entry=0x7fd2f3fe1fb0, mattype=..., b=
    ..., info=@0x7fd2f3fe1f30: 0, rcon=@0x7fd2f3fe1f38: -1, sing_handler=0x7fd3418606f0 <solve_singularity_warning(double)>, singular_fallback=true, transt=blas_no_trans) at /home/jwe/src/octave/liboctave/array/dMatrix.cc:1625
#11 0x00007fd341862093 in xleftdiv(Matrix const&, Matrix const&, MatrixType&, blas_trans_type) (a=..., b=..., typ=..., transt=transt@entry=blas_no_trans)
    at /home/jwe/src/octave/libinterp/corefcn/xdiv.cc:353
#12 0x00007fd340ebc7df in oct_binop_ldiv(octave_base_value const&, octave_base_value const&) (a1=<optimized out>, a2=...)
    at /home/jwe/src/octave/libinterp/operators/op-m-m.cc:91
#13 0x00007fd341210f34 in octave::binary_op(octave::type_info&, octave_value::binary_op, octave_value const&, octave_value const&)
    (ti=..., op=octave_value::op_ldiv, v1=..., v2=...) at /home/jwe/src/octave/libinterp/octave-value/ov.h:1417
#14 0x00007fd3412b4413 in octave::tree_binary_expression::evaluate(octave::tree_evaluator&, int) (this=0x7fd1bd2b2e40, tw=...)
    at /home/jwe/src/octave/libinterp/parse-tree/pt-binop.cc:140
#15 0x00007fd3412b02aa in octave::tree_simple_assignment::evaluate(octave::tree_evaluator&, int) (this=0x7fd1bd2b2ed0, tw=...)
    at /home/jwe/src/octave/libinterp/parse-tree/pt-assign.cc:101
#16 0x00007fd3412cd89d in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fd2e8005878, stmt=<optimized out>)
    at /home/jwe/src/octave/libinterp/parse-tree/pt-eval.cc:3772
#17 0x00007fd3412bbc04 in octave::tree_statement::accept(octave::tree_walker&) (tw=..., this=0x7fd1bd2b2f10)
    at /home/jwe/src/octave/libinterp/parse-tree/pt-stmt.h:124
#18 octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fd2e8005878, lst=...)
    at /home/jwe/src/octave/libinterp/parse-tree/pt-eval.cc:3857
#19 0x00007fd3412ccc5f in octave::tree_statement_list::accept(octave::tree_walker&) (tw=..., this=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--
    at /home/jwe/src/octave/libinterp/parse-tree/pt-stmt.h:201
#20 octave::tree_evaluator::visit_while_command(octave::tree_while_command&) (this=0x7fd2e8005878, cmd=...)
    at /home/jwe/src/octave/libinterp/parse-tree/pt-eval.cc:4190
#21 0x00007fd3412cd806 in octave::tree_evaluator::visit_statement(octave::tree_statement&) (this=0x7fd2e8005878, stmt=...)
    at /home/jwe/src/octave/libinterp/parse-tree/pt-eval.cc:3747
#22 0x00007fd3412bbc04 in octave::tree_statement::accept(octave::tree_walker&) (tw=..., this=0x7fd1bd2ba5f0)
    at /home/jwe/src/octave/libinterp/parse-tree/pt-stmt.h:124
#23 octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) (this=0x7fd2e8005878, lst=...)
    at /home/jwe/src/octave/libinterp/parse-tree/pt-eval.cc:3857
#24 0x00007fd3412c4ef7 in octave::tree_statement_list::accept(octave::tree_walker&) (tw=..., this=0x7fd1bd25fc30)
    at /home/jwe/src/octave/libinterp/parse-tree/pt-stmt.h:201
#25 octave::tree_evaluator::execute_user_function(octave_user_function&, int, octave_value_list const&) (this=this@entry=0x7fd2e8005878, user_function=
    ..., nargout=nargout@entry=2, xargs=...) at /home/jwe/src/octave/libinterp/parse-tree/pt-eval.cc:3503

The Matrix:lssolve stack frame shows

#9  0x00007fd33ff37b7e in Matrix::lssolve (this=0x7fd2f3fe1fb0, b=..., info=@0x7fd2f3fe1f30: 0, rank=@0x7fd2f3fe1e68: 0, rcon=@0x7fd2f3fe1f38: -1)
    at /home/jwe/src/octave/liboctave/array/dMatrix.cc:2051
2051                  F77_XFCN (dgelsd, DGELSD, (m, n, nrhs, tmp_data, m, pretval,
(gdb) list
2046                  rcon = octave::numeric_limits<double>::NaN ();
2047                  retval = Matrix (n, b_nc, octave::numeric_limits<double>::NaN ());
2048                }
2049              else
2050                {
2051                  F77_XFCN (dgelsd, DGELSD, (m, n, nrhs, tmp_data, m, pretval,
2052                                             maxmn, ps, rcon, tmp_rank,
2053                                             work.fortran_vec (), lwork,
2054                                             piwork, tmp_info));
2055
(gdb) p m
$1 = 76
(gdb) p n
$2 = 75
(gdb) p nrhs
$3 = 1
(gdb) p tmp_data
$4 = <optimized out>
(gdb) p m
$5 = 76
(gdb) p maxmn
$6 = 76
(gdb) p ps
$7 = <optimized out>
(gdb) p rcon
$8 = (double &) @0x7fd2f3fe1f38: -1
(gdb) p lwork
$9 = 6601
(gdb) p piwork
$10 = <optimized out>

In lssolve, I see that we want to query the LAPACK functions for the workspace requirements but there are comments there about the calculation being broken in some versions of LAPACK so we also compute the sizes.  Are the formulas correct?  I can't tell from the info on the stack after the crash whether we end up using our own calculated values or the ones that LAPACK provides.  Our calculation is complicated, so it seems quite possible that it is incorrect, or that the requirements may have changed since this code was written.

If it is not a problem with the workspace size that results in a memory fault, then the problem begins to look more like a bug in DGELSD or OpenBLAS.

This crash occurred after my changes here:

http://hg.savannah.gnu.org/hgweb/octave/rev/3ab696e02f55

but I don't think those have anything to do with this crash as they are completely unrelated to matrix calculations.

With some effort, I might be able to use gdb to extract the data necessary to recreate the call to lssolve that resulted in the crash, but if this is a random memory issue the crash may still not be easily reproducible.

Cross-referencing a thread on Discourse that is relevant for comment #163:
https://octave.discourse.group/t/bists-with-high-memory-consumption/1275

The Windows 32bit runners on GitHub are failing this test with the following error:

>>>>> processing C:\msys64\mingw32\share\octave\7.0.0\m\sparse\gmres.m
***** test
 dim = 100;
 A = spdiags ([[1./(2:2:2*(dim-1)) 0]; 1./(1:2:2*dim-1); ...
 [0 1./(2:2:2*(dim-1))]]', -1:1, dim, dim);
 A = A'*A;
 b = rand (dim, 1);
 [x, resvec] = gmres (@(x) A*x, b, dim, 1e-10, dim,...
                      @(x) x./diag (A), [], []);
 assert (x, A\b, 1e-9*norm (x, Inf));
 [x, flag] = gmres (@(x) A*x, b, dim, 1e-10, 1e6,...
                    @(x) diag (diag (A)) \ x, [], []);
 assert (x, A\b, 1e-9*norm (x, Inf));
 [x, flag] = gmres (@(x) A*x, b, dim, 1e-10, 1e6,...
                    @(x) x ./ diag (A), [], []);
 assert (x, A\b, 1e-7*norm (x, Inf));
!!!!! test failed
out of memory or dimension too large for Octave's index type

So, it could be the Linux OOM killer at work here.

And another similar one on gcc-debian:
http://buildbot.octave.org:8010/#/builders/33/builds/654/steps/7/logs/stdio

Another one with "sparse/gmres.m" as the last line in the logs. This time on stable-gcc-lto-debian:
http://buildbot.octave.org:8010/#/builders/35/builds/231/steps/7/logs/stdio

And if I build w/o java it still crashes:

May 13 15:09:49 ryzen systemd-coredump[1661457]: Process 1659460 (lt-octave-gui) of user 1001 dumped core.

                                                 Stack trace of thread 1659599:
                                                 #0  0x00007f0496a5a82a dgemm_ (libopenblas.so.0)
                                                 #1  0x00007f049d8d1714 n/a (/home/dima/src/octave/gcc_debug/liboctave/.libs/liboctave.so.8.0.1)
                                                 #2  0x00007f049d8d17b5 n/a (/home/dima/src/octave/gcc_debug/liboctave/.libs/liboctave.so.8.0.1)
                                                 #3  0x00007f04a204d6bb n/a (/home/dima/src/octave/gcc_debug/libinterp/.libs/liboctinterp.so.8.0.1)

Dmitri.
--

It is a long shot, but what is openblas library on those debian systems? In fedora openblas is serial, openblasp is pthread, and
openblaso is OpenMP. I have some occasional segafaults in odd places during make check on AMD/ryzen systems when I used default openblas library (none of them happens during individual tests). Lately it is on "make check" at bug-53468 (???).

It is not happening on intel cpus (fedora's buildbots are on i7-2600 at the moment, but used to be on amd fx-8350).
The crash does not happen if I use OpenMP interface
(I have a vague recollection that openmp is recommended for multithreaded application so the threads don't clash).

Dmitri.
--

Another one on gcc-debian with sparse/gmres.m as the last test in the logs:
http://buildbot.octave.org:8010/#/builders/33/builds/584/steps/7/logs/stdio

I have the same hg_id. "make check" passes with fail on memory.m
(this may be due to ASAN?). But no crash.

what is output of "glxinfo -B"?

Do you get a crash with  LIBGL_ALWAYS_SOFTWARE=1 ?

Dmitri.
--

It's not necessary to print. `plot([0 1])` already crashes with the same stack.

I cannot reproduce it on Fedora 34.

Dmitri.
--

ASAN segfaults on exit. You should run
make as:

ASAN_OPTIONS="leak_check_at_exit=0" make

Dmitri.
--

The bugs reported so far on this page attached below. All occurred under debian. None under Fedora.

clang-5.0-debian
special-matrix/hadamard.m
strings/hex2dec.m
statistics/range.m

clang-debian
plot/draw/hist.m
polynomial/polyval.m
optimization/lsqnonneg.m
set/ismember.m
_run_test_suite_.m before "print_test_file_name" and after "print_pass_fail" ?

stable-clang-debian
io.tst
specfun/factorial.m

gcc-7-debian
plot/draw/trisurf.m

stable-gcc-7-debian
sparse/gmres.m

gcc-lto-debian
java/javaaddpath.m
sparse/gmres.m
sparse/gmres.m
sparse/gmres.m

stable-gcc-lto-debian
struct.tst

gcc-debian
sparse/gmres.m
sparse/gmres.m
sparse/gmres.m

Another one with "sparse/gmres.m":
http://buildbot.octave.org:8010/#/builders/35/builds/203/steps/7/logs/stdio

Just fyi:

On fedora coredumps are saved by systemd-coredump to /var/lib/systemd/coredump/

The trick is to preserve the actual executable that caused it
(so you can get an actual trace).
If I am quick I'd stop buildbots before a new build overwrites the
last one.

Dmitri.
--

Yet another one with "sparse/gmres.m":
http://buildbot.octave.org:8010/#/builders/33/builds/407/steps/7/logs/stdio

I'm not sure if I correctly understand the backtrace. It looks like somehow an output from gdb ("Python Exception <class 'gdb.error'> There is no member named _M_dataplus.") made it into Octave's output buffer (in thread 22)...
Is that expected?

IIRC, there was a report in the past where Java didn't play well with gdb. Could this be one of those cases again?

I disabled the buildbot jobs for the build mentioned in comment #132 so that it would not be wiped out by a later execution of the buildbot stable-clan-debian build.

I didn't find a core file there.  Maybe I missed it?  Anyway, I was able to execute the tests in a loop until I triggered a segfault.  The output from make at the time of the segfault was

sparse/spconvert.m .............................................fatal: caught signal Segmentation fault -- stopping myself...
/bin/bash: line 1: 4174536 Segmentation fault      (core dumped) /bin/bash ../run-octave --no-init-file --silent --no-history -p /scratch/buildbot/workers/jwe-debian-x86_64-0/stable-clang-debian/build/test/mex /scratch/buildbot/workers/jwe-debian-x86_64-0/stable-clang-debian/build/../src/test/fntests.m /scratch/buildbot/workers/jwe-debian-x86_64-0/stable-clang-debian/build/../src/test
make[3]: *** [Makefile:31664: check-local] Error 139
make[3]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-0/stable-clang-debian-save/build'
make[2]: *** [Makefile:27801: check-am] Error 2
make[2]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-0/stable-clang-debian-save/build'
make[1]: *** [Makefile:27503: check-recursive] Error 1
make[1]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-0/stable-clang-debian-save/build'
make: *** [Makefile:27803: check] Error 2

The full strack trace for all threads is attached.  A simple "where" command in gdb shows the process was in thread 1 at the time of the fault.  The interpreter appears to be attempting to execute a shell command with "system".  There is no direct call to system in the tests for spconvert.m, so maybe it is from a call to "print_usage" when it is calling makeinfo to format the help text?


(file #50140)

My buildbot systems all have the following packages installed:

ii  libsuitesparse-dev:amd64    1:5.8.1+dfsg-2 amd64
ii  libsuitesparseconfig5:amd64 1:5.8.1+dfsg-2 amd64

That's interesting.  We should probably wait a few more days to see if this establishes itself as a true pattern.

http://buildbot.octave.org:8010/#/builders/32/builds/124/steps/7/logs/stdio

"clang-debian" again. Seems to be crashed just before "print_test_file_name" and after "print_pass_fail" this time.

clang-debian/113

  profiler/profshow.m ............................................ pass    4/4
  set/intersect.m ................................................ pass   28/28
fatal: caught signal Segmentation fault -- stopping myself...
/bin/bash: line 1: 941528 Segmentation fault      (core dumped) /bin/bash ../run-octave --no-init-file --silent --no-history -p /scratch/buildbot/workers/jwe-debian-x86_64-0/clang-debian/build/test/mex /scratch/buildbot/workers/jwe-debian-x86_64-0/clang-debian/build/../src/test/fntests.m /scratch/buildbot/workers/jwe-debian-x86_64-0/clang-debian/build/../src/test
make[3]: *** [Makefile:31841: check-local] Error 139
  set/ismember.m .................................................make[3]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-0/clang-debian/build'
make[2]: *** [Makefile:27961: check-am] Error 2
make[2]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-0/clang-debian/build'
make[1]: *** [Makefile:27663: check-recursive] Error 1
make[1]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-0/clang-debian/build'
make: *** [Makefile:27963: check] Error 2
program finished with exit code 2
elapsedTime=380.436578

On fedora there is a service that stores coredumps in its own directory
(and writes a trace to syslog).
I assume there is something similar available for debian, e.g.:
https://manpages.debian.org/stretch/systemd-coredump/coredumpctl.1.en.html

You'd still need to act fast if you want to get the trace
since the executable gets wiped out too...

Sincerely,

Dmitri.
--

And another one:
http://buildbot.octave.org:8010/#/builders/31/builds/96/steps/7/logs/stdio

"gcc-lto-debian" with "java/javaaddpath.m" last in the log.

jwe: Thanks for checking.
I guess that probably proves that the segfaults are not caused by bug #58790.

for i in $(find . -type f);do cat $i | grep -q Segmentation && cat $i | awk '/\./ && /Leaving/ && /directory/' && ls $i; done

@jwe: Not sure if this applies to all distributions:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-captun

IIUC, something similar to `echo 2 > /proc/sys/vm/overcommit_memory` has immediate effect. The change from comment #90 only has effect after the next re-boot.

For the Fortran libraries we use (or the libraries like BLAS and Lapack that have interfaces defined by their Fortran heritage) the important thing is not pointer size (those libraries don't use pointers like we think of in C++ or even modern Fortran) but the size of the integers that appear in their public interfaces for things like array dimensions and pivot vectors (which sort of serve the purpose of pointers, but are offsets into specific arrays).

On 64-bit systems, Octave now uses 64-bit integers for array dimensions and indexing by default.  But most Linux distributions supply BLAS and Lapack libraries (and other libraries that depend on them) compiled to use 32-bit integers.  That's a Fortran legacy thing, where INTEGER and REAL (i.e., single precision floating point numbers) typically occupy the same amount of storage.

As already noted, we also currently handle the case of Octave built with 64-bit dimensions and indexing and calling Fortran libraries that are compiled to use 32-bit integers for dimensions and indexing.

The current assumption is that all Fortran libraries that Octave uses will use the same convention, either 32-bit or 64-bit integers for dimensions and indexing.

I see no point in attempting to handle a mixture (some libraries using 32-bit and others using 64-bit integers) because these libraries also depend on each other.  It seems to me that mixing them arbitrarily is likely to lead to more confusion than just requiring that they all use the same convention.

Note also that (at least as I understood the state of things the last time I looked at this problem in detal) we don't really have a check to ensure that all the libraries that Octave depends on actually use the same convention.  We attempt to test the integer size of the BLAS library in the configure script.  That test requires executing a program, so guesses (or configure options) are used when cross compiling.  Whatever is determined for the BLAS library is assumed for all the rest.

comment #109:

> You can build Octave with 64bit indexing inside Octave but still use BLAS/LAPACK (and related) libraries that use 32bit indexing.

I was not able to make the configure report put out this:

  64-bit array dims and indexing:       yes
  64-bit BLAS array dims and indexing:  yes

When I used the reference blas and lapack implementations, I got instead:

  64-bit array dims and indexing:       yes
  64-bit BLAS array dims and indexing:  no

Perhaps that's intended, and if not, perhaps it's due to how the reference blas implementation is built on NixOS:

https://github.com/NixOS/nixpkgs/blob/8a5eb89b0f70999c08ce9ce6df89238671e186dc/pkgs/build-support/alternatives/blas/default.nix

I'm building up a benchmark table where every blas implementation is tested, against Octave 6.0.90 and 5.2.0, and I'll report the results (probably tomorrow) in NixOS' discourse thread. The benchmark I'm using is this: https://openbenchmarking.org/test/system/octave-benchmark .

After reading the discourse message:
There might be a misunderstanding. You can build Octave with 64bit indexing inside Octave but still use BLAS/LAPACK (and related) libraries that use 32bit indexing.
Octave itself is careful enough to not call those libraries with too large indices.

From the download page of the Octave Windows builds:
https://www.gnu.org/software/octave/download#ms-windows

> Unless your computer has more than ~32GB of memory and you need to solve linear algebra problems with arrays containing more than ~2 billion elements, this version will offer no advantage over the recommended Windows-64 version above.

Aaaah. I misunderstood your "you" referring to "me".

After reading the link from SuiteSparse: I don't know if Octave will work correctly with Intel MKL BLAS (which they recommend). I believe there have been reports about buggy behavior.

From an Octave point of view: Use either the reference BLAS (probably not the best performance) or OpenBLAS with optimizations for your processor (to have better performance).
Only use 64bit indexing in the Fortran libraries if you really need it.

comment #103:

> I never recommended using OpenBLAS. In fact, I asked whether you could reproduce with the reference BLAS libraries.
>
> If you ask for my recommendation, I'd advice against going down that rabbit hole of 64bit indices in BLAS/LAPACK. Instead stick to 32bit indices for BLAS/LAPACK and all related numeric libraries.

Yes you do: https://octave.org/doc/v5.2.0/External-Packages.html#External-Packages quoting:

> Basic Linear Algebra Subroutine library. Accelerated BLAS libraries such as OpenBLAS (https://www.openblas.net/) or ATLAS (http://math-atlas.sourceforge.net) are recommended for best performance. The reference implementation (http://www.netlib.org/blas) is slow, unmaintained, and suffers from certain bugs in corner case inputs.

The tests that previously failed, succeed when I use the same openblas for octave, qrupdate, suitesparse and arpack, though I'm worried about the degraded performence warning of suitesparse..

I never recommended using OpenBLAS. In fact, I asked whether you could reproduce with the reference BLAS libraries.

If you ask for my recommendation, I'd advice against going down that rabbit hole of 64bit indices in BLAS/LAPACK. Instead stick to 32bit indices for BLAS/LAPACK and all related numeric libraries.

Thanks for the advise Markus, I'll look into it!

If you are using OpenBLAS with 64bit indices, take care to also build all dependent numeric libraries with 64bit indices.
In the case of chol, a potentially incompatible library might be qrupdate.

Thanks for the prompt replies. Here's how my disto, NixOS, is building openblas:

https://github.com/doronbehar/nixpkgs/blob/pkg/octave/pkgs/development/libraries/science/math/openblas/default.nix

As can be seen in the `postInstall` attribute. the `liblapack.so` and `libblas.so` libraries are linked to the same shared object. You probably both understand this subject better then I am, but I think that if there's a incompatibility issue here, it's between openblas and it self, or between octave and it.

NixOS is building every package in a sandbox. Hence, no other blas / lapack libraries are potentially used if they were not declared in the inputs of the build "recipe". Please rest assured that my build wasn't influenced by a mixture of blas implementations, if that was your concern.

I've re-initiated the build of octave, so in a while I should be able to tell you Dimitri the output of `ldd src/.libs/octave-cli`.

P.S The Nix Expression for Octave which I'm testing is here:

https://github.com/doronbehar/nixpkgs/blob/pkg/octave/pkgs/development/interpreters/octave/default.nix

Could you re-run the tests for the build with 64bit OpenBLAS?
Does it segfault again at the same test?

I'm not sure if we are running continuous tests for that configuration. AFAICT, 64bit BLAS/LAPACK libraries are still quite rare.
It might be we are doing something wrong in Octave. But it might also be an error in OpenBLAS. It might be not very well tested with 64bit indices...

Is this segmentation fault reproducible?

Does the same happen if you used reference BLAS and LAPACK instead of OpenBLAS?

Thanks.
I admit that it is quite unlikely that this could be caused by overcommitting memory in that case.
But we'll be sure when the next crash occurs.

@jwe: To test whether these random crashes are caused by bug #58790, could you increase the swap size or (for some time) disable memory overcommit on the workers?

See e.g. here for how that could be done on Linux:
https://serverfault.com/a/142003

Another one for gcc-lto-debian:
http://buildbot.octave.org:8010/#/builders/31/builds/39/steps/7/logs/stdio

Last test in the log was sparse/gmres.m.

I wonder if this is bug #58790, i.e. Octave is killed by the kernel because available memory on the system was low.

AFAICT, the old builders will disappear from the waterfall view by themselves when they have been inactive for a while.
All stable builders disappeared for me from time to time when there was little action on the stable branch.
If you scroll very far down (and I mean veeeery far), I guess the epfl builders will eventually pop in.

I don't plan to delete history, but I don't really want them displayed by default.  The old entries take up horizontal space and add clutter in the waterfall display.  But now I see there is a "show old builders" option in the waterfall display settings.  The option is disabled by default, but toggling it removes the old builders from the display.  The reversed sense of this setting is a known bug.  I will see about changing the default in the master.cfg file, at least for the waterfall display page.

comment #79:

> [...] but they still appear on the buildbot web display.  I'm not sure how to remove them.

You have to remove them from the state.sqlite database with SQL statements (while the Buildbot Master is not running).  Or remove that file entirely to forget all history.  In any case make a backup before working on that file.

# sqlite3 state.sqlite
SQLite version 3.26.0 2018-12-01 12:34:55
Enter ".help" for usage hints.
sqlite> .tables
build_properties       change_users           patches
builder_masters        changes                scheduler_changes
builders               changesource_masters   scheduler_masters
builders_tags          changesources          schedulers
buildrequest_claims    configured_workers     sourcestamps
buildrequests          connected_workers      steps
builds                 logchunks              tags
buildset_properties    logs                   users
buildset_sourcestamps  masters                users_info
buildsets              migrate_version        workers
change_files           object_state
change_properties      objects


sqlite> SELECT * FROM workers;
1|ubuntu-1804-worker-01|{"admin": null, "host": null, "access_uri": null, "version": "2020.08.12"}|0|0


sqlite> SELECT * FROM builders;
1|octave-stable||097d6ab83a4824a0a88317812b687c5e919fc4db
2|octave-mxe-stable-w64-64||eaf8b4220ced7c8ec9bc8f95edc55307a214ee23
3|octave-mxe-stable-w64||5e82c51d06a1833b0065bb0281c1ed231b688796
4|octave-mxe-stable-w32||2dc32fc9b95a4f0839a75aff983d6a552e92625d
5|octave-stable-doxygen||1bf254e1654404c338635dadaec105637dc3e932

Yes, I rearranged the jobs on my buildbot systems in an attempt to balance the loads.

I also upgraded those systems to the latest Debian testing packages and lost some old compilers (Clang 4 & 5 and GCC 7) so those builds are no longer active but they still appear on the buildbot web display.  I'm not sure how to remove them.

The buildbots still seem to crash randomly during the BISTs:
http://buildbot.octave.org:8010/#/builders/19/builds/1756/steps/7/logs/stdio (gcc-7-debian)
http://buildbot.octave.org:8010/#/builders/12/builds/1898/steps/7/logs/stdio (clang-5.0-debian)

Both crashes occurred roughly at the same time. (But on different workers. Or have they been re-assigned recently?)

Reverting status to "Confirmed" after jwe's comment #74.
I guess we'll see with time if the build bots are still occasionally crashing while running the test suite.

The changes in comment #72 are very likely to have contributed to fixing bug #56952 though.

If I correctly understand jwe's fix, it changes the order of tasks done when closing Octave.
Does running the test suite with "make check" open and close Octave repeatedly?
I never verified that. But I assumed it would be comparable to running "__run_test_suite__" at the Octave prompt.

It looks like Rik's patch (shortcut processing of txt files) made it to crash with both gcc and clang.

Dmitri.

What appears to be happening is that the _gnuplot_init_.oct file (the one that defines the gnuplot graphics toolkit) is being closed before the toolkit is unloaded.  So when that happens, the pointer to the toolkit object is invalid.  The same problem exists with fltk.  It doesn't happen with qt because that is not dynamically loaded (with dlopen).  The same sequence of events happens when Octave is compiled with GCC, but for whatever reason, it the crash isn't happening.  So, it seems that we need to either prevent the toolkit's .oct file from being dlclosed (I thought the mlock in the init function would do that!) or we need to ensure that when it is dlclosed, that it is also removed/unregistered from the toolkit list.

I'm attaching a possible change to consider.  I'm not sure is the best solution, but it should at least avoid the crash.

(file #49678)

Build with CLANG:

unset DISPLAY
./run-octave --eval "figure (1,\"visible\",\"off\")"
octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
fatal: caught signal Segmentation fault -- stopping myself...
Segmentation fault (core dumped)

Build with GCC - no segfault:

unset DISPLAY
./run-octave --eval "figure (1,\"visible\",\"off\")"
octave: X11 DISPLAY environment variable not set
octave: disabling GUI features

I'm working on stable.  It looks like a delay didn't help though.

Are you working off stable or default?
In any case here is the output on 45a9dcee45db+ (stable)

rm -f src/octave-gui-6.0.1 && \
cd src && ln -s octave-gui octave-gui-6.0.1
rm -f doc/interpreter/plot-axesproperties.texi-t doc/interpreter/plot-axesproperties.texi && /bin/sh run-octave --norc --silent --no-history --path ../doc/interpreter --eval "genpropdoc ('axes');" > doc/interpreter/plot-axesproperties.texi-t && mv doc/interpreter/plot-axesproperties.texi-t doc/interpreter/plot-axesproperties.texi
fatal: caught signal Segmentation fault -- stopping myself...
/bin/sh: line 1: 2062846 Segmentation fault      (core dumped) /bin/sh run-octave --norc --silent --no-history --path ../doc/interpreter --eval "genpropdoc ('axes');" > doc/interpreter/plot-axesproperties.texi-t
make[2]: *** [Makefile:31022: doc/interpreter/plot-axesproperties.texi] Error 139
make[2]: Leaving directory '/home/dima/src/octave/clang_debug'
make[1]: *** [Makefile:27468: all-recursive] Error 1
make[1]: Leaving directory '/home/dima/src/octave/clang_debug'
make: *** [Makefile:11093: all] Error 2

Dmitri.
--

The build bots show that the segfault has now shifted from the "txt" images to the calling of genpropdoc.m.  I think it is significant that genpropdoc creates graphics figures and objects in order to query their default properties.  I went ahead and merged the change I made to the image generation files on stable to default since it seems to have improved things.

clang-devel-9.0.1-2.fc31.x86_64
clang-libs-9.0.1-2.fc31.x86_64
clang-tools-extra-9.0.1-2.fc31.x86_64
clang-9.0.1-2.fc31.x86_64

i still can't reproduce. switches are:

./configure CC=clang CXX=clang++
make V=1 -j12
C compiler:                    clang  -pthread  -Wall -W -Wshadow -Wformat -Wpointer-arith -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wcast-align -Wcast-qual -g -O2
C++ compiler:                  clang++  -pthread  -Wall -W -Wshadow -Woverloaded-virtual -Wold-style-cast -Wformat -Wpointer-arith -Wwrite-strings -Wcast-align -Wcast-qual -g -O2

;-(((

It still crashes on my workstation:

ake[2]: Entering directory '/home/dima/src/octave/clang_debug'
/bin/sh config.status oct-conf-post.h-tmp oct-conf-post.h
config.status: creating oct-conf-post.h-tmp
config.status: executing oct-conf-post.h commands
/bin/sh config.status liboctave/mk-version-h.sh-tmp liboctave/mk-version-h.sh
config.status: creating liboctave/mk-version-h.sh-tmp
config.status: executing liboctave/mk-version-h.sh commands
/bin/sh config.status libinterp/corefcn/mk-mxarray-h.sh-tmp libinterp/corefcn/mk-mxarray-h.sh
config.status: creating libinterp/corefcn/mk-mxarray-h.sh-tmp
config.status: executing libinterp/corefcn/mk-mxarray-h.sh commands
/bin/sh config.status build-aux/subst-config-vals.sh-tmp build-aux/subst-config-vals.sh
config.status: creating build-aux/subst-config-vals.sh-tmp
config.status: executing build-aux/subst-config-vals.sh commands
/bin/sh config.status liboctave/external/mk-f77-def.sh-tmp liboctave/external/mk-f77-def.sh
config.status: creating liboctave/external/mk-f77-def.sh-tmp
config.status: executing liboctave/external/mk-f77-def.sh commands
rm -f doc/interpreter/plot-axesproperties.texi-t doc/interpreter/plot-axesproperties.texi && /bin/sh run-octave --norc --silent --no-history --path ../doc/interpreter --eval "genpropdoc ('axes');" > doc/interpreter/plot-axesproperties.texi-t && mv doc/interpreter/plot-axesproperties.texi-t doc/interpreter/plot-axesproperties.texi
fatal: caught signal Segmentation fault -- stopping myself...
/bin/sh: line 1: 2052762 Segmentation fault      (core dumped) /bin/sh run-octave --norc --silent --no-history --path ../doc/interpreter --eval "genpropdoc ('axes');" > doc/interpreter/plot-axesproperties.texi-t
make[2]: *** [Makefile:31022: doc/interpreter/plot-axesproperties.texi] Error 139
make[2]: Leaving directory '/home/dima/src/octave/clang_debug'
make[1]: *** [Makefile:27468: all-recursive] Error 1
make[1]: Leaving directory '/home/dima/src/octave/clang_debug'
make: *** [Makefile:11093: all] Error 2

And the backtrace:

(gdb) thread apply all bt

Thread 2 (Thread 0x7f4753af6700 (LWP 2052933)):
#0  0x00007f487422e4dc in sigtimedwait () from /lib64/libc.so.6
#1  0x00007f48745ca95c in sigwait () from /lib64/libpthread.so.0
#2  0x00007f487bfdd3cf in signal_watcher (arg=0x7f487d7fbbe0 <octave::generic_sig_handler(int)>) at ../liboctave/wrappers/signal-wrappers.c:697
#3  0x00007f48745c02de in start_thread () from /lib64/libpthread.so.0
#4  0x00007f48742f1e83 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f487e074940 (LWP 2052762)):
#0  0x00007f487d67abce in octave::graphics_toolkit::close (this=0x15d3410) at ../libinterp/corefcn/graphics-toolkit.h:279
#1  0x00007f487d676f8a in octave::gtk_manager::unload_all_toolkits (this=0x10ae3c0) at ../libinterp/corefcn/gtk-manager.h:107
#2  0x00007f487d671cab in octave::interpreter::shutdown (this=0x10ad150) at ../libinterp/corefcn/interpreter.cc:902
#3  0x00007f487cb6d975 in octave::cli_application::execute (this=0x7ffcb8683990) at ../libinterp/octave.cc:381
#4  0x0000000000401839 in main (argc=15, argv=0x7ffcb8683cb8) at ../src/main-cli.cc:95

Dmitri.
--

Actually serial make crashes as well.

Dmitri.
--

What's interesting is that the failures all seem to be with files which are not related to actual plotting.  It seems that it is the generation of images in the ".txt" format which are failing, but looking at the m-files in doc/interpreter one sees

  if (strcmp (typ , "txt"))
    image_as_txt (d, nm);

and then

## generate something for the texinfo @image command to process
function image_as_txt (d, nm)
  fid = fopen (fullfile (d, [nm ".txt"]), "wt");
  fputs (fid, "\n");
  fputs (fid, "+---------------------------------+\n");
  fputs (fid, "| Image unavailable in text mode. |\n");
  fputs (fid, "+---------------------------------+\n");
  fclose (fid);
endfunction

So, no real plotting is being done and it may be the speed with which the graphics system is setup and torn down which is the problem.

Taking plotimages.m as representative, the function begins

function plotimages (d, nm, typ)
  set_graphics_toolkit ();
  set_print_size ();
  hide_output ();
  outfile = fullfile (d, [nm "." typ]);
  if (strcmp (typ, "png"))
    set (groot, "defaulttextfontname", "*");
  endif
  if (strcmp (typ, "eps"))
    d_typ = "-depsc2";
  else
    d_typ = ["-d", typ];
  endif

  if (strcmp (typ , "txt"))
    image_as_txt (d, nm);

and then ends with

  hide_output ();
endfunction

Shooting in the dark, what if we move the test for the "txt" format to the top of the file with this code

function plotimages (d, nm, typ)

  if (strcmp (typ , "txt"))
    image_as_txt (d, nm);
    return;
  endif

  set_graphics_toolkit ();
  set_print_size ();

The graphics system will never get invoked.

this one crashed yesterday and then again 4 hours ago with a core dumped:

http://buildbot.octave.org:8010/#/builders/12/builds/1847

The shell commands appear to be run using /bin/sh in non-interactive mode so startup files like .profile are not executed, at least as far as I can tell.

The following build on one of my buildbot systems failed:

http://buildbot.octave.org:8010/#/builders/22/builds/516

I've been repeatedly running the test suite using this build for the last 8 hours or so and it hasn't failed once.  I'm using

while true ; do
  if nice -n 19 xvfb-run -a -s 'screen 0 640x480x24' make V=1 check ; then
    echo "OK $?"
  else
    echo "NOT OK: $?"
    break
  fi
done

I said earlier that I would set the default ulimit for the buildbots so that we would generate core files, but I'm not sure of the best way to do that.  If I understand correctly, buildbot starts new shells to do each shell command step and I'd rather not have to add ulimit commands to each one.  So it seems that a change like this should be made on the worker systems instead but I don't know what startup file to add the ulimit command to on the build worker system.  Any ideas?

@John: The GUI Object (a Figure here) already stores a reference to the underlying graphics_object, see this excerpt from Object.h:

    // Store the graphics object directly so that it will exist when
    // we need it.  Previously, it was possible for the graphics
    // toolkit to get a handle to a figure, then have the interpreter
    // thread delete the corresponding object before the graphics
    // toolkit (GUI) thread had a chance to display it.  It should be OK
    // to store this object and use it in both threads (graphics_object
    // uses a std::shared_ptr) provided that we protect access with
    // mutex locks.
    graphics_object m_go;

After this addition, we should have changed all the logic and removed m_handle since m_go lets us access the object directly.

Anyway, is it me or what we see in the backtrace is a crash in Mesa, not in Octave?

I built with enabled address sanitizer flags and ran "make check". During the fixed tests in publish/publish.tst, I got a heap-buffer-overflow.
I was able to reproduce reproduce it twice (out of two tests) when running the complete test suite.

The attached log contains the backtrace and other info from the address sanitizer.

I'm not sure if this is related or something different. But it might be a graphics/threading issue afaict.

Wrt what Pantxo wrote on the maintainer's mailing list [1]: I came across this blog post [2]. It looks like the general idea could be applied cross-platform.
Would that be helpful?

[1]: https://lists.gnu.org/archive/html/octave-maintainers/2020-06/msg00067.html

[2]: https://devblogs.microsoft.com/oldnewthing/20130712-00/?p=3823

(file #49330)

Low prio and JFYI: I have spent a considerable amount of time to force a segfault in Fedora, either with gcc or with clang. E.g. i ran the test suite in a forever loop over a day. I also had no luck with "nice -19" that is the adjustment on the build bots.

Fedora Core 31
gcc version 9.3.1
clang version 9.0.1
Target: x86_64

I haven't found a "no-extras" buildbot that crashed with that error so far.

The only one I found with a crash was this one here:
http://buildbot.octave.org:8010/#/builders/7/builds/1615/steps/6/logs/stdio

Please help improve Octave by contributing tests for these files
(see the list in the file /scratch/buildbot/workers/jwe-debian-x86_64-2/no-extras-debian/build/test/fntests.log).
double free or corruption (out)
fatal: caught signal Aborted -- stopping myself...
/bin/bash: line 1: 3679392 Aborted                 /bin/bash ../run-octave --no-init-file --silent --no-history -p /scratch/buildbot/workers/jwe-debian-x86_64-2/no-extras-debian/build/test/mex /scratch/buildbot/workers/jwe-debian-x86_64-2/no-extras-debian/build/../src/test/fntests.m /scratch/buildbot/workers/jwe-debian-x86_64-2/no-extras-debian/build/../src/test
make[3]: *** [Makefile:31583: check-local] Error 134
make[3]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-2/no-extras-debian/build'
make[2]: *** [Makefile:27717: check-am] Error 2
make[2]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-2/no-extras-debian/build'
make[1]: *** [Makefile:27419: check-recursive] Error 1
make[1]: Leaving directory '/scratch/buildbot/workers/jwe-debian-x86_64-2/no-extras-debian/build'
make: *** [Makefile:27719: check] Error 2
program finished with exit code 2
elapsedTime=784.988686

But that one looks different and has probably been fixed in the meantime.

Also I haven't found a Fedora buildbot that crashed with that signature. (The ones mentioned in some of the comments here are due to bug #55225).

If I correctly understood yesterday, the Debian workers use Xvfb for plotting with the "qt" graphics toolkit while running the test suite.
Could that be related? Does Octave use a different code paths if the framebuffer is virtual?

The buildbots don't explicitly enable core dumps, so it depends on the prevailing system settings.  We could change that.

Thanks for the hint. I'll try to produce a core dump. But the crashes are very rare for me. I've seen maybe one every few weeks or months...

While some of the crashes on the build bots occured when plotting tests run, they generally seem to be in more random places.
I've probably missed a bunch of them (see bug #58393). But here a list of crashes with the last test occurring in the log in no particular order grouped by builders:

clang-4.0-debian:
  miscellaneous/copyfile.m
  plot/appearance/title.m
  plot/appearance/legend.m
  sparse/sprand.m
  linear-algebra/bandwidth.m
  optimization/optimget.m

clang-5.0-debian:
  special-matrix/hadamard.m
  miscellaneous/tar.m
  statistics/movmedian.m
  plot/appearance/annotation.m

gcc-7-debian:
  miscalleneous/isfile.m

gcc-7-lto-debian:
  java/usejava.m

At first glance, it doesn't look like these functions have anything special in common. I'm not sure if it has anything to do with graphics.

And here (clang-4.0-debian):
http://buildbot.octave.org:8010/#/builders/10/builds/1548/steps/7/logs/stdio

I tried to reproduce by running the test suite in gdb "make check RUN_OCTAVE_OPTIONS=-g" repeatedly. But it looks like the crash never occurs in the debugger (or I was just unlucky).
However, my gcc is version 9.3.0. But even with that version I've seen the occasional crash when running the test suite. Unfortunately, these happened when I didn't run in a debugger.
Is there any way to get any additional information when a program crashed in Ubuntu 20.04 (like the system log on Windows)?

Also looking on which buildbots failed, I'm starting to doubt whether the crashes depend on compiler or version.

And another one from a clang buildbot (clang-4.0-debian):
http://buildbot.octave.org:8010/#/builders/10/builds/1539/steps/7/logs/stdio

The other bug is probably bug #56952.

I was trying to point out the difference in comment #18 between a bug in the test suite and a bug building documentation.

This report is about the test suite, and I think we can make more progress here because it should be possible to build an image with debugging symbols and then run 'make check' until a segfault is captured.

Is it just me or does it seem like this bug has now been completely hijacked by a different bug?

The original bug was about segmentation faults running the test suite with clang builds. The recent comments seem to be about segmentation faults when building the doc images.

These are separate issues, please keep that in mind and do with this report whatever you think is best.

Buildbots job run with V=1, so the latest failure
http://buildbot.octave.org:8010/#/builders/11/builds/1521/steps/5/logs/stdio

But it is not 100% reproducible. Sometimes it etex, sometimes it
is GraphicsMagick, sometimes it is postscript.

Dmitri.
--
 

I t always looks to me that there is a race condition in parallel make (octave starts writing an output file to disk, but etex
already uses it to compile some document etc...). It never happens with serial make.

Dmitri.
--

When compiled with debug flags (-O0 -ggdb3) the failure rate is much lower and the one I got did not generate crash dump:

 GEN      doc/interpreter/splinefit6.png
  MAKEINFO ../doc/interpreter/octave.info
  TEXI2DVI doc/interpreter/octave.dvi
  MAKEINFO doc/interpreter/octave.html/.octave-html-stamp
/usr/bin/texi2dvi: etex exited with bad status, quitting.
make[2]: *** [Makefile:31088: doc/interpreter/octave.dvi] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/home/dima/src/octave/gcc_debug'
make[1]: *** [Makefile:27415: all-recursive] Error 1
make[1]: Leaving directory '/home/dima/src/octave/gcc_debug'
make: *** [Makefile:11050: all] Error 2

Dmitri.
--

This is going to be really hard to debug unless we can get a stacktrace.

For starters, does someone have a machine set up which mimics one of the failing buildbot configurations (such as gcc on Fedora or an older version of clang)?  Can you get semi-repeatable crashes on that machine?

The errors might be more prevalent than the current green markings in the buildbot's waterfall view might suggest.
If a test causes the buildbot to interrupt with a segmentation fault, the overall test is still marked as "green".
I wrote about this a while back on the mailing list:
https://octave.1599824.n4.nabble.com/buildbots-False-pass-results-for-segmentation-fault-in-test-td4695266.html

Maybe someone with more expertise in setting up buildbot could take a look?

That won't solve the actual issue. But it might make it easier to judge the impact and prevalence.

The previous problem seemed to be with the clang compiler for old versions (4 & 5).  It might be the same case with gcc on Fedora, the compiler there is old or bad in some manner.  What were version numbers for gcc and Fedora?

Thanks for testing again.

There seem to be occasional segfaults also for the gcc-7-lto-debian buildbot:
http://buildbot.octave.org:8010/#/builders/24/builds/1141/steps/6/logs/stdio

This might be related or it might be caused by something different.

@Mike: Thanks for your tests. The segfaults occur only intermittently on the buildbots. It might be that they only occur when the machine is under heavy load. E.g. when an mxe job is running on the same machine at the same time.

Could you please try to stress your machine while you are running the test suite and check if it still doesn't segfault?

That wasn't clear to me either.
Would it make sense to update the buildbots to use newer clang versions? Or is there a particular reason they run with clang 4 and clang 5?

8 is reasonably new, the fault is with clang 4 and 5.
Fedora has 9 and it is also fine.

Dmitri.
--

Maybe need to compile a version with debugging symbols and attempt "__run_test_suite__" manually under a debugger so that a backtrace can be obtained.

This could be a problem with clang, but more likely it is something generic that we are doing incorrectly that only occasionaly surfaces for combinations of compiler, libraries, and machine.