bugGNU Octave - Bugs: bug #61742, Crash on exit from threading

 
 

bug #61742: Crash on exit from threading

Submitter:  None
Submitted:  Wed 29 Dec 2021 05:40:49 PM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Build Failure
Status:  Confirmed Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 31 Dec 2021 12:41:26 PM UTC, comment #14: 

Thanks.

Closing as upstream issue.

Markus Mützel <mmuetzel>
Group administrator
Fri 31 Dec 2021 12:39:51 PM UTC, comment #13: 

No, nothing needs to be fixed in Octave. Octave was only displaying the results of the library's segfault. It can be closed. Hopefully anyone with the same problem can find it and the solution.

Arun Giridhar <arungiridhar>
Group Member
Fri 31 Dec 2021 11:46:40 AM UTC, comment #12: 

IIUC, this is an upstream/packaging issue?
Is there anything that needs to be fixed in Octave to prevent this?

Markus Mützel <mmuetzel>
Group administrator
Thu 30 Dec 2021 04:42:35 PM UTC, comment #11: 

I solved the problem. It turned out the distro was automatically adding "USE_TLS=1" to the OpenBlas settings, which stands for "thread-local storage" as opposed to a central memory buffer. That setting alone was responsible for the segfaults on exit. Removing that setting solved the problem.

Now there is no crash on exit:

octave:1> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.19 NO_AFFINITY USE_OPENMP ZEN MAX_THREADS=12)
octave:2> a = randn(5000); tic; inv(a)*a; toc
Elapsed time is 2.29698 seconds.
octave:3>
octave:3> exit


This probably affected anybody using Arch or Manjaro or other Arch-derived distro. The solution is to remove USE_TLS.

Arun Giridhar <arungiridhar>
Group Member
Thu 30 Dec 2021 02:30:40 PM UTC, comment #10: 

Bug report submitted to OpenBlas here: https://github.com/xianyi/OpenBLAS/issues/3503

Arun Giridhar <arungiridhar>
Group Member
Thu 30 Dec 2021 01:32:21 PM UTC, comment #9: 

My library is the same as yours except for the number of cores/threads. I am slowly narrowing in on the cause of the crash, and I can now trigger a repeatable crash within OpenBlas 0.3.19 by setting CPP_THREAD_SAFETY_TEST=1 while configuring and building it. It passes the DGEMV threading tests and then crashes on exiting dgemv_tester with a segfault, causing a library build failure. This causes a crash when building 0.3.19 but NOT the git source of OpenBlas, so something is weird with the library. The good news is that I can trigger this crash independently of Octave.

I'll put something together and submit a bug report to OpenBlas. Will report here what I find out.

Due to the nature of my distro (Manjaro), the distro-supplied OpenBlas is just a PKGBUILD script to download and build the library from source. On the plus side, this tunes the library to the individual user. On the minus side, any errors will be weird and non-universal.

Arun Giridhar <arungiridhar>
Group Member
Thu 30 Dec 2021 02:29:37 AM UTC, comment #8: 


octave:3> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.19 DYNAMIC_ARCH NO_AFFINITY USE_OPENMP Zen MAX_THREADS=16)
octave:4> version
ans = 8.0.0
octave:5> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.19 DYNAMIC_ARCH NO_AFFINITY USE_OPENMP Zen MAX_THREADS=16)
octave:6> a=randn(5000);
octave:7> tic; inv(a)*a; toc
Elapsed time is 1.50311 seconds.
octave:8>

Ryzen 9 3950X 16-Core Processor

No crash.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 30 Dec 2021 01:44:43 AM UTC, comment #7: 

Thanks for that link, but it didn't work in my case. I don't have avx512 either, only avx and avx2 (Zen 2). I spent this evening changing various flags for building OpenBlas. In all the cases where the library could be built, it passed all the tests including the extra tests for thread safety. On running the OpenBlas benchmarks, they are able to handle dgemm for 200x200 with no problem, but somehow when called from Octave for any Matrix above 64x64 it causes a segfault on exit.

This is not a show stopper but it's a nuisance because it is repeatable but not universal. It means if a script is calling Octave and then examining the exit value it will conclude failure because of the crash on exit. But it is not clear why this happens at all.

The only hope is that enough others might have it that there's a fix, otherwise I'll simply have to downgrade OpenBlas.

Arun Giridhar <arungiridhar>
Group Member
Wed 29 Dec 2021 09:04:12 PM UTC, comment #6: 

FYI:

https://github.com/xianyi/OpenBLAS/issues/3494

(I do not have avx512 hardware)

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 29 Dec 2021 07:39:56 PM UTC, comment #5: 

Rebuilding OpenBlas with all-defaults did not help me. Julia is not affected by it, only Octave, though both use the same OpenBlas library. Maybe Julia is not reporting things that went wrong and Octave is.

For my end, I don't see how Octave can do anything differently. I ought have to close as an upstream bug possibly.

Arun Giridhar <arungiridhar>
Group Member
Wed 29 Dec 2021 07:05:32 PM UTC, comment #4: 

I am having this problem on Manjaro. My OpenBlas was updated last night and today I am seeing random crashes on exit for matrix operations on matrix orders more than 64. I think for order 64 or lower it only uses a single thread so as soon as it uses multiple threads it causes a seg fault somehow.

Will see if rebuilding OpenBlas makes a difference. But no idea why it would affect things that are not even linear algebra related.

Arun Giridhar <arungiridhar>
Group Member
Wed 29 Dec 2021 06:54:23 PM UTC, comment #3: 

I suspect your openblas library is faulty. Not sure why does it affect the plot command though....

glxinfo is a part of mesa-demos package.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 29 Dec 2021 06:40:37 PM UTC, comment #2: 

No command "glxinfo" but it uses the nouveau driver with an nVidia card. No problems with it though. The OS is Arch Linux.

Anonymous
Wed 29 Dec 2021 06:27:22 PM UTC, comment #1: 

I cannot reproduce on neither Fedora 35 nor Ubuntu 20.10
with hg id 83f9f8bda883 @.

What is your OS?
What is the graphics card/driver? (glxinfo -B)

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 29 Dec 2021 05:40:49 PM UTC, original submission:  

Documentation cannot be built with Octave 8 hg id 0ef88c485786. The problem is something to do with threading.

This exits normally:

octave:1> exit


This exits normally:

octave:1> figure; exit


This crashes on exit:

octave:1> plot (1:10, 1:10, 'b.-'); exit
fatal: caught signal Segmentation fault -- stopping myself...
Segmentation fault (core dumped)


The above crash prevents figures from being made and therefore the documentation cannot be built.

Saying "export OMP_NUM_THREADS=1" at the bash prompt stops it from crashing and allows documentation to be built.

Once documentation is built, "make check" passes even on a fresh terminal where OMP_NUM_THREADS is not even defined, so it is not clear what the problem is.

This also affects libraries that use threading like BLAS:


octave:1> A = rand(100,100) - 0.5; A = (A ^ 3) .^ 3; [v,d] = eig(A);
octave:2> exit
fatal: caught signal Segmentation fault -- stopping myself...
Segmentation fault (core dumped)


After exporting OMP_NUM_THREADS=1, no crash on exit:

octave:1> A = rand(100,100) - 0.5; A = (A ^ 3) .^ 3; [v, d] = eig(A);
octave:2> exit


It only seems to crash on exit not during the computation.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by arungiridhar (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-12-31 mmuetzel StatusNeed Info Confirmed
        Open/ClosedOpen Closed
    2021-12-31 mmuetzel StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-bb6a.
    Corresponding source code