bugGNU Octave - Bugs: bug #47400, panic on Ctrl+c with multithreaded...

 
 

bug #47400: panic on Ctrl+c with multithreaded ATLAS or OpenBLAS

Submitter:  None
Submitted:  Sun 13 Mar 2016 06:57:30 PM UTC
   
 
Category:  Interpreter Severity:  4 - Important
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Fixed Assigned to:  lachlan
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 08 Mar 2018 09:02:02 PM UTC, comment #18: 

Multiple threads, multiple heads.  It's like Hercules trying to slay the Lernean Hydra.  As soon as you close one bug report about this, two more will be reported.

Rik <rik5>
Group administrator
Thu 08 Mar 2018 08:46:39 PM UTC, comment #17: 

Closing as fixed.  How many bug reports about Ctrl-C and threads do we have?

John W. Eaton <jwe>
Group administrator
Sun 10 Sep 2017 01:15:57 AM UTC, comment #16: 

This bug does not happen with precompiled 4.2.1 on Windows with OpenBLAS. It does happen with dev on OpenBLAS on Ubuntu.

Is the precompiled OpenBLAS single-threaded? That would explain this behavior.

Anonymous
Fri 30 Sep 2016 04:41:57 PM UTC, comment #15: 

I know you guys are working on this but it's in a pretty bad way on Linux xfce octave 4.2-rc2. If I interrupt code with CTRL-C (usually OpenBLAS) then it not only crashes Octave but logs me out of the session and kills my open programs.

E.g. I was comparing a Matlab, Octave calculation and checking Firefox... CTRL-C in Octave --> all programs closed and I was left at the login prompt.


Ceral Paquet <octavebugs>
Thu 14 Jul 2016 05:12:17 PM UTC, comment #14: 

I'm attaching a fairly short program that demonstrates the problem independent of Octave.  It also serves as a simple example of how Octave currently deals with signals.  My hope is that this program will make it easier to experiment with possible fixes.

If you interrupt the example program while dgetrf from ATLAS is executing with multiple threads, you will likely see problems.  I don't know how to make this work correctly.  Mixing threads and signal handling seems to be complicated.

Note that this issue is not just about ATLAS, but could happen with any external library that uses threads.  Maybe we need a better/different way to execute calls to library functions that could run for more than a fraction of a second?  Jumping out of the signal handler while executing random library code has always been risky.

(file #37873)

John W. Eaton <jwe>
Group administrator
Thu 14 Jul 2016 12:39:56 AM UTC, comment #13: 

I'm seeing the error below.

libinterp/corefcn/interpreter.cc:537:3: error: use of undeclared identifier 'alarm'
  alarm (5);    // Allow five seconds for clean-up, then "kill -9"
  ^
1 error generated.


Ben Abbott <bpabbott>
Group Member
Thu 14 Jul 2016 12:05:28 AM UTC, comment #12: 

Thanks for testing, Ben.

The part on toplev.cc has moved to interpreter.cc.  (See the attached patch.)  However, it doesn't affect the part that is supposed to stop the ^C errors.

I seem to recall that MacOS had issues other than just the BLAS.  Can you avoid the ^C error by running the BLAS single-threaded as Rik suggested in comment #7?

Also, this patch relies on the undocumented effect that Rik noticed: On Linux, calling popen seems to shut down unnecessary BLAS threads.  I wasn't expecting this patch to work on OSX or Windows.  This bug is currently marked as a Linux bug.

(file #37867)

Lachlan Andrew <lachlan>
Wed 13 Jul 2016 06:29:11 PM UTC, comment #11: 

file #37818 doesn't fully apply for me (toplev.cc). However, what did apply compiled.

A few ctrl-C's produced the result below.


^C==93856==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fff54b65000; bottom 0x70000029d000; size: 0x0fff548c8000 (17589309571072)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
>>
>> ASAN:DEADLYSIGNAL
=================================================================
==93856==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000003 (pc 0x000000000003 bp 0x1600cd93de44fb42 sp 0x70000029e8a0 T0)
==93856==Hint: pc points to the zero page.

>>     #0 0x2  (<unknown module>)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (<unknown module>)
==93856==ABORTING
panic: Abort trap: 6 -- stopping myself...
attempting to save variables to 'octave-workspace'...
save to 'octave-workspace' complete


Ben Abbott <bpabbott>
Group Member
Tue 12 Jul 2016 05:11:28 AM UTC, comment #10: 

With the attached patch, Rik's  tst_ctrlc  can successfully be interrupted well over 90% of the time on my system.

It is certainly not a real solution to the problem, but I think the real solution would required giving ATLAS and OpenBLAS new functionality to allow pthread_kill.

I have no idea how this will work with other OSes or BLASes.

I'm not sure if ensure_exit and force_exit should be put in sighandlers.cc.  That is the natural place, but I'd like the Kludge to be limited to as few files as possible, and having them in toplev.cc also allows them to be static.

Some of the things should probably be shielded by #if defined()s.

I'd really love feedback on this.  Thanks!

(file #37818)

Lachlan Andrew <lachlan>
Tue 12 Jul 2016 02:08:33 AM UTC, comment #9: 

If we just kill the openBLAS threads [using Rik's trick of popen("", "r")], it causes problems on exit.

OpenBLAS has a hook that runs when we call exit ():


#0  0x00007feaea124787 in sched_yield ()
    at ../sysdeps/unix/syscall-template.S:81
#1  0x00007feaec43178d in blas_thread_shutdown_ () from /usr/lib/libblas.so.3
#2  0x00007feaec4309cb in blas_shutdown () from /usr/lib/libblas.so.3
#3  0x00007feaf3b09fcf in _dl_fini () at dl-fini.c:253
#4  0x00007feaea09aa91 in __run_exit_handlers (status=0,
    listp=0x7feaea4075c8 <__exit_funcs>,
    run_list_atexit=run_list_atexit@entry=true) at exit.c:77
#5  0x00007feaea09ab15 in __GI_exit (status=<optimized out>) at exit.c:99
#6  0x00007feaf2da3e40 in octave_execute_interpreter ()
    at ../libinterp/octave.cc:1002
#7  0x0000000000401101 in main (argc=8, argv=0x7fffd43341d8)
    at ../src/main-cli.cc:92


It goes into a busy loop:


Single stepping until exit from function blas_thread_shutdown_,
which has no line number information.
sched_yield () at ../sysdeps/unix/syscall-template.S:81
81        in ../sysdeps/unix/syscall-template.S
(gdb)
sched_yield () at ../sysdeps/unix/syscall-template.S:82
82        in ../sysdeps/unix/syscall-template.S
(gdb)
0x00007feaec43178d in blas_thread_shutdown_ () from /usr/lib/libblas.so.3
(gdb)
Single stepping until exit from function blas_thread_shutdown_,
which has no line number information.
sched_yield () at ../sysdeps/unix/syscall-template.S:81
81        in ../sysdeps/unix/syscall-template.S
(gdb)
sched_yield () at ../sysdeps/unix/syscall-template.S:82
82        in ../sysdeps/unix/syscall-template.S


It looks like we will need to kill the threads through OpenBLAS itself. However, as indicated by the thread Mike pointed to, that is not currently possible.

If we can rely on the OS to do proper cleanup, then we could exit by sending ourselves a KILL or some such.  A slightly neater approach would be to set a timer for, say, 3 seconds before calling exit.  If the timer fires, then we KILL ourselves.  It's ugly, but less ugly than a crash or hang.

Lachlan Andrew <lachlan>
Mon 11 Jul 2016 01:51:17 PM UTC, comment #8: 

Thanks for the test script, Rik.

I notice that when I run this under gdb, I don't get a segfault.  Instead, the system stops with SIGINT received by the BLAS thread (often in dgemm_kernel_SANDYBRIDGE).

The very first time, I did get the segfault, and it seemed that the call stack had been overwritten, since the "real" functions were in fairly high locations, whereas several had locations like 000000100.

From the previous comments, I assume that the natural approach is:
1. Ensure that only the octave core thread receives ctrl-C (ensure SIGINT is masked off when the BLAS is started)
2. Make sure that the cleanup doesn't invalidate any of the data being operated on by the worker threads, until they can be stopped.

Is that right?

Lachlan Andrew <lachlan>
Fri 10 Jun 2016 03:53:33 AM UTC, comment #7: 

For easy testing, I have uploaded a script called tst_ctrlc.m.

Steps to produce a segfault:

1) Run octave
2) tst_ctrlc
3) Type Ctrl+C

Segfaults every time for me regardless of whether I am running Octave in the GUI or at the command line.

A workaround is to use


setenv OMP_NUM_THREADS 1


which stops multiple threads in the OpenBLAS library.



(file #37441)

Rik <rik5>
Group administrator
Mon 14 Mar 2016 01:11:17 AM UTC, comment #6: 

I poked around more with gdb. My ATLAS installation creates a thread per processor core and does what looks like a 40x40 submatrix multiplication (the 40 must have been what it decided was optimal at ATLAS build time). On hitting ctrl-c, only the Octave parser responds but the background ATLAS threads are all still running. I don't see yet why this should cause an Octave panic instead of the OS just harvesting the orphan threads quietly.

One workaround we can try is temporarily disabling ctrl-c or SIGINT if the matrix or exponent passed to mpower are sufficiently large. That may surprise the user a little if they want to halt, but will be less astonishing than losing the session to a panic. It also will be independent of the BLAS implementation. Thoughts?

Anonymous
Mon 14 Mar 2016 12:29:11 AM UTC, comment #5: 

Well the state of the interpreter at the time of the crash looks like the parser is resetting itself after the interrupt, but a secondary OpenBLAS worker thread is still running on behalf of the dgemm function.

I see in https://github.com/xianyi/OpenBLAS/issues/378 that there was some mention of using pthread_kill to propagate signals to worker threads within OpenBLAS, but it hasn't been fixed there yet.

Mike Miller <mtmiller>
Group Member
Mon 14 Mar 2016 12:02:13 AM UTC, comment #4: 

Is there really nothing we can do to avoid the crash?  I'd rather fix the real problem than document it if possible.

John W. Eaton <jwe>
Group administrator
Sun 13 Mar 2016 11:27:21 PM UTC, comment #3: 

Original crash was on multithreaded ATLAS, not OpenBlas, but the theme of crashing on multithreaded linear algebra libraries is consistent.

It's OK not to fix this crash in code because most users don't deal with large matrices or high exponents. But I would like to add a note about this behavior to a Docstring somewhere, to warn users to save everything before a large matrix power operation. What would be a good location? Data.cc in mpower?

Anonymous
Sun 13 Mar 2016 10:15:27 PM UTC, comment #2: 

Actually this bug has been previously reported and identified, see bug #43043, if you agree it is in fact caused by OpenBLAS for you.

Mike Miller <mtmiller>
Group Member
Sun 13 Mar 2016 09:48:50 PM UTC, comment #1: 

Thanks for the bug report. I can confirm the crash with multithreaded OpenBLAS. Octave is able to interrupt and resume correctly when using netlib BLAS and LAPACK or when restricting the number of threads to 1. Can you confirm? If so I'm not sure there's anything that Octave is doing incorrectly or can do to handle this scenario better.

Mike Miller <mtmiller>
Group Member
Sun 13 Mar 2016 06:57:30 PM UTC, original submission:  

Ctrl-C during execution in most cases halts properly and returns to the command prompt. This is a rare but repeatable case where ctrl-c causes a panic and crash.

Steps to reproduce:

1. Run the following at the prompt or as a script.

clear all
arr = rand(6561,6561); disp('calculating...')
result = (arr^28)(1,1)


2. Type ctrl-c. If ctrl-c was invoked before the "calculating" message, it returns to command prompt. If invoked after, it causes a panic and crash.

This kind of code arises frequently in transfer matrix methods in statistical mechanics. Looks like recovery from an interrupt in Lapack or BLAS execution is not possible?

hg id 10c32f940636

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #37873:  sigtest.c added by jwe (5KiB - text/x-csrc)
file #37441:  tst_ctrlc.m added by rik5 (76B - d2l/unknowntype)

 

Depends on the following items

Digest:
   bug dependencies.

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by octavebugs (Posted a comment)
  • -email is unavailable- added by bpabbott (Posted a comment)
  • -email is unavailable- added by lachlan (Updated the item)
  • -email is unavailable- added by rik5 (Updated the item)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 15 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-03-08 jwe StatusPatch Submitted Fixed
        Open/ClosedOpen Closed
    2016-07-14 jwe Attached File- Added sigtest.c, #37873
    2016-07-14 lachlan Attached File- Added bug_47400_blas_thread_crash.v2.cset, #37867
    2016-07-12 lachlan Attached File- Added bug_47400_blas_thread_crash.cset, #37818
        StatusNeed Info Patch Submitted
        Assigned toNone lachlan
    2016-06-28 lachlan Severity3 - Normal 4 - Important
        Priority3 - Low 5 - Normal
    2016-06-10 rik5 Attached File- Added tst_ctrlc.m, #37441
    2016-06-08 mtmiller Dependencies- bugs #37672 is dependent
    2016-05-17 mtmiller SummaryRepeatable panic on ctrl-c panic on Ctrl+c with multithreaded ATLAS or OpenBLAS
    2016-03-13 mtmiller Dependencies- Depends on bugs #43043
    2016-03-13 mtmiller Priority5 - Normal 3 - Low
        StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code