bugGNU Octave - Bugs: bug #47400, panic on Ctrl+c with multithreaded...

 
 

bug #47400: panic on Ctrl+c with multithreaded ATLAS or OpenBLAS

Submitted by:  None
Submitted on:  Sun 13 Mar 2016 06:57:30 PM UTC  
 
Category: InterpreterSeverity: 4 - Important
Priority: 5 - NormalItem Group: Segfault, Bus Error, etc.
Status: Patch SubmittedAssigned to: Lachlan Andrew <lachlan>
Originator Name: Originator Email: -unavailable-
Open/Closed: OpenRelease: dev
Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 10 Sep 2017 01:15:57 AM UTC, comment #16:

This bug does not happen with precompiled 4.2.1 on Windows with OpenBLAS. It does happen with dev on OpenBLAS on Ubuntu.

Is the precompiled OpenBLAS single-threaded? That would explain this behavior.

Anonymous
Fri 30 Sep 2016 04:41:57 PM UTC, comment #15:

I know you guys are working on this but it's in a pretty bad way on Linux xfce octave 4.2-rc2. If I interrupt code with CTRL-C (usually OpenBLAS) then it not only crashes Octave but logs me out of the session and kills my open programs.

E.g. I was comparing a Matlab, Octave calculation and checking Firefox... CTRL-C in Octave --> all programs closed and I was left at the login prompt.

Ceral Paquet <octavebugs>
Thu 14 Jul 2016 05:12:17 PM UTC, comment #14:

I'm attaching a fairly short program that demonstrates the problem independent of Octave. It also serves as a simple example of how Octave currently deals with signals. My hope is that this program will make it easier to experiment with possible fixes.

If you interrupt the example program while dgetrf from ATLAS is executing with multiple threads, you will likely see problems. I don't know how to make this work correctly. Mixing threads and signal handling seems to be complicated.

Note that this issue is not just about ATLAS, but could happen with any external library that uses threads. Maybe we need a better/different way to execute calls to library functions that could run for more than a fraction of a second? Jumping out of the signal handler while executing random library code has always been risky.

(file #37873)

John W. Eaton <jwe>
Project Administrator
Thu 14 Jul 2016 12:39:56 AM UTC, comment #13:

I'm seeing the error below.

Ben Abbott <bpabbott>
Project Member
Thu 14 Jul 2016 12:05:28 AM UTC, comment #12:

Thanks for testing, Ben.

The part on toplev.cc has moved to interpreter.cc. (See the attached patch.) However, it doesn't affect the part that is supposed to stop the ^C errors.

I seem to recall that MacOS had issues other than just the BLAS. Can you avoid the ^C error by running the BLAS single-threaded as Rik suggested in comment #7?

Also, this patch relies on the undocumented effect that Rik noticed: On Linux, calling popen seems to shut down unnecessary BLAS threads. I wasn't expecting this patch to work on OSX or Windows. This bug is currently marked as a Linux bug.

(file #37867)

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Wed 13 Jul 2016 06:29:11 PM UTC, comment #11:

file #37818 doesn't fully apply for me (toplev.cc). However, what did apply compiled.

A few ctrl-C's produced the result below.

Ben Abbott <bpabbott>
Project Member
Tue 12 Jul 2016 05:11:28 AM UTC, comment #10:

With the attached patch, Rik's tst_ctrlc can successfully be interrupted well over 90% of the time on my system.

It is certainly not a real solution to the problem, but I think the real solution would required giving ATLAS and OpenBLAS new functionality to allow pthread_kill.

I have no idea how this will work with other OSes or BLASes.

I'm not sure if ensure_exit and force_exit should be put in sighandlers.cc. That is the natural place, but I'd like the Kludge to be limited to as few files as possible, and having them in toplev.cc also allows them to be static.

Some of the things should probably be shielded by #if defined()s.

I'd really love feedback on this. Thanks!

(file #37818)

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Tue 12 Jul 2016 02:08:33 AM UTC, comment #9:

If we just kill the openBLAS threads [using Rik's trick of popen("", "r")], it causes problems on exit.

OpenBLAS has a hook that runs when we call exit ():

It goes into a busy loop:

It looks like we will need to kill the threads through OpenBLAS itself. However, as indicated by the thread Mike pointed to, that is not currently possible.

If we can rely on the OS to do proper cleanup, then we could exit by sending ourselves a KILL or some such. A slightly neater approach would be to set a timer for, say, 3 seconds before calling exit. If the timer fires, then we KILL ourselves. It's ugly, but less ugly than a crash or hang.

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Mon 11 Jul 2016 01:51:17 PM UTC, comment #8:

Thanks for the test script, Rik.

I notice that when I run this under gdb, I don't get a segfault. Instead, the system stops with SIGINT received by the BLAS thread (often in dgemm_kernel_SANDYBRIDGE).

The very first time, I did get the segfault, and it seemed that the call stack had been overwritten, since the "real" functions were in fairly high locations, whereas several had locations like 000000100.

From the previous comments, I assume that the natural approach is:
1. Ensure that only the octave core thread receives ctrl-C (ensure SIGINT is masked off when the BLAS is started)
2. Make sure that the cleanup doesn't invalidate any of the data being operated on by the worker threads, until they can be stopped.

Is that right?

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Fri 10 Jun 2016 03:53:33 AM UTC, comment #7:

For easy testing, I have uploaded a script called tst_ctrlc.m.

Steps to produce a segfault:

1) Run octave
2) tst_ctrlc
3) Type Ctrl+C

Segfaults every time for me regardless of whether I am running Octave in the GUI or at the command line.

A workaround is to use

which stops multiple threads in the OpenBLAS library.

(file #37441)

Rik <rik5>
Project Administrator
Mon 14 Mar 2016 01:11:17 AM UTC, comment #6:

I poked around more with gdb. My ATLAS installation creates a thread per processor core and does what looks like a 40x40 submatrix multiplication (the 40 must have been what it decided was optimal at ATLAS build time). On hitting ctrl-c, only the Octave parser responds but the background ATLAS threads are all still running. I don't see yet why this should cause an Octave panic instead of the OS just harvesting the orphan threads quietly.

One workaround we can try is temporarily disabling ctrl-c or SIGINT if the matrix or exponent passed to mpower are sufficiently large. That may surprise the user a little if they want to halt, but will be less astonishing than losing the session to a panic. It also will be independent of the BLAS implementation. Thoughts?

Anonymous
Mon 14 Mar 2016 12:29:11 AM UTC, comment #5:

Well the state of the interpreter at the time of the crash looks like the parser is resetting itself after the interrupt, but a secondary OpenBLAS worker thread is still running on behalf of the dgemm function.

I see in https://github.com/xianyi/OpenBLAS/issues/378 that there was some mention of using pthread_kill to propagate signals to worker threads within OpenBLAS, but it hasn't been fixed there yet.

Mike Miller <mtmiller>
Project Administrator
Mon 14 Mar 2016 12:02:13 AM UTC, comment #4:

Is there really nothing we can do to avoid the crash? I'd rather fix the real problem than document it if possible.

John W. Eaton <jwe>
Project Administrator
Sun 13 Mar 2016 11:27:21 PM UTC, comment #3:

Original crash was on multithreaded ATLAS, not OpenBlas, but the theme of crashing on multithreaded linear algebra libraries is consistent.

It's OK not to fix this crash in code because most users don't deal with large matrices or high exponents. But I would like to add a note about this behavior to a Docstring somewhere, to warn users to save everything before a large matrix power operation. What would be a good location? Data.cc in mpower?

Anonymous
Sun 13 Mar 2016 10:15:27 PM UTC, comment #2:

Actually this bug has been previously reported and identified, see bug #43043, if you agree it is in fact caused by OpenBLAS for you.

Mike Miller <mtmiller>
Project Administrator
Sun 13 Mar 2016 09:48:50 PM UTC, comment #1:

Thanks for the bug report. I can confirm the crash with multithreaded OpenBLAS. Octave is able to interrupt and resume correctly when using netlib BLAS and LAPACK or when restricting the number of threads to 1. Can you confirm? If so I'm not sure there's anything that Octave is doing incorrectly or can do to handle this scenario better.

Mike Miller <mtmiller>
Project Administrator
Sun 13 Mar 2016 06:57:30 PM UTC, original submission:

Ctrl-C during execution in most cases halts properly and returns to the command prompt. This is a rare but repeatable case where ctrl-c causes a panic and crash.

Steps to reproduce:

1. Run the following at the prompt or as a script.

2. Type ctrl-c. If ctrl-c was invoked before the "calculating" message, it returns to command prompt. If invoked after, it causes a panic and crash.

This kind of code arises frequently in transfer matrix methods in statistical mechanics. Looks like recovery from an interrupt in Lapack or BLAS execution is not possible?

hg id 10c32f940636

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #37873:  sigtest.c added by jwe (5KiB - text/x-csrc)
file #37441:  tst_ctrlc.m added by rik5 (76B - d2l/unknowntype)

 

Depends on the following items

Digest:
   bug dependencies.

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -unavailable- added by octavebugs (Posted a comment)
  • -unavailable- added by bpabbott (Posted a comment)
  • -unavailable- added by lachlan (Updated the item)
  • -unavailable- added by rik5 (Updated the item)
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 13 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 14 Jul 2016 05:12:17 PM UTCjweAttached File-=>Added sigtest.c, #37873
    Thu 14 Jul 2016 12:05:28 AM UTClachlanAttached File-=>Added bug_47400_blas_thread_crash.v2.cset, #37867
    Tue 12 Jul 2016 05:11:28 AM UTClachlanAttached File-=>Added bug_47400_blas_thread_crash.cset, #37818
      StatusNeed Info=>Patch Submitted
      Assigned toNone=>lachlan
    Tue 28 Jun 2016 01:07:31 AM UTClachlanSeverity3 - Normal=>4 - Important
      Priority3 - Low=>5 - Normal
    Fri 10 Jun 2016 03:53:33 AM UTCrik5Attached File-=>Added tst_ctrlc.m, #37441
    Wed 08 Jun 2016 05:03:32 PM UTCmtmillerDependencies-=>bugs #37672 is dependent
    Tue 17 May 2016 03:38:43 PM UTCmtmillerSummaryRepeatable panic on ctrl-c=>panic on Ctrl+c with multithreaded ATLAS or OpenBLAS
    Sun 13 Mar 2016 10:15:27 PM UTCmtmillerDependencies-=>Depends on bugs #43043
    Sun 13 Mar 2016 09:48:50 PM UTCmtmillerPriority5 - Normal=>3 - Low
      StatusNone=>Need Info

    Back to the top


    Powered by Savane 3.1-cleanup1