bugGNU Octave - Bugs: bug #52757, cntrl-C crash

 
 

bug #52757: cntrl-C crash

Submitter:  Michael Godfrey <godfrey>
Submitted:  Thu 28 Dec 2017 10:16:56 PM UTC
   
 
Category:  Interpreter Severity:  5 - Blocker
Priority:  7 - High Item Group:  Regression
Status:  Fixed Assigned to:  None
Originator Name:  godfrey Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 09 Jan 2018 12:31:15 PM UTC, comment #30: 

John,
Works for me. Thanks.

Michael

Michael Godfrey <godfrey>
Group Member
Tue 09 Jan 2018 02:51:04 AM UTC, comment #29: 

Oops, I think I fixed it with this change:

  http://hg.savannah.gnu.org/hgweb/octave/rev/f5bcbd321ba1

I don't know why I thought it would be a good idea to only loop around readline in the event of a Control-C.

John W. Eaton <jwe>
Group administrator
Tue 09 Jan 2018 12:44:51 AM UTC, comment #28: 

I did not make it explicit, but this problem causes
the user to have to use cntrl-d which kills octave.
So, it is quite serious.

Michael Godfrey <godfrey>
Group Member
Mon 08 Jan 2018 12:25:08 AM UTC, comment #27: 

Sorry to say there is a problem:

The following example shows that cntrl-c does not
work as before.

octave:1> input_hangc
try to type cntrl-c:
try to type cntrl-c:
try to type cntrl-c:
try to type cntrl-c:error: input: reading user-input failed!
error: called from
    input_hangc at line 1 column 4
octave:1>
==================
The first 3 lines were from using cntrl-c. The last line
used cntrl-d.

The program input_hangc.m is:

pc = input ("try to type cntrl-c:");
printf("OK\n");

Michael Godfrey <godfrey>
Group Member
Fri 05 Jan 2018 06:04:46 PM UTC, comment #26: 

Seems to be fixed.  Closing report.

Rik <rik5>
Group administrator
Fri 05 Jan 2018 12:34:30 PM UTC, comment #25: 

Works for me, too. Fedora 27.x86_64

Michael Godfrey <godfrey>
Group Member
Fri 05 Jan 2018 04:11:47 AM UTC, comment #24: 

This seems to fix crash on my computer as well.

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Thu 04 Jan 2018 07:30:30 PM UTC, comment #23: 

I checked in the following changeset:

  http://hg.savannah.gnu.org/hgweb/octave/rev/c5c11b07598a

This change fixes the crash for me.

It is also a major change in the way signal handling works.  See the comments in the changeset for some details.  I wouldn't be surprised if I broke something other than the ability to interrupt calls to library functions.  But that was intentional and at this point unavoidable as I don't know how to solve that problem easily.

I have not tested this on Windows, but will as soon as buildbot creates an installer for me.

John W. Eaton <jwe>
Group administrator
Tue 02 Jan 2018 05:23:45 PM UTC, comment #22: 

Sorry for the delay.  As I was looking at this issue and trying to write a summary of interrupt handling in Octave I ran into more trouble and confusion.  Briefly, the problem is that any library that we use may start threads.  Because we don't block signals, any thread may end up handling SIGINT.  If this happens in a section of code that is set to have interrupts processed immediately (by calling longjmp to return to the point of the library call) bad things will happen because the longjmp is happening in a different thread from where the corresponding setjmp was called.  We are almost sure to screw up the stack.

I'm attaching an example program that uses a slightly different approach to handle SIGINT when multiple threads may be active.  We might be able to use this approach in Octave.  It fixes part of the problem (signals vs. multiple threads) but it raises other issues (responsiveness to signals when calling library functions).  The basic idea is to block interrupts in all but one thread.  That thread simply waits for SIGINT to arrive and when it does, sets a variable to indicate that an interrupt has arrived.  Then, as in Octave, that variable is checked periodically to see whether an interrupt has occurred, and if so, some action is taken to interrupt the main process.

Note that with this approach, any library code called from the main thread will not respond to SIGINT because we have it blocked.  And if a library function unblocks SIGINT, we may have trouble because now we are waiting for SIGINT and the library also expects to handle it.  This issue extends to readline, so we will need to do something different to allow interrupts to be handled when readline is waiting for input.  Also note that we can't just throw an exception from a readline event hook function because readline will not clean up after itself if an exception is thrown.  (As a side note, we should check to ensure that any code that may be executed in the readline event hook function also will not throw an exception.)

I'm working on a patch that implements this approach to signal handling for Octave.  I'll comment here again when it's ready to try.


(file #42789)

John W. Eaton <jwe>
Group administrator
Sat 30 Dec 2017 01:53:56 AM UTC, comment #21: 

If you want to pass the signal to the process that you are debugging with gdb, try the gdb command


handle SIGINT pass nostop


Most of what I tried to do to debug this didn't help, but I do think I've made some progress in understanding what is happening.  The basic issue is that threads and signals mix very well.  The signal may be delivered to any thread.  Depending on which one, our current signal handler can cause some trouble.  I'm out of time for today but will try to give some more details and possible options tomorrow.

John W. Eaton <jwe>
Group administrator
Sat 30 Dec 2017 12:28:46 AM UTC, comment #20: 

I noticed one thing: After ^C and waiting for Octave to
exit there is still an Octave process running. In my case:
godfrey  12839  7.2  0.4 1238872 73372 ?       Sl   00:22   0:06 /usr/local/libexec/octave/4.3.0+/exec/x86_64-pc-linux-gnu/octave-gui --silent --no-gui

I noticed this because in one case the process was using 99%CPU.

Michael Godfrey <godfrey>
Group Member
Fri 29 Dec 2017 11:18:53 PM UTC, comment #19: 

I tried to run it under gdb, but I am not making much progress.
Ctrl-C seems to just interrrupts octave (abd return to gdb):


octave:1> version
ans = 4.3.0+
octave:2>
Thread 2 "lt-octave-gui" received signal SIGINT, Interrupt.
0x00007fffebcb7c4b in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
(gdb) fg
Continuing.



Dmitri.


Dmitri A. Sergatskov <dasergatskov>
Fri 29 Dec 2017 01:53:07 PM UTC, comment #18: 

John,

readline is involved, but keep in mind that:
./run-octave --no-gui-libs

fixes the problem. But, no 'qt'.

Michael

Michael Godfrey <godfrey>
Group Member
Fri 29 Dec 2017 01:17:26 PM UTC, comment #17: 

readline is involved, but I doubt it is the real issue.  Without readline, communication between the GUI and the interpreter will not work properly.

John W. Eaton <jwe>
Group administrator
Fri 29 Dec 2017 03:49:24 AM UTC, comment #16: 

May be it is a libreadline problem.
It is definitely looks like bug

https://savannah.gnu.org/bugs/?51903

except with ^C rather than with ^Z.
I recompile octave with --disable-readline and I do not see the problem. Fedora has libreadline6.3 installed as well, but I cannot
figure out how to convince octave to link to that library.
LD_PRELOAD trick does not work, I suspect 7 and 6.3 are not
ABI compatible.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 29 Dec 2017 02:23:16 AM UTC, comment #15: 

Dmitri,

That works for me too. Good, but this also
disables qt graphics_toolkit. This fact is
why I did not see this problem in the past.
I only recently switched to qt from fltk.

So, the failure is due to loading the qt libs.

Michael

Michael Godfrey <godfrey>
Group Member
Fri 29 Dec 2017 02:06:01 AM UTC, comment #14: 

Michael,

If you run
./run-octave --no-gui-libs
does it still crash?

Mine does not (on affected computer).

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 29 Dec 2017 01:57:30 AM UTC, comment #13: 

John,

Yes about 4.2.2 crash on exit. I am sure that is fixed
in current dev. My only problem on dev is the ^C crash.

Michael Godfrey <godfrey>
Group Member
Fri 29 Dec 2017 01:09:06 AM UTC, comment #12: 

I think I can reproduce it on another computer.
If I hit Ctrl-C it echoes it multiple times and the core dumps:


./run-octave --no-gui -f -q
octave:1> version
ans = 4.3.0+
octave:2> ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
octave:2> *** stack smashing detected ***: <unknown> terminated
Aborted (core dumped)


I see the following in journalctl:


Dec 28 19:00:59 amd systemd-coredump[2472]: Process 2325 (lt-octave-gui) of user 1001 dumped core.

                                            Stack trace of thread 2458:
                                            #0  0x00007f061fee766b raise (libc.so.6)
                                            #1  0x00007f061fee945b abort (libc.so.6)
                                            #2  0x00007f061ff31a57 __libc_message (libc.so.6)
                                            #3  0x00007f061ffdaa11 __GI___fortify_fail_abort (libc.so.6)
                                            #4  0x00007f061ffda9d2 __stack_chk_fail (libc.so.6)
                                            #5  0x00007f0622116a54 rl_getc (libreadline.so.7)
                                            #6  0x00007f0622116e8b rl_gather_tyi (libreadline.so.7)
                                            #7  0x00007f0622117145 rl_read_key (libreadline.so.7)
                                            #8  0x00007f06220ff4b5 readline_internal_char (libreadline.so.7)
                                            #9  0x00007f06220ffc95 readline (libreadline.so.7)
                                            #10 0x00007f06281dfee4 n/a (/home/dima/src/octave/gcc_def/liboctave/.libs/liboctave.so.4.0.0)
                                            #11 0x0000000000000000 n/a (n/a)
                                            #12 0x00007f062821eea6 n/a (/home/dima/src/octave/gcc_def/liboctave/.libs/liboctave.so.4.0.0)
Dec 28 19:00:59 amd audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@1-2471-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'


I am still not sure what is the difference between the two computers...

Dmitri.
--


Dmitri A. Sergatskov <dasergatskov>
Fri 29 Dec 2017 12:49:06 AM UTC, comment #11: 

WRT comment #9 -- I am pretty sure he understands where the
multiple libraries are coming from. There are some talks on fedora dev lists  about fixing this mess, but  I am not convinced that
this is the problem. I still think the currently shipped Fedora's openblas has some issues, I just cannot come up with a convincing arguments (a simple test case).

Dmitri.
--


 

Dmitri A. Sergatskov <dasergatskov>
Fri 29 Dec 2017 12:42:06 AM UTC, comment #10: 

Michael: I'm fairly certain that the crash on exit is a different issue and that it has been resolved.  So it should be fixed in 4.2.2.

John W. Eaton <jwe>
Group administrator
Fri 29 Dec 2017 12:40:47 AM UTC, comment #9: 

Dmitri: Regarding the Fedora bug report, Octave does not directly link with multiple BLAS libraries as Susi seems to imply.  But we do link to other libraries provided by Fedora packages that link with other BLAS libraries, correct?  So, it seems like Fedora packaging should come up with a policy so that the user is allowed to chose a single BLAS implementation no matter what libraries are linked.  I mean, how are we supposed to fix that problem?

John W. Eaton <jwe>
Group administrator
Fri 29 Dec 2017 12:23:31 AM UTC, comment #8: 

An added note: The Octave 4.2.1 installed with F27 has
a failure, too. But rhs happens on exit:

[godfrey@pbdsl4 Mead]$ /usr/bin/octave --no-gui
GNU Octave, version 4.2.1
Copyright (C) 2017 John W. Eaton and others.
This is free software; see the source code for copying conditions.
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  For details, type 'warranty'.

Octave was configured for "x86_64-redhat-linux-gnu".

Additional information about Octave is available at http://www.octave.org.

Please contribute if you find this software useful.
For more information, visit http://www.octave.org/get-involved.html

Read http://www.octave.org/bugs.html to learn how to submit bug reports.
For information about changes from previous versions, type 'news'.

octave:1> exit

panic: Aborted -- stopping myself...
attempting to save variables to 'octave-workspace'...
panic: attempted clean up failed -- aborting...
Aborted (core dumped)

Michael Godfrey <godfrey>
Group Member
Fri 29 Dec 2017 12:18:33 AM UTC, comment #7: 

I cannot reproduce it with any blas on my computer...

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Fri 29 Dec 2017 12:16:24 AM UTC, comment #6: 

Works for me too.  You could compile with debugging and try to get  a backtrace.  That might help.

Rik <rik5>
Group administrator
Fri 29 Dec 2017 12:04:31 AM UTC, comment #5: 

My system shows:
octave:1>  _octave_config_info_("BLAS_LIBS")
ans = -lblas

but, the ploblem happens even immediately after signon.

Michael Godfrey <godfrey>
Group Member
Thu 28 Dec 2017 11:15:52 PM UTC, comment #4: 

Mine is built against self-compiled OpenBlas


./run-octave --no-gui -q -f
octave:1> version
ans = 4.3.0+
octave:2> ^C
octave:2> __octave_config_info__ ("hg_id")
ans = aab2355f3a77
octave:3> ^C
octave:3> __octave_config_info__("BLAS_LIBS")
ans = -lopenblas


and OpenBlas is built with OpenMP interface rather than
with pthread.

Dmitri.

P.S. #1: Fedora's OpenBlas is buggy and they refuse to fix it:
https://bugzilla.redhat.com/show_bug.cgi?id=1507744

P.P.S. Somebody convinced me that in multithreaded application one should not use pthreaded Openblas, though I thought the issue
was only the performance.


Dmitri A. Sergatskov <dasergatskov>
Thu 28 Dec 2017 10:49:27 PM UTC, comment #3: 

Are you using openblas?  I can reproduce the problem with that, but not with the reference blas or atlas.

John W. Eaton <jwe>
Group administrator
Thu 28 Dec 2017 10:30:26 PM UTC, comment #2: 

Maybe your OS is not Linux fc27.x86_64?

Also, just tried on ans = abb075ada761

same failure.

Michael Godfrey <godfrey>
Group Member
Thu 28 Dec 2017 10:22:27 PM UTC, comment #1: 

Hmm... Works for me:



./run-octave --no-gui -f -q
octave:1> version
ans = 4.3.0+
octave:2> ^C
octave:2> __octave_config_info__ ("hg_id")
ans = abb075ada761
octave:3> ^C
octave:3> quit


Dmitri.


Dmitri A. Sergatskov <dasergatskov>
Thu 28 Dec 2017 10:16:56 PM UTC, original submission:  

I thought that this was fixed, but current dev shows:
octave:1> version
ans = 4.3.0+
octave:2> ^C
octave:2>
octave:2>
octave:2> * stack smashing detected *: <unknown> terminated
Aborted (core dumped)

This is  call using: octave --no-gui

HG ID for this build is "abb075ada761"

Linux:Linux pbdsl4 4.14.8-300.fc27.x86_64

Michael Godfrey <godfrey>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #42789:  thread-sig.cc added by jwe (10KiB - text/x-c++src)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by godfrey (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-01-10 jwe StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2018-01-09 jwe StatusNone Ready For Test
    2018-01-09 godfrey Severity3 - Normal 5 - Blocker
        Priority5 - Normal 7 - High
        StatusFixed None
        Open/ClosedClosed Open
    2018-01-05 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2018-01-04 jwe StatusWorks For Me Ready For Test
    2018-01-02 jwe Attached File- Added thread-sig.cc, #42789
    2017-12-29 rik5 StatusNone Works For Me

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code