bugGNU Octave - Bugs: bug #30477, "help" command hangs...

 
 

bug #30477: "help" command hangs because SIGCHLD handler is not async-signal-safe

Submitter:  Alex Lancaster <alexl>
Submitted:  Sun 18 Jul 2010 07:24:48 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 3.2.4 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 17 Jan 2011 11:28:28 PM UTC, comment #13: 

Indeed, that change appears to have fixed the problem.  When I tested revision 26a6435857bc using the procedure in file #21502, the problem did not occur.  When I reverted revision f359cfc6e24d, the problem comes back, so I know I am doing the right test.

Matt McCutchen <hashproduct>
Mon 17 Jan 2011 07:45:32 PM UTC, comment #12: 

I checked in the following change:

http://hg.savannah.gnu.org/hgweb/octave/rev/f359cfc6e24d

If this change fixes the problem for you, then I will close this report.

This problem was also reported again here:

https://savannah.gnu.org/bugs/?32137

John W. Eaton <jwe>
Group administrator
Fri 12 Nov 2010 09:28:09 PM UTC, comment #11: 

Addition to my comment 10.  I get results that are consistent
with Dimitri's in comment #7.  It appears that builtin
functions are more likely to work the first time, and after
the first help that works it continues to work.  It is
some, but not all,  help <function found in file...>, that
fail on first use of help.  This does suggest that the
analysis in comment #8 is relevant.

If all else fails, a possible "solution" might be to do a
fake "help sin" (or something equivalent) during initialization.

Better would be to discover what it is about the first successful
use that makes subsequent uses work.  It seems likely that this
is connected with malloc use somehow.

Michael Godfrey
p.s. Sorry that I included the "bt" inline below. I should have
     attached it.

Anonymous
Fri 12 Nov 2010 08:59:38 PM UTC, comment #10: 

This problem, for me, seems to "come and go."  Now, on a recent
development build, it is back.  Octave always hangs on a
help command which names an existing function.  It returns
if the name does not match.

Are others still getting this?
My system is Fedora:Linux pbds.stanford.edu 2.6.35.6-48.fc14.x86_64 #1 SMP Fri Oct 22 15:36:08 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux

A gdb bt is:

(gdb) run
Starting program: /usr/local/bin/octave
[Thread debugging using libthread_db enabled]
GNU Octave, version 3.3.53+
Copyright (C) 2010 John W. Eaton and others.
This is free software; see the source code for copying conditions.
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  For details, type `warranty'.

Octave was configured for "x86_64-unknown-linux-gnu".

Additional information about Octave is available at http://www.octave.org.

Please contribute if you find this software useful.
For more information, visit http://www.octave.org/help-wanted.html

Read http://www.octave.org/bugs.html to learn how to submit bug reports.

For information about changes from previous versions, type `news'.

octave:1> help legend
Detaching after fork from child process 13094.
^C
Program received signal SIGINT, Interrupt.
0x00000036192f349b in __lll_lock_wait_private () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install arpack-2.1-12.fc14.x86_64 atlas-3.8.3-18.fc14.x86_64 blas-3.2.2-2.fc14.x86_64 expat-2.0.1-10.fc13.x86_64 fltk-1.1.10-2.fc14.x86_64 fontconfig-2.8.0-2.fc14.x86_64 freetype-2.4.2-3.fc14.x86_64 glibc-2.12.90-18.x86_64 hdf5-1.8.5.patch1-4.fc14.x86_64 libX11-1.3.4-3.fc14.x86_64 libXau-1.0.6-1.fc14.x86_64 libXext-1.1.2-2.fc14.x86_64 libXft-2.1.14-1.fc13.x86_64 libXinerama-1.1-2.fc13.x86_64 libXrender-0.9.6-1.fc14.x86_64 libgcc-4.5.1-4.fc14.x86_64 libgfortran-4.5.1-4.fc14.x86_64 libstdc++-4.5.1-4.fc14.x86_64 libxcb-1.7-1.fc14.x86_64 mesa-libGLU-7.9-2.fc14.x86_64 ncurses-libs-5.7-8.20100703.fc14.x86_64 pcre-8.10-1.fc14.x86_64 qrupdate-1.1.1-1.fc13.x86_64 readline-6.1-2.fc14.x86_64 suitesparse-3.4.0-2.fc12.x86_64 zlib-1.2.5-2.fc14.x86_64
(gdb) bt
#0  0x00000036192f349b in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x000000361927ee8c in _L_lock_11783 () from /lib64/libc.so.6
#2  0x000000361927c927 in malloc () from /lib64/libc.so.6
#3  0x00000036232ba78d in operator new(unsigned long) () from /usr/lib64/libstdc++.so.6
#4  0x00007ffff75fa1a2 in octave_child_list::instance_ok () at sighandlers.cc:821
#5  0x00007ffff75fa529 in octave_child_list::wait () at sighandlers.cc:850
#6  0x00007ffff75fa59d in sigchld_handler () at sighandlers.cc:234
#7  <signal handler called>
#8  0x0000003619277f0c in malloc_consolidate () from /lib64/libc.so.6
#9  0x00000036192798e8 in _int_malloc () from /lib64/libc.so.6
#10 0x000000361927c932 in malloc () from /lib64/libc.so.6
#11 0x00000036232ba78d in operator new(unsigned long) () from /usr/lib64/libstdc++.so.6
#12 0x000000362329cfb9 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) ()
   from /usr/lib64/libstdc++.so.6
#13 0x000000362329dd4b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_M_clone(std::allocator<char> const&, unsigned long) () from /usr/lib64/libstdc++.so.6
#14 0x000000362329de3c in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) () from /usr/lib64/libstdc++.so.6
#15 0x0000003623296e13 in std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow(int) () from /usr/lib64/libstdc++.so.6
#16 0x00000036232912bb in std::basic_ostream<char, std::char_traits<char> >::put(char) ()
   from /usr/lib64/libstdc++.so.6
#17 0x00007ffff763e868 in run_command_and_return_output (args=<value optimized out>,
    nargout=-19392) at toplev.cc:793
#18 Fsystem (args=<value optimized out>, nargout=-19392) at toplev.cc:973
#19 0x00007ffff76b323b in octave_builtin::do_multi_index_op (this=0x765c98, nargout=2,
    args=..., lvalue_list=0x7fffffffb620) at ov-builtin.cc:129
#20 0x00007ffff76b2197 in octave_builtin::subsref (this=0x765c98, type="(", idx=
---Type <return> to continue, or q <return> to quit---
    std::list = {...}, nargout=2, lvalue_list=<value optimized out>) at ov-builtin.cc:64
#21 0x00007ffff7787196 in octave_value::subsref (this=<value optimized out>,
    type=<value optimized out>, idx=<value optimized out>, nargout=<value optimized out>,
    lvalue_list=<value optimized out>) at ov.cc:1211
#22 0x00007ffff783561d in tree_index_expression::rvalue (this=<value optimized out>, nargout=2,
    lvalue_list=0x7fffffffbea0) at pt-idx.cc:408
#23 0x00007ffff7809d05 in tree_multi_assignment::rvalue (this=0xc875c0) at pt-assign.cc:365
#24 0x00007ffff7808a1d in tree_multi_assignment::rvalue1 (this=0xc875c0, nargout=0)
    at pt-assign.cc:320
#25 0x00007ffff7820d55 in tree_evaluator::visit_statement (this=0x7ffff7ffb5d8,
    stmt=<value optimized out>) at pt-eval.cc:729
#26 0x00007ffff781ab90 in tree_evaluator::visit_statement_list (this=0x7ffff7ffb5d8, lst=...)
    at pt-eval.cc:765
#27 0x00007ffff781d796 in tree_evaluator::visit_unwind_protect_command (this=0x7ffff7ffb5d8,
    cmd=<value optimized out>) at pt-eval.cc:996
#28 0x00007ffff7820c9d in tree_evaluator::visit_statement (this=0x7ffff7ffb5d8, stmt=...)
    at pt-eval.cc:700
#29 0x00007ffff781ab90 in tree_evaluator::visit_statement_list (this=0x7ffff7ffb5d8, lst=...)
    at pt-eval.cc:765
#30 0x00007ffff777d2ed in octave_user_function::do_multi_index_op (this=0x6aa040, nargout=2,
    args=<value optimized out>, lvalue_list=0x7fffffffcba0) at ov-usr-fcn.cc:450
#31 0x00007ffff7777896 in octave_user_function::subsref (this=0x6aa040, type="(", idx=
    std::list = {...}, nargout=2, lvalue_list=0x7fffffffcba0) at ov-usr-fcn.cc:303
#32 0x00007ffff7787196 in octave_value::subsref (this=<value optimized out>,
    type=<value optimized out>, idx=<value optimized out>, nargout=<value optimized out>,
    lvalue_list=<value optimized out>) at ov.cc:1211
#33 0x00007ffff783561d in tree_index_expression::rvalue (this=<value optimized out>, nargout=2,
    lvalue_list=0x7fffffffcba0) at pt-idx.cc:408
#34 0x00007ffff7809d05 in tree_multi_assignment::rvalue (this=0xa1e720) at pt-assign.cc:365
---Type <return> to continue, or q <return> to quit---
#35 0x00007ffff7808a1d in tree_multi_assignment::rvalue1 (this=0xa1e720, nargout=0)
    at pt-assign.cc:320
#36 0x00007ffff7820d55 in tree_evaluator::visit_statement (this=0x7ffff7ffb5d8,
    stmt=<value optimized out>) at pt-eval.cc:729
#37 0x00007ffff781ab90 in tree_evaluator::visit_statement_list (this=0x7ffff7ffb5d8, lst=...)
    at pt-eval.cc:765
#38 0x00007ffff781ebdc in tree_evaluator::visit_switch_command (this=0x7ffff7ffb5d8,
    cmd=<value optimized out>) at pt-eval.cc:847
#39 0x00007ffff7820c9d in tree_evaluator::visit_statement (this=0x7ffff7ffb5d8, stmt=...)
    at pt-eval.cc:700
#40 0x00007ffff781ab90 in tree_evaluator::visit_statement_list (this=0x7ffff7ffb5d8, lst=...)
    at pt-eval.cc:765
#41 0x00007ffff7820c9d in tree_evaluator::visit_statement (this=0x7ffff7ffb5d8, stmt=...)
    at pt-eval.cc:700
#42 0x00007ffff781ab90 in tree_evaluator::visit_statement_list (this=0x7ffff7ffb5d8, lst=...)
    at pt-eval.cc:765
#43 0x00007ffff777d2ed in octave_user_function::do_multi_index_op (this=0x6a9ba8, nargout=0,
    args=<value optimized out>, lvalue_list=0x0) at ov-usr-fcn.cc:450
#44 0x00007ffff7777896 in octave_user_function::subsref (this=0x6a9ba8, type="(", idx=
    std::list = {...}, nargout=0, lvalue_list=0x0) at ov-usr-fcn.cc:303
#45 0x00007ffff7777fcc in octave_user_function::subsref (this=<value optimized out>,
    type=<value optimized out>, idx=<value optimized out>, nargout=<value optimized out>)
    at ov-usr-fcn.cc:286
#46 0x00007ffff7787129 in octave_value::subsref (this=<value optimized out>,
    type=<value optimized out>, idx=<value optimized out>, nargout=<value optimized out>)
    at ov.cc:1202
#47 0x00007ffff77871a5 in octave_value::subsref (this=<value optimized out>,
    type=<value optimized out>, idx=<value optimized out>, nargout=<value optimized out>,
    lvalue_list=<value optimized out>) at ov.cc:1213
---Type <return> to continue, or q <return> to quit---
#48 0x00007ffff783561d in tree_index_expression::rvalue (this=<value optimized out>, nargout=0,
    lvalue_list=0x0) at pt-idx.cc:408
#49 0x00007ffff783760b in tree_index_expression::rvalue (this=<value optimized out>,
    nargout=<value optimized out>) at pt-idx.cc:278
#50 0x00007ffff782d72d in tree_index_expression::rvalue1 (this=0xb64900, nargout=0)
    at pt-idx.cc:419
#51 0x00007ffff7820d55 in tree_evaluator::visit_statement (this=0x7ffff7ffb5d8,
    stmt=<value optimized out>) at pt-eval.cc:729
#52 0x00007ffff781ab90 in tree_evaluator::visit_statement_list (this=0x7ffff7ffb5d8, lst=...)
    at pt-eval.cc:765
#53 0x00007ffff763c8b9 in main_loop () at toplev.cc:570
#54 0x00007ffff75d9ae7 in octave_main (argc=<value optimized out>, argv=<value optimized out>,
    embedded=0) at octave.cc:894
#55 0x000000361921ee7d in __libc_start_main () from /lib64/libc.so.6
#56 0x00000000004006e9 in _start ()
(gdb)

Anonymous
Mon 20 Sep 2010 03:07:06 AM UTC, comment #9: 

I can reproduce the problem with the current development Octave.  Attached is a new gdb transcript.

John, I have a guess as to why the gdb procedure did not work for you.  It requires the multi-executable support added in gdb 7.1; perhaps you have an older version of gdb.  I found an alternative that should work with older gdb versions: instead of attaching gdb to the child, wrap makeinfo with a shell script that stops itself so the user can continue it and get the SIGCHLD at the desired time.  The makeinfo wrapper and the transcript to reproduce the bug using it are attached.

(file #21502, file #21503, file #21504)

Matt McCutchen <hashproduct>
Sun 19 Sep 2010 08:35:55 PM UTC, comment #8: 

I tried to repeat the test with the current development Octave, but I got a build failure; I submitted bug #31074.

> OR, it would help if someone could explain how Octave's sigchld_handler function in src/sighandlers.cc should be changed so that it will work properly on your system.


Suppose the main program is in the middle of a call to __libc_malloc (the common backend to malloc, operator new, etc.).  If a SIGCHLD arrives, sigchld_handler is invoked, and it may invoke __libc_malloc again via octave_child_list::instance_ok.  __libc_malloc takes a lock, so the reentrant call causes a hang.  If it weren't for that lock, attempting to allocate on a heap already in an intermediate state from the outer __libc_malloc would cause corruption.

See the discussion of async-signal-safe functions in the POSIX standard:

http://www.opengroup.org/onlinepubs/9699919799/xrat/V4_xsh_chap02.html#tag_22_02_04_04

The typical solution would be to redesign the sigchld_handler so that it never calls __libc_malloc.  It would also work to block SIGCHLD in any part of the main program where a SIGCHLD could arrive during a call to __libc_malloc, or switch to handling SIGCHLD via signalfd (but signalfd is only available on Linux, and this problem probably affects other unices too).

Matt McCutchen <hashproduct>
Tue 14 Sep 2010 09:25:29 PM UTC, comment #7: 

On different computers all with Fedora 13, but different
hardware and different octave tip i get essentially 100%
reproducible error if I do "help help" first thing upon
octave invocation. If I do instead "help sin" (that
works correctly), then after that "help help" works
fine.

I know it is not too helpful, but that all i got at
the moment.

Sincerely,

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Tue 14 Sep 2010 05:34:14 PM UTC, comment #6: 

I'm unable to reproduce this problem just by typing "help somefunction".

I tried to duplicate the steps in the gdb transcript

  https://bugzilla.redhat.com/attachment.cgi?id=432711

using the development version of Octave on my Debian system.  I made it as far as the

  break _exit
  continue

step, but then I saw

(gdb) break _exit
Breakpoint 4 at 0x7ffff7df4e90: file ../sysdeps/unix/sysv/linux/_exit.c, line 33. (2 locations)
(gdb) continue
Continuing.

Program exited normally.
(gdb) inferior 1
Inferior has no threads.

I don't know what to do at this point.  If you give me something that shows how to reproduce the problem I might be able to debug it.

OR, it would help if someone could explain how Octave's sigchld_handler function in src/sighandlers.cc should be changed so that it will work properly on your system.

John W. Eaton <jwe>
Group administrator
Sat 31 Jul 2010 08:10:00 PM UTC, comment #5: 

Today, after few days of running without any problem, I have
triggered this bug again. Now, I would trigger it with almost 100%
certainty. I tried running octave from a dummy user account (to make sure there is nothing in my environment have changed) and the
bug triggers there as well. "help" now hangs on the existing help entries, it just returns on invalid ones.
This is still with 10823:3d89d262f5d4 tip.

Sincerely,

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Mon 26 Jul 2010 04:00:36 AM UTC, comment #4: 

I assume the code has changed since 3.3.51 and now bug either
is not there or is not triggered.

I really do not know gdb and do not understand what is happening
in that log, so cannot comment.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 26 Jul 2010 03:28:45 AM UTC, comment #3: 

Re comment #2: has the code changed, or did you just get lucky?  On the Fedora bug report, there is a demo of how to force the interleaving that triggers the bug with gdb.

Matt McCutchen <hashproduct>
Mon 26 Jul 2010 12:47:17 AM UTC, comment #2: 

I can NOT reproduce this with a recent (2010-07-25) hg tip.

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Tue 20 Jul 2010 02:33:50 AM UTC, comment #1: 

I can reproduce it with 3.2.4 as well as with 3.3.51.
I can reproduce this almost 100% of a time with 3.3.51 with
help junk111
(or any other non-existing entry).
With valid entries it is still intermittent. Setting
"more off" seems to make the bug less probable.

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Sun 18 Jul 2010 07:24:48 PM UTC, original submission:  

A downstream user, Matt McCutchen, has submitted the following bug report of a hang in our Bugzilla instance:

https://bugzilla.redhat.com/show_bug.cgi?id=578665

The backtrace is here:

https://bugzilla.redhat.com/attachment.cgi?id=403844

I can't reproduce the problem myself, but here's his description for how to reproduce it:

"Description of problem:
Octave's SIGCHLD handler calls operator new, which is not async-signal-safe.  This can cause the "help" command to hang if the SIGCHLD from the info subprocess arrives while octave was already in a call to operator new.

Version-Release number of selected component (if applicable):
octave-3.2.3-4.fc12.x86_64

How reproducible:
Rarely and randomly.  I haven't found any tricks to make it happen more often.

Steps to Reproduce:
1. Start octave.
2. help clear

Actual results:
Octave hangs (see attached backtrace).

Expected results:
The "help" command completes. "

Alex Lancaster <alexl>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #21502:  octave-dev-30477-gdb-repro.typescript added by hashproduct (17KiB - application/octet-stream)
file #21503:  makeinfo added by hashproduct (333B - application/octet-stream)
file #21504:  octave-dev-30477-oldgdb-repro.typescript added by hashproduct (14KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by hashproduct (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by alexl (Submitted the item)
  • -email is unavailable- added by alexl (original reporter in Fedora)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-01-18 jwe StatusNeed Info Fixed
        Open/ClosedOpen Closed
    2010-09-20 hashproduct Attached File- Added octave-dev-30477-gdb-repro.typescript, #21502
        Attached File- Added makeinfo, #21503
        Attached File- Added octave-dev-30477-oldgdb-repro.typescript, #21504
    2010-09-14 jwe StatusNone Need Info
    2010-07-18 alexl Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code