bugGNU Octave - Bugs: bug #49532, [octave forge] (parallel)...

 
 

bug #49532: [octave forge] (parallel) segmentation fault on matrix multiplication on macOS

Submitter:  None
Submitted:  Fri 04 Nov 2016 06:17:10 PM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Fixed Assigned to:  None
Originator Name:  Ioan Magdau Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 4.2.0
Operating System:  * Mac OS Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 21 Apr 2020 01:35:39 PM UTC, comment #12: 

Since the new implementation of parcellfun/pararrayfun uses completely independent Octave sessions, I don't think this report applies anymore and mark it at fixed. For new problems, a new report can be added.

Olaf Till <i7tiol>
Group Member
Tue 13 Dec 2016 08:13:13 PM UTC, comment #11: 

if I run

octave --no-window-system

the code runs fine.

Ioan Magdau <imagdau>
Thu 17 Nov 2016 06:07:05 PM UTC, comment #10: 

It should be added that, since the segfault is in one of the two child processes, you should attach to a child process with gdb -p PID. For this, you can insert a 'pause (long_time)' as the first command into your function 'par_test'. Or better, as the second command, and the first command should print out the pid of the process (getpid, as mentioned in the last post), but set page_screen_output to false before doing this.

For clarity: run pararrayfun with your changed par_test, and from another terminal, attach with gdb to the printed out pid, before par_test finishes its 'pause()'. Then give the command 'run' from gdb and hope to see the caught segfault. Then 'bt'.

But in debugging parallel processes it can happen that the segfault doesn't occur under debugging conditions. I'd believe you have good chances if the segfault occurs even if you set nproc to 1 in pararrayfun. Then you should try the debugging with nproc==1.

Olaf Till <i7tiol>
Group Member
Thu 17 Nov 2016 03:48:39 AM UTC, comment #9: 

Try either "gdb octave-cli" or run Octave, and then attach gdb to it in a separate terminal (use getpid in Octave, and gdb -p PID to attach to the running process).

Updating release tag to 4.2.0, please update to 4.2.0 when you can since it has been released.

Mike Miller <mtmiller>
Group Member
Wed 16 Nov 2016 02:29:28 AM UTC, comment #8: 

I managed to install gdb. I run gdb octave and then inside gdb: r testing.m.
This gives:

parcellfun: 0/2 jobs donepanic: Segmentation fault: 11 -- stopping myself...
panic: Segmentation fault: 11 -- stopping myself...
warning: parcellfun: premature exit due to closed pipe
warning: called from
    parcellfun at line 347 column 11
    pararrayfun at line 85 column 26
    testing at line 12 column 1

warning: broken pipe
warning: broken pipe
octave exited with signal 1
[Inferior 1 (process 10307) exited normally]
(gdb) bt
No stack.
(gdb)

Any ideas why there is no stack? Am I doing something wrong?

Anonymous
Thu 10 Nov 2016 04:29:41 PM UTC, comment #7: 

I tried octave-cli and it gives the same error. No difference.
gdb is giving me a hard time, haven't managed to make it work yet.

Anonymous
Tue 08 Nov 2016 08:35:37 PM UTC, comment #6: 

Have you tried octave-cli instead of "octave --no-gui" when logged in locally on the machine? They are not the same, "octave --no-gui" still starts a QApplication and runs the Octave interpreter on a secondary thread.

When you see the message "octave: no graphical display found" and "octave: disabling GUI features", that means that octave-cli is being run because it is running in a headless environment.

So again, the problem appears to me to be due to the use of fork in the parallel package when the interpreter is not the primary thread and it is running in a Qt context (even without the GUI).

If you have the gdb debugger, you can try running Octave in the way that causes it to segfault, attach to the process with gdb in a separate terminal, and then trigger the segfault. See http://wiki.octave.org/Debugging_Octave for some information on using gdb with Octave.

Mike Miller <mtmiller>
Group Member
Tue 08 Nov 2016 07:36:19 PM UTC, comment #5: 

This is becoming very strange. I have physical access to the Mac computer today. When I try to run the test code directly, it gives the old error:

parcellfun: 0/2 jobs donepanic: Segmentation fault: 11 -- stopping myself...
panic: Segmentation fault: 11 -- stopping myself...
warning: parcellfun: premature exit due to closed pipe
warning: called from
parcellfun at line 347 column 11
pararrayfun at line 85 column 26
testing at line 90 column 1
warning: broken pipe
warning: broken pipe

with both gui and no gui


If I access the Mac remotely from my Linux machine, and run octave testing.m, again I get:

parcellfun: 0/2 jobs donepanic: Segmentation fault: 11 -- stopping myself...
panic: Segmentation fault: 11 -- stopping myself...
warning: parcellfun: premature exit due to closed pipe
warning: called from
    parcellfun at line 347 column 11
    pararrayfun at line 85 column 26
    testing at line 90 column 1

warning: broken pipe
warning: broken pipe


If I, additionally, logout from the Mac and login via ssh from my Linux machine, and run octave testing.m,

Nov  8 08:20:53  octave[1055] <Error>: Set a breakpoint at CGSLogError to catch errors as they are logged.
Nov  8 08:20:53  octave[1055] <Error>: This user is not allowed access to the window system right now.
octave: no graphical display found
octave: disabling GUI features
Nov  8 08:20:53  octave-cli-4.2.0-rc2[1055] <Error>: Set a breakpoint at CGSLogError to catch errors as they are logged.
Nov  8 08:20:53  octave-cli-4.2.0-rc2[1055] <Error>: This user is not allowed access to the window system right now.
parcellfun: 2/2 jobs done

So now it complains when starting octave, but the code runs fine. I am clueless.

About BLAS, I did otools -L /usr/lib/libblas.dylib this is what I get:
 
/usr/lib/libblas.dylib:
/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.0.0)

I don't know how to check the stack on octave.

Does this help?

Ioan Magdau <imagdau>
Tue 08 Nov 2016 07:31:04 PM UTC, comment #4: 


> I want to help solve the issue.


Then can you please try to use the bug tracker web interface to participate?

Mike Miller <mtmiller>
Group Member
Tue 08 Nov 2016 06:45:12 PM UTC, comment #3: 

Ioan - your replies via direct email reply are not sent to the bug tracker, please copy the relevant details and submit them on the bug tracker web interface.

To summarize what you have provided privately, it sounds like octave-cli (Octave command-line only, no GUI threading) works fine for you, but segfaults when you try to use the parallel package with Qt loaded. If all you care about is running the interpreter in command mode, perhaps you should always use octave-cli for consistent results. Do you want to help resolve this problem beyond that point?

Mike Miller <mtmiller>
Group Member
Sun 06 Nov 2016 09:05:48 AM UTC, comment #2: 

As yet there was no reply to Mikes request for additional information. Especially the information regarding BLAS would be important for verification that this is the right track of thinking.

This would be not the first issue with threads in combination with 'fork' in parcellfun. I think here we are at the limit of what can be done with just 'fork'ing.

A solution would be a different way, similar to what is done in netcellfun, but with local servers, spawned with something like fork/ecexve.

I'm now planning to implement such an alternative solution, but it will be in a different function than parcellfun, maybe within netcellfun.

This alternative way could even make a Windows (non-cygwin) version theoretically possible, using CreateProcess, though I currently can't assess the possibilities for process communication there.

Olaf Till <i7tiol>
Group Member
Fri 04 Nov 2016 09:32:46 PM UTC, comment #1: 

Adding parallel package maintainer in cc.

A debug stack trace would probably be helpful here in discovering the reason for the segmentation fault.

Which BLAS library is being used? Does setting OMP_NUM_THREADS=1 in the environment before running Octave change anything?

Mike Miller <mtmiller>
Group Member
Fri 04 Nov 2016 06:17:10 PM UTC, original submission:  

--------------code---------------

pkg load parallel;

function par_test(t)

    N=128;
    A=randi(N,N);
    B=randi(N,N);
    A*B;
   
endfunction

pararrayfun(nproc, @par_test, 1:3, "UniformOutput", false)

--------------Error--------------

parcellfun: 0/2 jobs donepanic: Segmentation fault: 11 -- stopping myself...
panic: Segmentation fault: 11 -- stopping myself...
warning: parcellfun: premature exit due to closed pipe
warning: called from
    parcellfun at line 347 column 11
    pararrayfun at line 85 column 26
    testing at line 90 column 1

warning: broken pipe
warning: broken pipe

--------------Info--------------

parallel version 3.1.1

-------------Comments----------

simply running par_test(1) works
running in parallel works for N<128, gives a seg fault for N>=128
running in parallel works on a Linux machine


Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Updated the item)
  • -email is unavailable- added by imagdau (Posted a comment)
  • -email is unavailable- added by i7tiol (Posted a comment)
  • -email is unavailable- added by mtmiller
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-04-21 i7tiol StatusNeed Info Fixed
        Open/ClosedOpen Closed
    2019-02-26 mtmiller Carbon-CopyRemoved 80942 -
    2017-08-13 jwe Summaryparallel package: segmentation fault on matrix multiplication on macOS [octave forge] (parallel) segmentation fault on matrix multiplication on macOS
    2016-11-17 mtmiller Release4.2.0-rc2 4.2.0
    2016-11-04 mtmiller StatusNone Need Info
        Summarymatrix multiplication in package parallel parallel package: segmentation fault on matrix multiplication on macOS
        Carbon-Copy- Added i7tiol

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code