bugGNU Octave - Bugs: bug #47478, [octave forge] (parallel)...

 
 

bug #47478: [octave forge] (parallel) parcellfun() hangs when used with @fft2()

Submitter:  Brett Merrick <octave_up>
Submitted:  Sun 20 Mar 2016 05:19:32 PM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 4.0.0 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 21 Apr 2020 01:47:11 PM UTC, comment #15: 

The current implementation of parcellfun/pararrayfun shouldn't have problems with threads anymore and the original problem runs for me now, except that "UniformOutput" has to be set to false.
Closing and marking as fixed.

Olaf Till <i7tiol>
Group Member
Sat 25 Feb 2017 04:44:01 PM UTC, comment #14: 

FTR, the mentioned unrelated problem with the child not returning is bug #50400.

Olaf Till <i7tiol>
Group Member
Sat 25 Feb 2017 10:06:12 AM UTC, comment #13: 

Forgot to write:

make check yielded

Summary:

  PASS     15008
  FAIL        20
  XFAIL       38
  SKIPPED    204

with and without the patch.

Olaf Till <i7tiol>
Group Member
Sat 25 Feb 2017 10:00:58 AM UTC, comment #12: 

Attached a fix, and a test without package 'parallel', printing 3 lines without and 4 with the fix (child doesn't return due to unrelated problem with current tip).

Rationale:

Previously, fftw threads, handled by a 'singleton' object, were created only at first usage of an fft function. If created before fork, child sees threads have been created, but has only one of these threads. Just recreating the threads in the child was tried in previous posts but did not work. See 'man pthread_atfork' for possible problems with mutexes.

The patch uses 'pthread_atfork()' to register functions, which de-initialize the fftw singleton objects, to be called by the parent before each fork(). Re-initialization, in parent and child, doesn't need to be registered, because it's handled by the singleton objects.

The patch is a minimal solution. It moves a function from private: to public:. Or should a new 'atfork_cleanup_list' be created (similar to existing 'singleton_cleanup_list') and the fftw-deinitializer be added to it? Should a test file be created, which might hang Octave?

(file #39827, file #39828)

Olaf Till <i7tiol>
Group Member
Tue 22 Mar 2016 04:30:43 PM UTC, comment #11: 

Ok, I see the typo now, thank you, I fixed that here:

http://hg.savannah.gnu.org/hgweb/octave/rev/e1c921322ba2

Briefly, the reason for removing conditionals from the headers is so we don't need to install config.h anymore. This header caused conflicts with other autotools-based projects. It also makes Octave's API more consistent and stable if it doesn't change depending on how Octave was built, if the same functions and symbols are always defined.

Mike Miller <mtmiller>
Group Member
Tue 22 Mar 2016 08:16:30 AM UTC, comment #10: 

I had inserted test output into maybe_init_threads into the block where fftw_init_threads is called, and it was called for each process, as expected. It only did not help to fix the bug.

As for the supposed typo, I'm aware that there are both FFTW3F and FFTW3, what I meant is that at one instance FFTW3F was used at the wrong place (within a member function of the normal double-precission class).

I already got the impression that removing preprocessor conditionals from the header was purposed, but wasn't sure. What is the rationale behind this?

Olaf Till <i7tiol>
Group Member
Mon 21 Mar 2016 11:48:21 PM UTC, comment #9: 

Thanks for the revision. Your patch applies for me now, I've made some further tweaks to it but parcellfun still hangs on the fft2 call with the original demo script, so I haven't seen it work yet. I set a breakpoint for fftw_init_threads and only saw it called once, but that's as far as I've debugged.

Attaching my revised patch, note that we are removing all conditionals out of public Octave header files. And that is not a typo, there should be two #defines in config.h, one for single-precision FFTW3F and one for double-precision FFTW3, each one is tested independently for the presence of multithreading support.

(file #36716)

Mike Miller <mtmiller>
Group Member
Mon 21 Mar 2016 08:05:53 PM UTC, comment #8: 

Mike, I have updated (and slightly changed) the patch. Unfortunately, a test showed that it does not fix the bug any more, although it seemed so with the last version (but I was in a hurry then, I hope I had not made a mistake in testing ...).

So this was no success. Don't know if I'll try to dig deeper in. I attach the patch for the record.

BTW, there is an unrelated probable typo of JWE in liboctave/numeric/oct-fftw.cco (ctave_fftw_planner::threads (int nt)): HAVE_FFTW3F_THREADS should probably be HAVE_FFTW3_THREADS (without the last F).

(file #36714)

Olaf Till <i7tiol>
Group Member
Mon 21 Mar 2016 07:33:18 PM UTC, comment #7: 

Thanks for looking into it.

Olaf, I tried it and parcellfun does indeed work on fft2 when fft2 has not been previously called in that Octave instance.

As to whether it is a smart or stupid thing to use parcellfun on fft2, my quick tests suggest that that depends entirely on the matrix sizes involved.

On small matrices, straight calls to fft2 are much faster than via parcellfun. On very large matrices, fft2 makes little use of the available cores, even when I set: fftw("threads", 4).
In such cases, it is faster -on my machine at least- to use parcellfun with 4 processes on fft2 with fftw("threads", 1), but only if the available RAM is sufficient to prevent swap memory from being used.

Your mileage may vary.

Thanks again for the help guys!

Brett Merrick <octave_up>
Mon 21 Mar 2016 05:46:38 PM UTC, comment #6: 

The patch does not apply to the current Octave default branch, so I can't test, but this does seem like the right approach. Can you update to the current oct-fftw sources?

Mike Miller <mtmiller>
Group Member
Mon 21 Mar 2016 07:34:20 AM UTC, comment #5: 

Agreed. So attached is a patch that fixes this in Octave (development version).

The patch probably requires some cleanup, which I postpone until someone says the patch will actually be applied once it is cleaned up enough.

(file #36708)

Olaf Till <i7tiol>
Group Member
Sun 20 Mar 2016 11:49:54 PM UTC, comment #4: 

That sounds a plausible explanation, Olaf.

I think it is important that calls to fft2 via parcellfun should "work" in some sense, even if the solution is to have a special case that doesn't call it in parallel.  This is quite a natural thing to try, and even completely stupid user code should not cause Octave to crash or hang.

Lachlan Andrew <lachlan>
Sun 20 Mar 2016 10:01:27 PM UTC, comment #3: 

From my tests, what is hanging is the call to fft2 itself. And it only happens if fft2 had been already called in the parent (before calling it with parcellfun).

My current knowledge on threading implementations is not so great, so the next should be taken with caution.

I think principally a multithreaded function can be called in parallel by multiple processes, as in parcellfun, although there will probably be no advantage in doing this. But there may be preconditions.

In the current case, I'd guess (without studying the threading implementation in libfftw) that the call to the libfftw function 'int fftw_init_threads (void)' must be performed in each process. But Octave, using static class members, provides for calling this function only once in an Octave session. The call takes place if some fft user function is called for the first time. So if fft2 is called in the parent, the call to fftw_init_threads() is performed in the parent and not in the children generated by parcellfun, although it probably should be repeated there.

Conclusions:

1. parcellfun probably can't do anything about it, so the bug is probably not in parcellfun.

2. It's probably not worth calling fft2 with parcellfun, even if it should work. You can test it, if you start Octave anew and call parcellfun without calling fft2 before in the parent. Please do this and compare the time with pure fft2 invocations.

3. If we want to deal with this issue (despite of 2.), one could try to write some testcode which repeats the call to fftw_init_threads, call this testfunction in parcellfun and see if it helps. But my prediction is that a submitted patch to Octave which moves the call of fftw_init_threads out of the one-time-initialization won't be considered for application.

Olaf Till <i7tiol>
Group Member
Sun 20 Mar 2016 08:47:20 PM UTC, comment #2: 

Thanks for taking the time to test it on your machine.

Like you, I also wondered whether the fft* functions are already multi-threaded. They do not appear to be. This I say based on visual inspection of the CPU core loads, using the System Monitor in Mint 17, while calling fft2 on a large matrix. If I use a matrix large enough, a single call to fft2 can take about 15 seconds, in which only one CPU core seems to be doing all the work.

In fact, that observation was the main reason I tried parcellfun on fft2. The 2D FFT is the bottleneck in my computations, and fft2 does not seem to make good use of the available CPU cores. So I figured I'd pass different matrices to different processes.


Brett Merrick <octave_up>
Sun 20 Mar 2016 06:45:32 PM UTC, comment #1: 

I can confirm this with your example here. Adding the parallel package maintainer to the report.

Just a guess, but maybe because the fft* functions are already multi-threaded?

Mike Miller <mtmiller>
Group Member
Sun 20 Mar 2016 05:19:32 PM UTC, original submission:  

I have used parcellfun() with much succes before, and I think it is an excellent function. Unfortunately, whenever I try to use parcellfun() to perform fft2() calls in parallel, that call to parcellfun() never finishes. I have to then kill Octave because it hangs.

Included is a self-contained script that causes this behavior when I run it from the terminal by typing (without the quotes): "octave bug_demo.m"

My setup:
Linux Mint 17
GNU Octave 4.0.0

I installed GNU Octave as follows:
sudo apt-add-repository ppa:octave/stable
sudo apt-get update
sudo apt-get install octave

Any help is much appreciated, thanks!

Brett Merrick <octave_up>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #39827:  reset-fftw-at-fork.cset added by i7tiol (3KiB - application/octet-stream)
file #39828:  demo_47478.m added by i7tiol (509B - text/x-objcsrc)
file #36716:  bug-47478-mtm.patch added by mtmiller (6KiB - text/x-diff)
file #36714:  bug-47478-not-working.cset added by i7tiol (6KiB - application/octet-stream)
file #36708:  bug-47478.cset added by i7tiol (5KiB - application/octet-stream)
file #36701:  bug_demo.m added by octave_up (653B - text/x-objcsrc - run it from the terminal by typing (without the quotes): "octave bug_demo.m")

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Updated the item)
  • -email is unavailable- added by lachlan (Posted a comment)
  • -email is unavailable- added by i7tiol (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by mtmiller
  • -email is unavailable- added by octave_up (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 17 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-04-21 i7tiol StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2017-08-13 jwe Summaryparallel package: parcellfun() hangs when used with @fft2() [octave forge] (parallel) parcellfun() hangs when used with @fft2()
    2017-02-25 i7tiol Attached File- Added reset-fftw-at-fork.cset, #39827
        Attached File- Added demo_47478.m, #39828
    2016-03-21 mtmiller Attached File- Added bug-47478-mtm.patch, #36716
        StatusConfirmed In Progress
    2016-03-21 i7tiol Attached File- Added bug-47478-not-working.cset, #36714
    2016-03-21 i7tiol Attached File- Added bug-47478.cset, #36708
    2016-03-20 mtmiller CategoryOctave Function Octave Package
        Item GroupSegfault, Bus Error, etc. Incorrect Result
        StatusNone Confirmed
        Summaryparcellfun() hangs when used with @fft2() parallel package: parcellfun() hangs when used with @fft2()
        Carbon-Copy- Added i7tiol
    2016-03-20 octave_up Carbon-CopyRemoved octave_up -
    2016-03-20 octave_up Attached File- Added bug_demo.m, #36701
        Carbon-Copy- Added octave_up

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code