bugGNU Octave - Bugs: bug #55296, [octave forge] (parallel) Issue...


bug #55296: [octave forge] (parallel) Issue with parcellfun after update

Submitted by:  None
Submitted on:  Thu 27 Dec 2018 01:02:34 PM UTC  
Category:  Octave Forge Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error
Status:  Invalid Assigned to:  Olaf Till <i7tiol>
Originator Name:  Michael Rackl Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  4.2.2
Operating System:  GNU/Linux

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Sun 20 Jan 2019 06:04:26 PM UTC, comment #9: 

For "UniformOutput" set to true, the user function is expected to return only scalars. (This is stated in the help-text of cellfun, to which the help-text of parcellfun refers.)

I'm currently not sure if we should check in parcellfun if the returned values are scalars since I haven't yet measured the performance impact.

Sorry, but I haven't got the time to check if older versions of parcellfun "accepted" your code, or to help you with your code.

Closing this bug as invalid.

Olaf Till <i7tiol>
Project MemberIn charge of this item.
Sat 19 Jan 2019 06:16:28 PM UTC, comment #8: 

Thanks for the input!

I am sorry, but I think I misunderstood the question about what anonym_Start returns. The single results we expect are scalars, but anonym_Start collects them in an array if there is more than one result. Thus, anonym_Start could return an array or a scalar, i.e., a 1x1 array.

If we set 'UniformOutput' to 'false', the code goes past the call to 'parcellfun' without error. This is not bad for a first try. (However, we run into problems thereafter because our code was not built to handle cells.)

I would be interested in what changed in 'parcellfun', because it used to be able to handle our structure. This could help us to understand what we need to change in our code.

Could you please advise?

Mon 14 Jan 2019 09:23:14 PM UTC, comment #7: 

Please really check it, with 'isscalar ()', for each set of arguments. The returned error is what is to be expected if a function doesn't always return scalars and "UniformOutput" is set to true (which is the default). Also, the arguments to anonym_Start differ in dimensions from call to call, which may mean that the same applies to the returned result of this function.

Olaf Till <i7tiol>
Project MemberIn charge of this item.
Mon 14 Jan 2019 06:31:40 PM UTC, comment #6: 

Yes, we are expecting scalars from 'anonym_Start'.

Fri 11 Jan 2019 09:00:53 AM UTC, comment #5: 

Does your function 'anonym_Start' always return scalars?

Olaf Till <i7tiol>
Project MemberIn charge of this item.
Wed 09 Jan 2019 07:05:06 PM UTC, comment #4: 


is more information about this required for now?

Sun 30 Dec 2018 12:34:16 PM UTC, comment #3: 

The problem can now be reproduced. We established a new branch within the GitHub project https://github.com/DECALIOC/DEcalioc

Please make sure you clone the "debug" branch.

Your help is greatly appreciated!

Sun 30 Dec 2018 09:41:18 AM UTC, comment #2: 

Thank you very much for your reply! I will try to provide a minimal example, but it will take quite a while to build the helper functions to replace the simulation model.

Regarding your questions:

parcellfun is called in line 94 of DEcalioc.m. A file with the variables is attached (variables.mat).

[results_all{k}] = parcellfun(fix(Input.maxCPU/Input.cpu(k)), anonym_Start, runs_all, 'VerboseLevel', 0);

I suggest to have a look at the variable "runs_all", because we suspect its structure is causing the issue, even though it did not seem to be a problem with parallel 3.1.1.

(file #45785)

Fri 28 Dec 2018 03:27:42 PM UTC, comment #1: 

I can't guess from your report what's happening (and I can't afford the time for going on with guessing). Please provide a minimal example for triggering this error. If the called user code should be too difficult to reduce in size to be attached here, you could tell us, as a possible start, at least the exact command line of the call to parcellfun, with the values of all arguments, possibly within an attached data file. Also, you could tell the number and dimensions of output of your user code. This could give a clue even without having the called user code available, but the most effective way to find the cause is by reproducing the bug.

Olaf Till <i7tiol>
Project MemberIn charge of this item.
Thu 27 Dec 2018 01:02:34 PM UTC, original submission:  


we am maintaining a code package for simulation parameter calibration on GitHub (https://github.com/DECALIOC/DEcalioc).


We have to run a set of parameters on a model. We want run multiple processes in parallel, where each process could require more than one CPU core. The function we call with parcellfun calls a batch script for each process (which needs 1 to N CPU cores) and also collects the results to feed them back to parcellfun.


The code used to run under parallel version 3.1.1. However, with version 3.1.3, there appear to have been made changes that cause our code to fail with the error:

"can't reshape 2x21 array to 1x13 array
parcellfun at line 452 column 21
DEcalioc at line 94 column 18"

I have attached the state after the code fails in a MAT file, where "runs_all" does not seem to match up with the result cell returned from parcellfun.


For different ncpu values, the error message changes:
ncpu = 8

error: reshape: can't reshape 2x21 array to 1x13 array
error: called from
    parcellfun at line 452 column 21
    DEcalioc at line 94 column 18

ncpu = 4

error: reshape: can't reshape 2x21 array to 1x5 array
error: called from
    parcellfun at line 452 column 21
    DEcalioc at line 94 column 18

ncpu = 1

parcellfun: 1/1 jobs done
parcellfun: waiting for child process 13621 to close
error: reshape: can't reshape 2x21 array to 1x1 array
error: called from
    parcellfun at line 452 column 21
    DEcalioc at line 94 column 18


Octave 4.2.2
Ubuntu 18.04.1 LTS

>> pkg list

Package Name  | Version | Installation directory
          io *|  2.4.12 | /home/michael/octave/io-2.4.12
       optim *|   1.5.3 | /home/michael/octave/optim-1.5.3
    parallel *|   3.1.3 | /home/michael/octave/parallel-3.1.3
  statistics *|   1.4.0 | /home/michael/octave/statistics-1.4.0
         stk *|   2.5.1 | /home/michael/octave/stk-2.5.1
      struct *|  1.0.15 | /home/michael/octave/struct-1.0.15



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #45785:  variables.mat added by None (3KiB - application/octet-stream)
file #45747:  Files.zip added by None (2KiB - application/zip)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by i7tiol (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-20 i7tiol StatusNeed Info => Invalid
        Open/ClosedOpen => Closed
    2018-12-30 None Attached File- => Added variables.mat, #45785
    2018-12-29 mtmiller SummaryPackage "parallel": Issue with parcellfun after update => [octave forge] (parallel) Issue with parcellfun after update
    2018-12-28 i7tiol StatusNone => Need Info
        Assigned toNone => i7tiol
    2018-12-27 None Attached File- => Added Files.zip, #45747

    Back to the top

    Powered by Savane 3.5