bugGNU Octave - Bugs: bug #55296, [octave forge] (parallel) Issue...

 
 

bug #55296: [octave forge] (parallel) Issue with parcellfun after update

Submitter:  None
Submitted:  Thu 27 Dec 2018 01:02:34 PM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Invalid / Not an Octave Bug Assigned to:  i7tiol
Originator Name:  Michael Rackl Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 4.2.2
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 20 Jan 2019 06:04:26 PM UTC, comment #9: 

For "UniformOutput" set to true, the user function is expected to return only scalars. (This is stated in the help-text of cellfun, to which the help-text of parcellfun refers.)

I'm currently not sure if we should check in parcellfun if the returned values are scalars since I haven't yet measured the performance impact.

Sorry, but I haven't got the time to check if older versions of parcellfun "accepted" your code, or to help you with your code.

Closing this bug as invalid.

Olaf Till <i7tiol>
Group Member
Sat 19 Jan 2019 06:16:28 PM UTC, comment #8: 

Thanks for the input!

I am sorry, but I think I misunderstood the question about what anonym_Start returns. The single results we expect are scalars, but anonym_Start collects them in an array if there is more than one result. Thus, anonym_Start could return an array or a scalar, i.e., a 1x1 array.

If we set 'UniformOutput' to 'false', the code goes past the call to 'parcellfun' without error. This is not bad for a first try. (However, we run into problems thereafter because our code was not built to handle cells.)

I would be interested in what changed in 'parcellfun', because it used to be able to handle our structure. This could help us to understand what we need to change in our code.

Could you please advise?

Anonymous
Mon 14 Jan 2019 09:23:14 PM UTC, comment #7: 

Please really check it, with 'isscalar ()', for each set of arguments. The returned error is what is to be expected if a function doesn't always return scalars and "UniformOutput" is set to true (which is the default). Also, the arguments to anonym_Start differ in dimensions from call to call, which may mean that the same applies to the returned result of this function.

Olaf Till <i7tiol>
Group Member
Mon 14 Jan 2019 06:31:40 PM UTC, comment #6: 

Yes, we are expecting scalars from 'anonym_Start'.

Anonymous
Fri 11 Jan 2019 09:00:53 AM UTC, comment #5: 

Does your function 'anonym_Start' always return scalars?

Olaf Till <i7tiol>
Group Member
Wed 09 Jan 2019 07:05:06 PM UTC, comment #4: 

Hi,

is more information about this required for now?

Anonymous
Sun 30 Dec 2018 12:34:16 PM UTC, comment #3: 

The problem can now be reproduced. We established a new branch within the GitHub project https://github.com/DECALIOC/DEcalioc

Please make sure you clone the "debug" branch.

Your help is greatly appreciated!

Anonymous
Sun 30 Dec 2018 09:41:18 AM UTC, comment #2: 

Thank you very much for your reply! I will try to provide a minimal example, but it will take quite a while to build the helper functions to replace the simulation model.

Regarding your questions:

parcellfun is called in line 94 of DEcalioc.m. A file with the variables is attached (variables.mat).

[results_all{k}] = parcellfun(fix(Input.maxCPU/Input.cpu(k)), anonym_Start, runs_all, 'VerboseLevel', 0);

I suggest to have a look at the variable "runs_all", because we suspect its structure is causing the issue, even though it did not seem to be a problem with parallel 3.1.1.

(file #45785)

Anonymous
Fri 28 Dec 2018 03:27:42 PM UTC, comment #1: 

I can't guess from your report what's happening (and I can't afford the time for going on with guessing). Please provide a minimal example for triggering this error. If the called user code should be too difficult to reduce in size to be attached here, you could tell us, as a possible start, at least the exact command line of the call to parcellfun, with the values of all arguments, possibly within an attached data file. Also, you could tell the number and dimensions of output of your user code. This could give a clue even without having the called user code available, but the most effective way to find the cause is by reproducing the bug.

Olaf Till <i7tiol>
Group Member
Thu 27 Dec 2018 01:02:34 PM UTC, original submission:  

Hello,

we am maintaining a code package for simulation parameter calibration on GitHub (https://github.com/DECALIOC/DEcalioc).

PROBLEM STATEMENT

We have to run a set of parameters on a model. We want run multiple processes in parallel, where each process could require more than one CPU core. The function we call with parcellfun calls a batch script for each process (which needs 1 to N CPU cores) and also collects the results to feed them back to parcellfun.

ISSUE

The code used to run under parallel version 3.1.1. However, with version 3.1.3, there appear to have been made changes that cause our code to fail with the error:

"can't reshape 2x21 array to 1x13 array
parcellfun at line 452 column 21
DEcalioc at line 94 column 18"

I have attached the state after the code fails in a MAT file, where "runs_all" does not seem to match up with the result cell returned from parcellfun.

FINDINGS so far

For different ncpu values, the error message changes:
ncpu = 8

error: reshape: can't reshape 2x21 array to 1x13 array
error: called from
    parcellfun at line 452 column 21
    DEcalioc at line 94 column 18

ncpu = 4

error: reshape: can't reshape 2x21 array to 1x5 array
error: called from
    parcellfun at line 452 column 21
    DEcalioc at line 94 column 18

ncpu = 1

parcellfun: 1/1 jobs done
parcellfun: waiting for child process 13621 to close
error: reshape: can't reshape 2x21 array to 1x1 array
error: called from
    parcellfun at line 452 column 21
    DEcalioc at line 94 column 18



SYSTEM SPECIFICATIONS

Octave 4.2.2
Ubuntu 18.04.1 LTS
64bit

>> pkg list

Package Name  | Version | Installation directory
--------------+---------+-----------------------
          io *|  2.4.12 | /home/michael/octave/io-2.4.12
       optim *|   1.5.3 | /home/michael/octave/optim-1.5.3
    parallel *|   3.1.3 | /home/michael/octave/parallel-3.1.3
  statistics *|   1.4.0 | /home/michael/octave/statistics-1.4.0
         stk *|   2.5.1 | /home/michael/octave/stk-2.5.1
      struct *|  1.0.15 | /home/michael/octave/struct-1.0.15

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #45785:  variables.mat added by None (3KiB - application/octet-stream)
file #45747:  Files.zip added by None (2KiB - application/zip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by i7tiol (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-20 i7tiol StatusNeed Info Invalid / Not an Octave Bug
        Open/ClosedOpen Closed
    2018-12-30 None Attached File- Added variables.mat, #45785
    2018-12-29 mtmiller SummaryPackage &quot;parallel&quot;: Issue with parcellfun after update [octave forge] (parallel) Issue with parcellfun after update
    2018-12-28 i7tiol StatusNone Need Info
        Assigned toNone i7tiol
    2018-12-27 None Attached File- Added Files.zip, #45747

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code