bugGNU Parallel - Bugs: bug #47997, --pipepart should default --block...

 
 

bug #47997: --pipepart should default --block to something good

Submitter:  Ole Tange <tange>
Submitted:  Tue 24 May 2016 08:43:50 AM UTC
   
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 26 May 2016 07:21:41 PM UTC, comment #3: 

Fixed in [51f212e54].

Ole Tange <tange>
Group administrator
Tue 24 May 2016 01:13:52 PM UTC, comment #2: 

Find size of file.
Compute number of job slots (maybe by adding dummy jobs?)
Compute --block

Ole Tange <tange>
Group administrator
Tue 24 May 2016 09:03:59 AM UTC, comment #1: 

What is the optimal, if the program has a long startup time?

(sleep 10; cat)

What is the optimal, if the program is slow/fast?

Can you compute the optimal?
Startuptime, records/second, finishing time.

First version can assume startuptime/finishingtime = 0

Ole Tange <tange>
Group administrator
Tue 24 May 2016 08:43:50 AM UTC, original submission:  

If you know the input size, then you can give a good suggestion for --block to give the optimal runtime.

It should be less than 1 job per CPU in total and bigger than 100 jobs per CPU in total.

Based on different file types figure out what the optimal size would be and let that be the default for --block when using --pipepart.

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-05-26 tange StatusNone Fixed
        Assigned toNone tange
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code