bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #47997, --pipepart should default --block...

 
 

bug #47997: --pipepart should default --block to something good

Submitted by:  Ole Tange <tange>
Submitted on:  Tue 24 May 2016 08:43:50 AM UTC  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 26 May 2016 07:21:41 PM UTC, comment #3:

Fixed in [51f212e54].

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 24 May 2016 01:13:52 PM UTC, comment #2:

Find size of file.
Compute number of job slots (maybe by adding dummy jobs?)
Compute --block

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 24 May 2016 09:03:59 AM UTC, comment #1:

What is the optimal, if the program has a long startup time?

(sleep 10; cat)

What is the optimal, if the program is slow/fast?

Can you compute the optimal?
Startuptime, records/second, finishing time.

First version can assume startuptime/finishingtime = 0

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 24 May 2016 08:43:50 AM UTC, original submission:

If you know the input size, then you can give a good suggestion for --block to give the optimal runtime.

It should be less than 1 job per CPU in total and bigger than 100 jobs per CPU in total.

Based on different file types figure out what the optimal size would be and let that be the default for --block when using --pipepart.

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 26 May 2016 07:21:41 PM UTCtangeStatusNone=>Fixed
      Assigned toNone=>tange
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1