bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #42365, --pipepart and --roundrobin does...

 
 

bug #42365: --pipepart and --roundrobin does not round robin

Submitted by:  Ole Tange <tange>
Submitted on:  Thu 15 May 2014 12:06:00 AM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 06 Oct 2015 03:40:10 PM UTC, comment #3:

Fixed in [18ab03c98da3].

Anonymous
Wed 04 Jun 2014 01:02:35 PM UTC, comment #2:

To find the --round-robin block size we need to know the number of jobs requested. This gives a chicken-egg-problem as the number of jobs from --pipepart is used to compute the number of jobslots.

Can we somehow push loads of dummyjobs on the queue, compute number of jobslots, remove the dummyjobs, find the blocksize and return the jobs?

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 31 May 2014 06:50:14 AM UTC, comment #1:

Why would you ever want to do --round-robin with --pipe-part?

The goal with --round-robin is to minimize the number of processing jobs - typically because startup is expensive.

So with --pipepart we could simply let --round-robin mean: Based on number of jobs and size of file, compute the optimal --block-size so each processing job gets an equal share.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 15 May 2014 12:06:00 AM UTC, original submission:

parallel --pipepart --block 3 -a file -t --round-robin wc

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 18 Oct 2015 03:37:03 PM UTCtangeStatusNone=>Fixed
      Assigned toNone=>tange
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1