bugGNU Parallel - Bugs: bug #42365, --pipepart and --roundrobin does...

 
 

bug #42365: --pipepart and --roundrobin does not round robin

Submitter:  Ole Tange <tange>
Submitted:  Thu 15 May 2014 12:06:00 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 06 Oct 2015 03:40:10 PM UTC, comment #3: 

Fixed in [18ab03c98da3].

Anonymous
Wed 04 Jun 2014 01:02:35 PM UTC, comment #2: 

To find the --round-robin block size we need to know the number of jobs requested. This gives a chicken-egg-problem as the number of jobs from --pipepart is used to compute the number of jobslots.

Can we somehow push loads of dummyjobs on the queue, compute number of jobslots, remove the dummyjobs, find the blocksize and return the jobs?

Ole Tange <tange>
Group administrator
Sat 31 May 2014 06:50:14 AM UTC, comment #1: 

Why would you ever want to do --round-robin with --pipe-part?

The goal with --round-robin is to minimize the number of processing jobs - typically because startup is expensive.

So with --pipepart we could simply let --round-robin mean: Based on number of jobs and size of file, compute the optimal --block-size so each processing job gets an equal share.



Ole Tange <tange>
Group administrator
Thu 15 May 2014 12:06:00 AM UTC, original submission:  

parallel --pipepart --block 3 -a file -t --round-robin wc

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-10-18 tange StatusNone Fixed
        Assigned toNone tange
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code