bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #31678, -m and -X should try to divide the...

 
 

bug #31678: -m and -X should try to divide the jobs

Submitted by:  Ole Tange <tange>
Submitted on:  Tue 16 Nov 2010 11:09:09 PM UTC  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: None
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 02 Jan 2011 09:11:36 PM UTC, comment #3:

Implemented in [57cf4df2b]:

Distribute args between jobslots for -X when hitting EOF (the simple version described earlier).

Released as alpha version.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 07 Dec 2010 08:21:57 AM UTC, comment #2:

To do either solution the arg-getter must know #jobslots. Maybe have a global variable that is corrected every time SSHLogin->{max_jobs_running} is changed.

What should these do? Assume one cmd line = 10000 args.

#args = 10 -j8 => Run 2jobs on 2cpus and 1job on 6cpu
#args = 100 -j8 => Run 13jobs on 4cpus and 12job on 4cpu
#args = 10000 -j8 => Run 1250jobs on 8cpus
Complex: #args = 10008 -j8 => Run 1251jobs on 8cpus
Simple: #args = 10008 -j8 => Run 2jobs on 1cpu 1jobs on 7cpus

processes_available_by_system_limit depends on #args, so redistribution should also be incorporated here.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Dec 2010 08:35:12 AM UTC, comment #1:

The hard solution:

Read arguments until eof or all jobslots full of arguments.

If eof: Redistribute the jobs.

A simpler solution may be:

Read arguments until eof or this jobslot is full of arguments.

If eof: Set maxargs = (#args/#jobslots)+1. Unget args and get them again.

The only situation where the simpler is less optimal is if there are so many args that they fit more than one but less than #jobslots lines.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 16 Nov 2010 11:09:09 PM UTC, original submission:

-j +0 -m should divide all the file names across one instantiation per processor core.

This only applies if hitting end of file.

If end of file is not met after filling up one line per core, no special action is needed.

If end of file is hit the arguments should be spread around to the cores.

It will postpone starting the first job until enough arguments are read to fill all lines or eof.

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 02 Jan 2011 09:11:36 PM UTCtangeOpen/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1