bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #43134, -m doesn't split evenly if...

 
 

bug #43134: -m doesn't split evenly if max_length is reached

Submitted by:  Chris Povirk <cpovirk>
Submitted on:  Wed 03 Sep 2014 02:29:14 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Invalid
Privacy: PublicAssigned to: None
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 07 Sep 2014 12:05:28 PM UTC, comment #3:

The rationale behind the current behaviour is to start a job as soon as you can. If reading the arguments from stdin then arguments may trickle in, and reading more arguments than needed will slow down the starting of the command.

So when have you read enough arguments to start a command?

In the current behaviour you have enough when you have one argument more than will fit on the command line, and therefore GNU Parallel stops reading and starts the command, before reading more. GNU Parallel only reads all arguments if --eta or --bar is set:

# Here one command is run before waiting for more input followed by CTRL-D
(seq 10; cat) | parallel -Xj4 -us 20 echo

The behaviour you describe would require GNU Parallel to read a full line (minus one argument) for each jobslot before starting the first job, because if EOF was met at the second to last jobslot, then all arguments would have to be reshuffled between all the jobslots.

You should feel free to discuss on the mailing list whether this behaviour is preferable, and if people like that better, you should feel free to submit a patch that does this.

I do not feel strongly either way, so if you can convince the mailing list, then that is good enough for me. I will also be willing to spend time on implementing and documenting it if I am paid to do so.

Ole Tange <tange>
Project Administrator
Fri 05 Sep 2014 07:33:57 PM UTC, comment #2:

Thanks for the response. I'm still trying to get my head completely around it. There are more tradeoffs here than I'd realized.

In explaining "the reason for this behaviour," you say "when there are not enough arguments to keep all CPUs busy: then you prefer not having idle CPUs." But here Parallel produces a set of commands that leaves idle CPUs. I could understand if we were optimizing for "as few commands to be run as possible with as many args," but we don't get that, either. Nor do we even get equitable distribution of arguments among the commands that we do run.

In short, your explanation has helped support my understanding of the implementation, but I'm still fuzzy on why that implementation is desired.

My working theory is something like this: If we can't pass all the args to a single command, then we have to be concerned that we can't pass them all to 12 commands, either. So why not try to build all 12 commands first to see whether it works? My guess is that the reason is about complexity, perhaps conflicts with the demands of other features. I had thought that perhaps the goal was to avoid having to read in the full input before starting, but as far as I can tell, Parallel already does this. My next best theory is that the goal is to avoid delaying the start of the first task for long, though I would hope that it's not too long, since we're basically just processing every argument a second time.

Additionally, I'm a bit confused by the --xargs note. As far as I can tell, passing --xargs has no effect on either of my two examples. I thought perhaps that this might mean that I needed to use it instead of -m rather than in addition, but doing so left me further confused:

$ /usr/bin/parallel --gnu -j 12 --xargs echo ::: {1..24}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

$ /usr/bin/parallel -s 50 --gnu -j 12 --xargs echo ::: {1..24}
<same>

But maybe I should back up from all this and lay out what I'm really trying to do. I would like to distribute arguments evenly to 12 commands. Perhaps this doesn't call for using Parallel at all, but I like that it can handle output interleaving, divide into more than 12 commands if I go over the command-line limit, and probably address other problems that I would never even think of. Perhaps I have overlooked the flags that cover this case?

Chris Povirk <cpovirk>
Wed 03 Sep 2014 09:12:51 PM UTC, comment #1:

This is not a bug, but you can help by rewriting the section in the man page, so it is more clear why you see this behaviour.

-m works by reading an argument. If that fits on the command line, it is added. Then it tries again until the command line is too long.

When the command line is too long, the last element is removed, the command is run, and the process starts over.

But there is an exception to that rule: If we hit end-of-file so there are no more arguments, then all remaining arguments will be spread over the number of job slots.

You experience only the exception in your first example.

In your second example the first line is generated the regular way: GNU Parallel tries to fit 19 arguments, fails on the 19th, then it runs with just 18. After that it reads 5 more elements (which fit easily on the command line), and hits end-of-file. Because of end-of-file the last 5 are spread over the 12 job slots - giving 1 argument to 5 job slots.

- o -

The reason for this behaviour is the rationale that we expect that the startup time is pretty big, so you want as few commands to be run as possible with as many args - except when there are not enough arguments to keep all CPUs busy: then you prefer not having idle CPUs.

You can force not to activate the exception by --xargs.

Ole Tange <tange>
Project Administrator
Wed 03 Sep 2014 02:29:14 PM UTC, original submission:

With a high default max_length, things work as expected:

$ /usr/bin/parallel --gnu -j 12 -m echo ::: {1..24}
1 2
3 4
5 6
7 8
9 10
11 12
13 14
15 16
17 18
19 20
21 22
23 24

With an artificially low max_length (or, more likely in practice, a high number of args), the first command grabs as much as it can, leaving little for the other commands:

$ /usr/bin/parallel -s 50 --gnu -j 12 -m echo ::: {1..24}
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
19
20
21
22
23
24

The problem seems to be that populate() performs its -m and -X logic only if it has reached the end of its arguments.

Chris Povirk <cpovirk>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Posted a comment)
  • -unavailable- added by cpovirk (Submitted the item)
  • -unavailable- added by cpovirk
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 03 Sep 2014 09:12:51 PM UTCtangeStatusNone=>Invalid
      Open/ClosedOpen=>Closed
    Wed 03 Sep 2014 02:29:14 PM UTCcpovirkCarbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup1