Fri 05 Sep 2014 07:33:57 PM UTC, comment #2:
Thanks for the response. I'm still trying to get my head completely around it. There are more tradeoffs here than I'd realized.
In explaining "the reason for this behaviour," you say "when there are not enough arguments to keep all CPUs busy: then you prefer not having idle CPUs." But here Parallel produces a set of commands that leaves idle CPUs. I could understand if we were optimizing for "as few commands to be run as possible with as many args," but we don't get that, either. Nor do we even get equitable distribution of arguments among the commands that we do run.
In short, your explanation has helped support my understanding of the implementation, but I'm still fuzzy on why that implementation is desired.
My working theory is something like this: If we can't pass all the args to a single command, then we have to be concerned that we can't pass them all to 12 commands, either. So why not try to build all 12 commands first to see whether it works? My guess is that the reason is about complexity, perhaps conflicts with the demands of other features. I had thought that perhaps the goal was to avoid having to read in the full input before starting, but as far as I can tell, Parallel already does this. My next best theory is that the goal is to avoid delaying the start of the first task for long, though I would hope that it's not too long, since we're basically just processing every argument a second time.
Additionally, I'm a bit confused by the --xargs note. As far as I can tell, passing --xargs has no effect on either of my two examples. I thought perhaps that this might mean that I needed to use it instead of -m rather than in addition, but doing so left me further confused:
$ /usr/bin/parallel --gnu -j 12 --xargs echo ::: {1..24}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ /usr/bin/parallel -s 50 --gnu -j 12 --xargs echo ::: {1..24}
<same>
But maybe I should back up from all this and lay out what I'm really trying to do. I would like to distribute arguments evenly to 12 commands. Perhaps this doesn't call for using Parallel at all, but I like that it can handle output interleaving, divide into more than 12 commands if I go over the command-line limit, and probably address other problems that I would never even think of. Perhaps I have overlooked the flags that cover this case?
|