bugfindutils - Bugs: bug #29512, [feature request] xargs parallel...


bug #29512: [feature request] xargs parallel processing env-val improvement

Submitted by:  None
Submitted on:  Sat Apr 10 15:45:43 2010  
Category: xargsSeverity: 1 - Wish
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: James Youngman <jay>
Originator Name: Tomas MudrunkaOriginator Email: -unavailable-
Open/Closed: ClosedRelease: None
Fixed Release: 4.5.11

Add a New Comment (Rich MarkupRich Markup):

You are not logged in

Please log in, so followups can be emailed to you.


(Jump to the original submission Jump to the original submission)

Sun May 15 17:32:28 2011, comment #17:

Committed to the git repository.

James Youngman <jay>
Project AdministratorIn charge of this item.
Wed May 11 18:52:39 2011, comment #16:

Cool, go for it :) thx

Wed May 11 11:26:58 2011, comment #15:

Again, I've updated the patch to target the current head revision.

If you're happy with the updated version (you can try the patch against release 4.5.10) I can merge it.

(file #23388)

James Youngman <jay>
Project AdministratorIn charge of this item.
Sun Aug 15 12:06:19 2010, comment #14:

I've updated the patch to target the current head revision.

(file #21224)

James Youngman <jay>
Project AdministratorIn charge of this item.
Mon Jul 26 12:09:20 2010, comment #13:

I believe you mean that it conflicts with 'moreutils', not 'more' - at least on Fedora, parallel(1) is provided by the moreutils package. Maybe there is some room for merging the two implementations of parallel(1)? At any rate, that is a side issue, compared to whether the findutils package should add more parallel support in xargs(1).

Eric Blake <ericb>
Project Administrator
Mon Jul 26 11:15:40 2010, comment #12:

Tange: well parallel seems to not be very popular for distribution maintainers. my distribution even does not have package with parralel and they're saying something about conflicts with gnu more.

i think this is usefull patch as xargs are (almost) everywhere.

Mon Jul 26 06:42:53 2010, comment #11:

It seems to be an awful lot of hoops to jump through when you could just do the following using GNU Parallel http://www.gnu.org/software/parallel/:

command | parallel -j+0 -S server1,server2,: process

This would run number_of_cpu_cores jobs on server1, server2, and the local computer. If GNU Parallel is installed on server1 and server2 it will automatically detect the number of cpu cores there.

Ole Tange <tange>
Mon Apr 12 11:07:49 2010, comment #10:

now it's easy to map any values to slots, eg:
seq 0 25 |./xargs -P3 -n3 --process-slot-var=XARGS_SLOT bash -c 'echo [$XARGS_SLOT] $(echo "ssh a;ssh b;ssh c" | cut -d ";" -f $[ $XARGS_SLOT + 1] ); sleep 0.5' x

Mon Apr 12 10:53:59 2010, comment #9:

Cool, thanks :-)
Will it be merged to the git?

  1. ;) harvie@harvie-ntb xargs $ seq 0 25 |./xargs -P3 -n3 --process-slot-var=XARGS_SLOT bash -c 'echo [$XARGS_SLOT] $@; sleep 0.5' x

[0] 0 1 2
[1] 3 4 5
[2] 6 7 8
[0] 9 10 11
[1] 12 13 14
[2] 15 16 17
[0] 18 19 20
[1] 21 22 23
[2] 24 25

Sun Apr 11 22:28:06 2010, comment #8:

Try the attached patch.

$ seq 0 25 |./xargs/xargs -P3 -n3 --process-slot-var=XARGS_SLOT bash -c 'echo [$XARGS_SLOT] $@; sleep 2' x
[2] 6 7 8
[1] 3 4 5
[0] 0 1 2
[1] 12 13 14
[2] 9 10 11
[0] 15 16 17
[1] 18 19 20
[2] 21 22 23
[0] 24 25

(file #20196)

James Youngman <jay>
Project AdministratorIn charge of this item.
Sun Apr 11 19:17:59 2010, comment #7:

>> By default though, xargs only runs one process at a time, so the output would look like this:
>> ...

pseudocode :-)
but i think you understanded what i mean ;-)

Sun Apr 11 19:15:39 2010, comment #6:

i've checked the source.
there is pids[i] array (so i want to know i value).

it's not such hard... there are two ways to pass the value...

1.) use
int execle(const char path, const char arg, char * const envp[]);
instead of
int execvp(const char file, char const argv[]);

2.) (i think this one is quicker)
call setenv() right at start of xargs and after fork just edit allocated env using and snprintf(getenv(), "%d", i);

but in fact i don't know how to get i after fork(), so i can't make patch...

Sun Apr 11 19:12:14 2010, comment #5:

By default though, xargs only runs one process at a time, so the output would look like this:

seq 0 99 | xargs bash -c 'echo $XARGS_SLOT'

James Youngman <jay>
Project AdministratorIn charge of this item.
Sun Apr 11 19:00:24 2010, comment #4:

>> If yes, why not use the PID of the child?

because PID is different for each process. when you will pass me xargs "slot", then i will run 400 tasks on 4 CPUs, the slot value will be from 1 to 4 only which is usefull.

i think you are still not understand me. i am speaking about something like this:

seq 0 99 | xargs bash -c 'echo $XARGS_SLOT'

so for example i will be able to run processes on slots 1 and 2 and the others pass somewhere else.

at this time you already know this value in xargs and all i want is to set it to envinronment :-) you know that you can run only N processes, so i guess that you have some array with filedescriptors or PIDs of those processes and you are waiting until some index of this array will exit, so you will put another process to this array. all i want is to know current index of my process in this array.

Sun Apr 11 17:06:47 2010, comment #3:

Is this the same thing as you are asking for:

"Set some environment variable, so that it has a different value in each child process."

If yes, why not use the PID of the child?

Even if we use a sequence starting at 1 and put the value in $XARGS_CPU, xargs cannot guarantee for example that a process with XARGS_CPU=8 can't usefully run locally, since the earlier processes with XARGS_CPU=2 up to XARGS_CPU=7 may already have finished. Hence knowing that $XARGS_CPU > (the number of CPUs) doesn't seem helpful.

James Youngman <jay>
Project AdministratorIn charge of this item.
Sun Apr 11 16:58:07 2010, comment #2:

OK i have solution which is more (very) simple to implement and which will be OK also...

just set ENV value XARGS_CPU which will contain number of proccess slot on which is the process running. and i will be able eg. to handle numbers higher than my CPU count on cluster or at least i will be able to let user know on which slot is which process running.

do you understand my weird english? ;-)

Sat Apr 10 20:53:13 2010, comment #1:

I believe I understand your use case. However, whichever way you slice it, xargs is the wrong tool for this job.

At the bottom end, this kind of job is best solved with a site- or task- specific script (or for trivial cases, with ssh(1), batch(1) and taskset(1)). At the top end, you need a much more sophisticated cluster management system (because as N gets large, you start needing to worry about machine replacement, storage bandwidth, workload repartitioning, etc.)

xargs just isn't going to evolve into anything resembling a job queueing system, let alone a cluster-management system.

For the use cases you have outlined, I'd suggest you make use of some purpose-built scripts. If such scripts are made very unwieldy by xargs' lack of a feature that is simple and yet likely to be generally useful, please go ahead and suggest a specific change.

James Youngman <jay>
Project AdministratorIn charge of this item.
Sat Apr 10 15:45:43 2010, original submission:

It would be usefull to have ability to launch xargs -P with different envinronment value for each N processes. (eg. 2 with blank, 4 with something in this env and 4 more with something different).

Nowadays i can run some command parralely on multiple CPU cores like this:

command | xargs -n 1 -P "$cpus" some_command --

but i am thinking about processing on whole cluster of computers with sshds, imagine that xxd command is VERY VERY CPU intensive and i will do something like this:

  1. ;) harvie@harvie-ntb Shared $ DPREFIX=''
  2. ;) harvie@harvie-ntb Shared $ echo lol | $DPREFIX xxd

0000000: 6c6f 6c0a lol.

  1. ;) harvie@harvie-ntb Shared $ DPREFIX='ssh harvie.cz'
  2. ;) harvie@harvie-ntb Shared $ echo lol | $DPREFIX xxd

Host key fingerprint is 41:72:28:3d:f5:f5:d3:a2:0b:f6:e5:c7:a2:c4:b0:d3

0000000: 6c6f 6c0a lol.

so i can simply trigger if the process will be launched localy, or remotely. now imagine bit more complex example:

cpus='2;4=ssh node1.harvie.cz;4=ssh node2.harvie.cz';
#now xargs will run 10 processes total.
#2 with blank DPREFIX env value
#+4 with DPREFIX='ssh node1.harvie.cz'
#+4 with DPREFIX='ssh node2.harvie.cz'
#you need to have public key copied on those nodes or something...

process() {
cat $1 | $DPREFIX xxd > $2
#handle failed ssh connection here
export -f process;
command | xargs -0 -n 1 -P "$cpus" bash -c 'process "$@"' --

So you will be able to run xargs on top of network cluster... Maybe this does not fit all of possible situations, but i think sometimes it can be really usefull.

this is just IMHO most cool of them, but there are many other ways how to make use of such feature...
eg.: you can use 4 cores, but run with bigger nice value on one of them, so there will be some CPU power left if you will want to work on something else.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

Attached Files


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by ericb (Posted a comment)
  • -unavailable- added by tange (Posted a comment)
  • -unavailable- added by jay (Posted a comment)
  • -unavailable- added by None (Submitted the item)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 12 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat Feb 2 14:56:16 2013jayOpen/ClosedOpen=>Closed
      Fixed ReleaseNone=>4.5.11
    Sun May 15 17:32:28 2011jayStatusCode Review=>Fixed
    Wed May 11 11:26:58 2011jayAttached File-=>Added 0001-Implement-xargs-process-slot-var.patch, #23388
    Wed May 11 11:25:16 2011jayAttached File#21224=>Removed
    Sun Aug 15 12:06:19 2010jayAttached File-=>Added 0001-Implement-xargs-process-slot-var.patch, #21224
    Sun Aug 15 12:05:47 2010jayAttached File#20196=>Removed
    Sun Apr 11 22:28:16 2010jayStatusNeed Info=>Code Review
    Sun Apr 11 22:28:06 2010jayAttached File-=>Added 0001-Implement-xargs-process-slot-var.patch, #20196
    Sun Apr 11 12:50:18 2010jayStatusNone=>Need Info
    Sat Apr 10 20:53:13 2010jaySeverity3 - Normal=>1 - Wish
      Assigned toNone=>jay

    Back to the top

    Powered by Savane 3.1-cleanup1