bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #39814, Feature request: --pipe with :::

 
 

bug #39814: Feature request: --pipe with :::

Submitted by:  Ole Tange <tange>
Submitted on:  Mon 19 Aug 2013 09:07:27 AM UTC  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: None
Privacy: PublicAssigned to: None
Open/Closed: Open

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 19 Aug 2013 09:07:27 AM UTC, original submission:

It never makes sense to have --pipe read from the command line. It will always be a pipe or from a file.

And it is a minimal restriction to have it only read from STDIN (i.e. -a will not work).

By making that restriction you could have commands like:

seq 3 | parallel --pipe 'echo {}; cat' ::: A B C

The idea here being that the same input will be sent to all three commands. So it will run:

seq 3 | parallel --pipe 'tee >(echo A; cat) >(echo B; cat) >(echo C; cat) >/dev/null'

but with normal --group buffering. Maybe like:

seq 3 | parallel --pipe 'tee >((echo A; cat)>/tmp/buffer1) >((echo B; cat)>/tmp/buffer2) >((echo C; cat)>/tmp/buffer3) >/dev/null'

There are several limiting problems:

  • Command line size: Too many arguments or combinations will make the command line too big
  • Number of processes: Too many arguments or combinations will spawn too many processes.
  • Getting the exit code back if one of the processes dies. The total exit code should be of the first that failed.
  • Cleanup will not be done if ctrl-C.

But if implemented it would make a pretty solution to:
http://stackoverflow.com/questions/1570328/run-few-commands-simultaneously

find ./incoming/kontraktor/ -type f -name '*.html' | sort |
parallel "awk 'NR % 3 == {}' | ./bin/foo.py -m 3 -b 3 | next_command" ::: 1 2 0 >> log/foo_log.log 2>&1

Ole Tange <tange>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    No Changes Have Been Made to This Item

    Back to the top


    Powered by Savane 3.1-cleanup