bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #37766, Combining --pipe and --shebang...

 
 

bug #37766: Combining --pipe and --shebang options

Submitted by:  Ole Tange <tange>
Submitted on:  Sun 18 Nov 2012 02:08:18 PM UTC  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 24 Nov 2012 04:37:19 PM UTC, comment #3:

Fixed in [eb3b6fb00]

> We should allow for putting options on the command interpreter. E.g:
>
> #!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p
>
> Also without the --pipe the {} should work as expected:
>
> #!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p {}
> {.}.out > {.}.log


This, however, does not work. The above should run:

/usr/bin/parallel -k -j24 /usr/bin/perl -p my_program "{} {.}.out > {.}.log"

The problem is that the script file name needs to be put into the
middle of the command line (above it is put after -p). As I do not see
a way to find right place to put in the script file name, I do not see
a way to get that part working.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 20 Nov 2012 10:47:47 AM UTC, comment #2:

What you want is to pass the rest of the file to python and let
parallel chunk up stdin to the python script.

I really like your idea, but it is clearly not a bug that it does not
work currently.

Your idea would be useful for any script (Shell, Perl, Python) which
can either process only one file or process stuff from stdin. Using
GNU Parallel your script can suddenly process many files/blocks of
data and in parallel.

I do not see we can change the behaviour of --shebang, but we can
invent a new option.

So it could be something like:

#!/usr/bin/parallel --shebang-program --pipe -k -j24 /usr/bin/python

(Please come up with a better name than --shebang-program)

This should accept data on stdin which should be chunked and passed to
the python program. So the program will be called like:

cat foo bar | my_program

or:

my_program foo bar

Without the --pipe:

#!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/python

should work like:

parallel -k -j24 /usr/bin/python my_program {}

So:

my_program foo bar
(echo foo; echo bar) | my_program

should do the same as:

parallel -k -j24 /usr/bin/python my_program {} ::: foo bar

We should allow for putting options on the command interpreter. E.g:

#!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p

Also without the --pipe the {} should work as expected:

#!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p {}
{.}.out > {.}.log

Are there things I am not covering? Are there use cases that this will
not cover?

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sun 18 Nov 2012 07:45:05 PM UTC, comment #1:

#!/usr/local/bin/parallel --shebang --pipe -k -j24 cat
a
b
c
d

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sun 18 Nov 2012 02:08:18 PM UTC, original submission:

#!/usr/bin/parallel --shebang --pipe -k -j24 /usr/bin/python

import sys

if len(sys.argv) > 1:
MIN_FREQ = float(sys.argv[1])
else:
MIN_FREQ = 10

infile = sys.stdin
outfile = sys.stdout

for line in infile:
cols = line.split('\t')
if float(cols[1])>= MIN_FREQ:
outfile.write(line)

Works:
paste <(seq 11 20) <(seq 10) | parallel --pipe -k -j24 /usr/bin/python tmp/example

Should work but does not:
paste <(seq 11 20) <(seq 10) | tmp/example

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 24 Nov 2012 04:37:19 PM UTCtangeStatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Tue 20 Nov 2012 10:47:47 AM UTCtangeSeverity2 - Minor=>1 - Wish
    Sun 18 Nov 2012 02:08:43 PM UTCtangeSeverity3 - Normal=>2 - Minor
      StatusNone=>Confirmed
      Assigned toNone=>tange

    Back to the top


    Powered by Savane 3.1-cleanup