bugGNU Parallel - Bugs: bug #37766, Combining --pipe and --shebang...

 
 

bug #37766: Combining --pipe and --shebang options

Submitter:  Ole Tange <tange>
Submitted:  Sun 18 Nov 2012 02:08:18 PM UTC
   
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 24 Nov 2012 04:37:19 PM UTC, comment #3: 

Fixed in [eb3b6fb00]

> We should allow for putting options on the command interpreter. E.g:
>
> #!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p
>
> Also without the --pipe the {} should work as expected:
>
> #!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p {}
> {.}.out > {.}.log


This, however, does not work. The above should run:

/usr/bin/parallel -k -j24 /usr/bin/perl -p my_program "{} {.}.out > {.}.log"

The problem is that the script file name needs to be put into the
middle of the command line (above it is put after -p). As I do not see
a way to find right place to put in the script file name, I do not see
a way to get that part working.

Ole Tange <tange>
Group administrator
Tue 20 Nov 2012 10:47:47 AM UTC, comment #2: 

What you want is to pass the rest of the file to python and let
parallel chunk up stdin to the python script.

I really like your idea, but it is clearly not a bug that it does not
work currently.

Your idea would be useful for any script (Shell, Perl, Python) which
can either process only one file or process stuff from stdin. Using
GNU Parallel your script can suddenly process many files/blocks of
data and in parallel.

I do not see we can change the behaviour of --shebang, but we can
invent a new option.

So it could be something like:

#!/usr/bin/parallel --shebang-program --pipe -k -j24 /usr/bin/python

(Please come up with a better name than  --shebang-program)

This should accept data on stdin which should be chunked and passed to
the python program. So the program will be called like:

  cat foo bar | my_program

or:

  my_program foo bar

Without the --pipe:

#!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/python

should work like:

  parallel -k -j24 /usr/bin/python my_program {}

So:

  my_program foo bar
  (echo foo; echo bar) | my_program

should do the same as:

  parallel -k -j24 /usr/bin/python my_program {} ::: foo bar

We should allow for putting options on the command interpreter. E.g:

#!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p

Also without the --pipe the {} should work as expected:

#!/usr/bin/parallel --shebang-program -k -j24 /usr/bin/perl -p {}
{.}.out > {.}.log

Are there things I am not covering? Are there use cases that this will
not cover?

Ole Tange <tange>
Group administrator
Sun 18 Nov 2012 07:45:05 PM UTC, comment #1: 

#!/usr/local/bin/parallel --shebang --pipe -k -j24 cat
a
b
c
d


Ole Tange <tange>
Group administrator
Sun 18 Nov 2012 02:08:18 PM UTC, original submission:  


#!/usr/bin/parallel --shebang --pipe -k -j24 /usr/bin/python

import sys

if len(sys.argv) > 1:
    MIN_FREQ = float(sys.argv[1])
else:
    MIN_FREQ = 10

infile = sys.stdin
outfile = sys.stdout

for line in infile:
    cols = line.split('\t')
    if float(cols[1])>= MIN_FREQ:
        outfile.write(line)

Works:
paste <(seq 11 20) <(seq 10) | parallel --pipe -k -j24 /usr/bin/python tmp/example

Should work but does not:
paste <(seq 11 20) <(seq 10) | tmp/example

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-11-24 tange StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2012-11-20 tange Severity2 - Minor 1 - Wish
    2012-11-18 tange Severity3 - Normal 2 - Minor
        StatusNone Confirmed
        Assigned toNone tange

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code