bugGNU Parallel - Bugs: bug #59666, ... | parallel --pipe -N1 foo :::...

 
 

bug #59666: ... | parallel --pipe -N1 foo ::: a b c should pass input to all jobs

Submitter:  Ole Tange <tange>
Submitted:  Sat 12 Dec 2020 12:09:49 PM UTC
   
 
Category:  None Severity:  1 - Wish
Priority:  5 - Normal Item Group:  None
Status:  In Progress Privacy:  Public
Assigned to:  tange Open/Closed:  Open
* Mandatory Fields

Post a Comment

Add a New Comment Rich Markup
   

Discussion

Sat 12 Dec 2020 05:16:00 PM UTC, comment #1: 

I have had the situation where I have data on a pipe that I want to give to `foo a` and `foo b` one line at a time. They should both receive all input.

I currently do:

  foo() { echo "$@"; cat; }
  foorun() { parallel --pipe -N1 foo "$@"; }
  export -f foorun foo

  seq 3 | parallel --pipe --tee foorun {} ::: a b
  seq 3 | parallel --pipe --tee foorun {} ::: a b :::+ c d
  seq 3 | parallel --pipe --tee foorun {} ::: a b ::: c d

I feel a bit silly having to build foorun(). So I have the feeling this ought do the same (it does not):

  seq 3 | parallel --pipe -N1 foo ::: a b
  seq 3 | parallel --pipe -N1 foo ::: a b
  seq 3 | parallel --pipe -N1 foo ::: a b :::+ c d
  seq 3 | parallel --pipe -N1 foo ::: a b ::: c d

The current behaviour works more like --xapply, so it should only work if you use :::+ or --xapply (this already works):

  seq 3 | parallel --pipe -N1 foo :::+ a b c

As a first version I intend to simply let GNU Parallel call parallel (very much as above). That means that sequence number and slot number will not give the correct values (and probably some other limitations). Similar to how --(n)onall works right now.

If this works, a later version will read block n from stdin. Add a(n) .. z(n) to the queue. When the queue is empty, read block n+1. Repeat.

So even if, say, c(x) is always fast, GNU Parallel will at most keep one block in memory per jobslot.

Ole Tange <tange>
Group administrator
Sat 12 Dec 2020 12:09:49 PM UTC, original submission:  

It should give this:

   $ seq 3 | parallel --pipe -N1 'echo {};cat' ::: a b
   a
   1
   b
   1
   a
   2
   b
   2
   a
   3
   b
   3

Ole Tange <tange>
Group administrator

 

Attached Files

This item currently has no attached files.

(Note: upload size limit is set to 4.0MiB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Dependencies

This item does not depend on any other items.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    Votes

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    History

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-12-12 tange Severity3 - Normal 1 - Wish
        StatusNone In Progress
        Assigned toNone tange

    Back to the top

    Powered by Savane 3.16-1b88.
    Corresponding source code