bugGNU Parallel - Bugs: bug #59666, ... | parallel --pipe -N1 foo :::...

 
 

bug #59666: ... | parallel --pipe -N1 foo ::: a b c should pass input to all jobs

Submitter:  Ole Tange <tange>
Submitted:  Sat 12 Dec 2020 12:09:49 PM UTC
   
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  In Progress
Privacy:  Public Assigned to:  tange
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 12 Dec 2020 05:16:00 PM UTC, comment #1: 

I have had the situation where I have data on a pipe that I want to give to `foo a` and `foo b` one line at a time. They should both receive all input.

I currently do:

  foo() { echo "$@"; cat; }
  foorun() { parallel --pipe -N1 foo "$@"; }
  export -f foorun foo

  seq 3 | parallel --pipe --tee foorun {} ::: a b
  seq 3 | parallel --pipe --tee foorun {} ::: a b :::+ c d
  seq 3 | parallel --pipe --tee foorun {} ::: a b ::: c d

I feel a bit silly having to build foorun(). So I have the feeling this ought do the same (it does not):

  seq 3 | parallel --pipe -N1 foo ::: a b
  seq 3 | parallel --pipe -N1 foo ::: a b
  seq 3 | parallel --pipe -N1 foo ::: a b :::+ c d
  seq 3 | parallel --pipe -N1 foo ::: a b ::: c d

The current behaviour works more like --xapply, so it should only work if you use :::+ or --xapply (this already works):

  seq 3 | parallel --pipe -N1 foo :::+ a b c

As a first version I intend to simply let GNU Parallel call parallel (very much as above). That means that sequence number and slot number will not give the correct values (and probably some other limitations). Similar to how --(n)onall works right now.

If this works, a later version will read block n from stdin. Add a(n) .. z(n) to the queue. When the queue is empty, read block n+1. Repeat.

So even if, say, c(x) is always fast, GNU Parallel will at most keep one block in memory per jobslot.

Ole Tange <tange>
Group administrator
Sat 12 Dec 2020 12:09:49 PM UTC, original submission:  

It should give this:

   $ seq 3 | parallel --pipe -N1 'echo {};cat' ::: a b
   a
   1
   b
   1
   a
   2
   b
   2
   a
   3
   b
   3

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-12-12 tange Severity3 - Normal 1 - Wish
        StatusNone In Progress
        Assigned toNone tange

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code