bugGNU Parallel - Bugs: bug #59666, ... | parallel --pipe -N1 foo :::...

 
 

bug #59666: ... | parallel --pipe -N1 foo ::: a b c should pass input to all jobs

Submitted by:  Ole Tange <tange>
Submitted on:  Sat 12 Dec 2020 12:09:49 PM UTC  
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  In Progress
Privacy:  Public Assigned to:  Ole Tange <tange>
Open/Closed:  Open

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 12 Dec 2020 05:16:00 PM UTC, comment #1: 

I have had the situation where I have data on a pipe that I want to give to `foo a` and `foo b` one line at a time. They should both receive all input.

I currently do:

  foo() { echo "$@"; cat; }
  foorun() { parallel --pipe -N1 foo "$@"; }
  export -f foorun foo

  seq 3 | parallel --pipe --tee foorun {} ::: a b
  seq 3 | parallel --pipe --tee foorun {} ::: a b :::+ c d
  seq 3 | parallel --pipe --tee foorun {} ::: a b ::: c d

I feel a bit silly having to build foorun(). So I have the feeling this ought do the same (it does not):

  seq 3 | parallel --pipe -N1 foo ::: a b
  seq 3 | parallel --pipe -N1 foo ::: a b
  seq 3 | parallel --pipe -N1 foo ::: a b :::+ c d
  seq 3 | parallel --pipe -N1 foo ::: a b ::: c d

The current behaviour works more like --xapply, so it should only work if you use :::+ or --xapply (this already works):

  seq 3 | parallel --pipe -N1 foo :::+ a b c

As a first version I intend to simply let GNU Parallel call parallel (very much as above). That means that sequence number and slot number will not give the correct values (and probably some other limitations). Similar to how --(n)onall works right now.

If this works, a later version will read block n from stdin. Add a(n) .. z(n) to the queue. When the queue is empty, read block n+1. Repeat.

So even if, say, c(x) is always fast, GNU Parallel will at most keep one block in memory per jobslot.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 12 Dec 2020 12:09:49 PM UTC, original submission:  

It should give this:

   $ seq 3 | parallel --pipe -N1 'echo {};cat' ::: a b
   a
   1
   b
   1
   a
   2
   b
   2
   a
   3
   b
   3

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-12-12 tange Severity3 - Normal => 1 - Wish
        StatusNone => In Progress
        Assigned toNone => tange

    Back to the top


    Powered by Savane 3.7