bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #37506, (de)multiplex


bug #37506: (de)multiplex

Submitted by:  Ole Tange <tange>
Submitted on:  Thu Oct 4 20:37:43 2012  
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):

You are not logged in

Please log in, so followups can be emailed to you.


Sun Dec 1 16:47:10 2013, comment #3:

--round-robin is a good-enough solution to this.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Wed Dec 5 21:30:39 2012, comment #2:

split -n r/8 --filter='grep -a ole'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue Oct 23 16:10:31 2012, comment #1:

GNU split can do something using --filter

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu Oct 4 20:37:43 2012, original submission:

As a companion for GNU Parallel it would be good to have a program that could spread stdin:

cat | demultiplex --recstart FOO --recend \n fifo1 fifo2 fifo3

This should write stdin one record at a time to the 3 fifos in parallel. It should do so non-blocking so that if fifo2 is reading much faster than fifo1 and fifo3 then fifo2 will get more of the data.

multiplex --recstart FOO --recend \n fifo1 fifo2 fifo3 | cat

This should work similar to 'cat', but read non-blocking from each of the fifos (using select). When one or more full records are read they are sent to stdout. This continues until EOF on all 3 fifos.

As performance is key we need to look at C++. Maybe using Inline with a fallback to Perl.

Ole Tange <tange>
Project AdministratorIn charge of this item.


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.


    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun Dec 1 16:47:10 2013tangeStatusNone=>Fixed
      Assigned toNone=>tange

    Back to the top

    Powered by Savane 3.1-cleanup1