bugGNU Parallel - Bugs: bug #37506, (de)multiplex

 
 

bug #37506: (de)multiplex

Submitter:  Ole Tange <tange>
Submitted:  Thu 04 Oct 2012 08:37:43 PM UTC
   
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 01 Dec 2013 04:47:10 PM UTC, comment #3: 

--round-robin is a good-enough solution to this.

Ole Tange <tange>
Group administrator
Wed 05 Dec 2012 09:30:39 PM UTC, comment #2: 

 split -n r/8 --filter='grep -a ole'

Ole Tange <tange>
Group administrator
Tue 23 Oct 2012 04:10:31 PM UTC, comment #1: 

GNU split can do something using --filter

Ole Tange <tange>
Group administrator
Thu 04 Oct 2012 08:37:43 PM UTC, original submission:  

As a companion for GNU Parallel it would be good to have a program that could spread stdin:

cat | demultiplex --recstart FOO --recend \n fifo1 fifo2 fifo3

This should write stdin one record at a time to the 3 fifos in parallel. It should do so non-blocking so that if fifo2 is reading much faster than fifo1 and fifo3 then fifo2 will get more of the data.

multiplex --recstart FOO --recend \n fifo1 fifo2 fifo3 | cat

This should work similar to 'cat', but read non-blocking from each of the fifos (using select). When one or more full records are read they are sent to stdout. This continues until EOF on all 3 fifos.

As performance is key we need to look at C++. Maybe using Inline with a fallback to Perl.

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-12-01 tange StatusNone Fixed
        Assigned toNone tange
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code