bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #37325, Inefficiency of --pipe -L

 
 

bug #37325: Inefficiency of --pipe -L

Submitted by:  Ole Tange <tange>
Submitted on:  Tue 11 Sep 2012 08:39:02 AM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 25 Nov 2012 11:20:36 PM UTC, comment #1:

Fixed in [6921d28]

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 11 Sep 2012 08:39:02 AM UTC, original submission:

The NEWS file mentions that '-L n --pipe' is useful for fastq. I tried this, and it works, but extremely slowly. On the other hand, using '-N' instead of '-L' works and is much faster.

I'm using GNU parallel 20120822. Here is how to reproduce the problem:

seq 100000 > fake.fastq

# This works and is fast:
time parallel --pipe -N10000 cat < fake.fastq > out1

# These work but are extremely slow:
time parallel --pipe -L10000 cat < fake.fastq > out2a
time parallel --pipe -L4 cat < fake.fastq > out2b
time parallel --pipe -L4 --block 1M cat < fake.fastq > out2c

(As you can see, I'm also not certain how -L is supposed to be used.)

I hope this is useful,
Martin Frith
http://www.cbrc.jp/~martin/

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 25 Nov 2012 11:20:36 PM UTCtangeStatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup