bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #38196, --pipe -L not as parallel as...

 
 

bug #38196: --pipe -L not as parallel as expected

Submitted by:  None
Submitted on:  Tue 29 Jan 2013 04:05:17 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Invalid
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 07 Feb 2013 10:18:56 AM UTC, comment #1:

This is not a bug:

-L1 means a record will be one line, so GNU Parallel will read as many records as it can fit in --block (which defaults to 1 megabyte).

To read a single line at a time do:

seq 10000 | parallel --pipe -L1 -n1 -j24 sleep 10

which means:

-L1 - one line is a record

-n1 - read at most one record

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 29 Jan 2013 04:05:17 PM UTC, original submission:

This is a resubmission of #38110 (--pipe not very parallel) but with what I now believe is the actual bug (or source of my confusion if this too is not a bug).

Before commit bbc9a53ea5fc746a0a3ef7e8733dfcd61ff05eea

seq 10000 parallel --pipe -L1 -j24 sleep 10

would spawn 24 processes as counted by something like

pgrep sleep | wc

After this commit, it only spawns one process. You can still spawn 24 processes with

seq 10000 parallel --pipe --block-size 1 -L1 -j24 sleep 10

but this is contrary to my intuition as to what -L should do. If -L is in fact supposed to behave this way, clarification in the manual would be helpful.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 07 Feb 2013 10:18:56 AM UTCtangeStatusNone=>Invalid
      Assigned toNone=>tange
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup