bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #34241, --pipe should not spawn unneeded...

 
 

bug #34241: --pipe should not spawn unneeded processes

Submitted by:  Ole Tange <tange>
Submitted on:  Wed 07 Sep 2011 01:57:01 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 02 Jul 2013 07:50:49 PM UTC, comment #7:

Fixed before 20130630 alpha.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 09 Apr 2013 03:52:00 PM UTC, comment #6:

This generates 10 files:

seq 1000 | parallel -j10 --pipe --block 1k --files wc

But only 4 jobs are run:

seq 1000 | parallel -j10 --pipe --block 1k wc

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 27 Dec 2012 10:08:47 PM UTC, comment #5:

Fixed in [bbc9a53ea5f0]

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 27 Dec 2012 09:54:47 PM UTC, comment #4:

Test:

seq 3 | src/parallel -j30 --pipe --block-size 10 cat\;echo o

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 27 Dec 2012 09:18:50 PM UTC, comment #3:

Test if this works

eval `echo $SHELL | grep -E "/(t)?csh" > /dev/null && echo setenv _FIRST_CHAR_FILE /tmp/$$.first_char_file || echo export _FIRST_CHAR_FILE=/tmp/$$.first_char_file`

dd bs=1 count=1 of=$_FIRST_CHAR_FILE >&/dev/null; test -s "$_FIRST_CHAR_FILE" && ( cat $_FIRST_CHAR_FILE; rm $_FIRST_CHAR_FILE; cat - ) | program

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sun 06 May 2012 03:15:55 PM UTC, comment #2:

The perl -e eof wrapper does not work: STDIN is not passed on to your_command.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Fri 04 May 2012 07:09:30 PM UTC, comment #1:

One solution is to wrap the command with a test for eof:

if perl -e 'eof() and exit(1)'; then (your;command;here); else true; fi

This could be put in sshlogin_wrap

Ole Tange <tange>
Project AdministratorIn charge of this item.
Wed 07 Sep 2011 01:57:01 PM UTC, original submission:

I believe the bug can be summed up as:

echo | parallel -j2 -N1 --pipe md5sum -c

should only give one error (from md5sum) instead of 2.

echo | parallel -r -j2 -N1 --pipe md5sum -c

should give no errors.

The bug is confirmed.

Not spawning unneeded processes

Currently GNU Parallel spawns a new process when a process dies.

It will require some redesign to get it to not spawn more processes if there is no more data to process. So do not expect a quick fix for this.

/Ole

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 8 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 02 Jul 2013 07:50:49 PM UTCtangeStatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Tue 09 Apr 2013 03:52:00 PM UTCtangeStatusFixed=>Confirmed
      Open/ClosedClosed=>Open
    Thu 27 Dec 2012 10:08:47 PM UTCtangeStatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Sun 06 May 2012 03:15:55 PM UTCtangeStatusNone=>Confirmed
    Fri 30 Mar 2012 12:40:12 AM UTCtangeAssigned toNone=>tange

    Back to the top


    Powered by Savane 3.1-cleanup