bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #34241, --pipe should not spawn unneeded...

 
 

bug #34241: --pipe should not spawn unneeded processes

Submitted by:  Ole Tange <tange>
Submitted on:  Wed 07 Sep 2011 01:57:01 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 02 Jul 2013 07:50:49 PM UTC, comment #7:

Fixed before 20130630 alpha.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 09 Apr 2013 03:52:00 PM UTC, comment #6:

This generates 10 files:

seq 1000 | parallel -j10 --pipe --block 1k --files wc

But only 4 jobs are run:

seq 1000 | parallel -j10 --pipe --block 1k wc

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 27 Dec 2012 10:08:47 PM UTC, comment #5:

Fixed in [bbc9a53ea5f0]

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 27 Dec 2012 09:54:47 PM UTC, comment #4:

Test:

seq 3 | src/parallel -j30 --pipe --block-size 10 cat\;echo o

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 27 Dec 2012 09:18:50 PM UTC, comment #3:

Test if this works

eval `echo $SHELL | grep -E "/(t)?csh" > /dev/null && echo setenv _FIRST_CHAR_FILE /tmp/$$.first_char_file || echo export _FIRST_CHAR_FILE=/tmp/$$.first_char_file`

dd bs=1 count=1 of=$_FIRST_CHAR_FILE >&/dev/null; test -s "$_FIRST_CHAR_FILE" && ( cat $_FIRST_CHAR_FILE; rm $_FIRST_CHAR_FILE; cat - ) | program

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sun 06 May 2012 03:15:55 PM UTC, comment #2:

The perl -e eof wrapper does not work: STDIN is not passed on to your_command.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Fri 04 May 2012 07:09:30 PM UTC, comment #1:

One solution is to wrap the command with a test for eof:

if perl -e 'eof() and exit(1)'; then (your;command;here); else true; fi

This could be put in sshlogin_wrap

Ole Tange <tange>
Project AdministratorIn charge of this item.
Wed 07 Sep 2011 01:57:01 PM UTC, original submission:

I believe the bug can be summed up as:

echo | parallel -j2 -N1 --pipe md5sum -c

should only give one error (from md5sum) instead of 2.

echo | parallel -r -j2 -N1 --pipe md5sum -c

should give no errors.

The bug is confirmed.

Not spawning unneeded processes

Currently GNU Parallel spawns a new process when a process dies.

It will require some redesign to get it to not spawn more processes if there is no more data to process. So do not expect a quick fix for this.

/Ole

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 8 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 02 Jul 2013 07:50:49 PM UTCtangeStatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Tue 09 Apr 2013 03:52:00 PM UTCtangeStatusFixed=>Confirmed
      Open/ClosedClosed=>Open
    Thu 27 Dec 2012 10:08:47 PM UTCtangeStatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Sun 06 May 2012 03:15:55 PM UTCtangeStatusNone=>Confirmed
    Fri 30 Mar 2012 12:40:12 AM UTCtangeAssigned toNone=>tange

    Back to the top


    Powered by Savane 3.1-cleanup1