bugGNU Parallel - Bugs: bug #34241, --pipe should not spawn unneeded...

 
 

bug #34241: --pipe should not spawn unneeded processes

Submitter:  Ole Tange <tange>
Submitted:  Wed 07 Sep 2011 01:57:01 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 02 Jul 2013 07:50:49 PM UTC, comment #7: 

Fixed before 20130630 alpha.

Ole Tange <tange>
Group administrator
Tue 09 Apr 2013 03:52:00 PM UTC, comment #6: 

This generates 10 files:

    seq 1000 | parallel -j10 --pipe --block 1k --files wc

But only 4 jobs are run:

    seq 1000 | parallel -j10 --pipe --block 1k wc

Ole Tange <tange>
Group administrator
Thu 27 Dec 2012 10:08:47 PM UTC, comment #5: 

Fixed in [bbc9a53ea5f0]

Ole Tange <tange>
Group administrator
Thu 27 Dec 2012 09:54:47 PM UTC, comment #4: 

Test:

seq 3 | src/parallel -j30  --pipe --block-size 10 cat\;echo o

Ole Tange <tange>
Group administrator
Thu 27 Dec 2012 09:18:50 PM UTC, comment #3: 

Test if this works

eval `echo $SHELL | grep -E "/(t)?csh" > /dev/null && echo setenv _FIRST_CHAR_FILE /tmp/$$.first_char_file || echo export _FIRST_CHAR_FILE=/tmp/$$.first_char_file`

dd bs=1 count=1 of=$_FIRST_CHAR_FILE >&/dev/null; test -s "$_FIRST_CHAR_FILE" && ( cat $_FIRST_CHAR_FILE; rm $_FIRST_CHAR_FILE; cat - ) | program


Ole Tange <tange>
Group administrator
Sun 06 May 2012 03:15:55 PM UTC, comment #2: 

The perl -e eof wrapper does not work: STDIN is not passed on to your_command.

Ole Tange <tange>
Group administrator
Fri 04 May 2012 07:09:30 PM UTC, comment #1: 

One solution is to wrap the command with a test for eof:

 if perl -e 'eof() and exit(1)'; then (your;command;here); else true; fi

This could be put in sshlogin_wrap


Ole Tange <tange>
Group administrator
Wed 07 Sep 2011 01:57:01 PM UTC, original submission:  

I believe the bug can be summed up as:

  echo | parallel -j2 -N1 --pipe md5sum -c

should only give one error (from md5sum) instead of 2.

  echo | parallel -r -j2 -N1 --pipe md5sum -c

should give no errors.

The bug is confirmed.


Not spawning unneeded processes


Currently GNU Parallel spawns a new process when a process dies.

It will require some redesign to get it to not spawn more processes if there is no more data to process. So do not expect a quick fix for this.


/Ole

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-07-02 tange StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2013-04-09 tange StatusFixed Confirmed
        Open/ClosedClosed Open
    2012-12-27 tange StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2012-05-06 tange StatusNone Confirmed
    2012-03-30 tange Assigned toNone tange

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code