bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #50697, --jobs 300% uses 1/3 of ssh hosts...

 
 

bug #50697: --jobs 300% uses 1/3 of ssh hosts rather than 3 jobs per core

Submitted by:  pbugrep <pbugrep>
Submitted on:  Sat 01 Apr 2017 03:11:08 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Works For Me
Privacy: PublicAssigned to: None
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 02 Apr 2017 08:46:19 PM UTC, comment #1:

I have tried reproducing your bug with:

ulimit -n 2000; seq 200000 | parallel -j300% -S1/s{1..150}.l 'sleep 1000;echo'

but was unable to.

As you probably know it is extremely hard to fix an error that you cannot reproduce yourself.

So I will close this bug for now.

But see if you can reproduce the problem consistently, and then find out which parts you can vary. Is the 200k entries important? Is the 150 hosts? Is the --jobs? When you can reproduce the problem it will be easier for you to make a MCVE.

When you are able to do so, please re-open the bug.

Ole Tange <tange>
Project Administrator
Sat 01 Apr 2017 03:11:08 PM UTC, original submission:

This is a sporadic bug with parallel failing to use all ssh hosts. In a jobfile of 200k entries with 150 hosts, only 50 (always the same ones) were being utilized when I did --jobs 300%. Seen while using single core AWS ssh hosts without parallel installed.

Switching to --jobs <INT> seems to work around the issue.

pbugrep <pbugrep>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Posted a comment)
  • -unavailable- added by pbugrep (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 02 Apr 2017 08:46:19 PM UTCtangeStatusNone=>Works For Me
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1