bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #43518, GNU Parallel doesn't obey servers'...

 
 

bug #43518: GNU Parallel doesn't obey servers' jobs capacity when an ssh login file is reloaded

Submitted by:  Douglas A. Augusto <daaugusto>
Submitted on:  Mon 03 Nov 2014 05:36:35 PM UTC  
Votes:  47  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 15 Nov 2014 10:05:26 PM UTC, comment #7:

You're right, it seems to be a fine compromise. Thanks again for your work on extending GNU Parallel to handle the on-the-fly inclusion/exclusion of remote servers.

Douglas A. Augusto <daaugusto>
Tue 11 Nov 2014 06:15:29 PM UTC, comment #6:

It is on purpose, but maybe it is a bad choice.

I have chosen to re-use the already computed ncpu if a machine comes back. The reason for this is that it takes time to recompute the number of CPUs, and I would think it would normally be the case that the number of CPUs of a server does not change.

The only situation where this does the wrong thing is when you start out by forcing a number of CPUs and then later asks GNU Parallel to auto-detect it.

If it proves to be a problem in a real-life situation, then it will be changed.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Tue 11 Nov 2014 05:39:19 PM UTC, comment #5:

Hi Tange, thank you very much; it seems to be working like a charm! There is one minor issue, however. If I had this server entry:

10/:

and I changed to

:

Then GNU Parallel will consider that the server ':' has still 10 slots instead of trying to guess how many slows it actually has. I think that just commenting out the line (2312)

if($sshlogin->{'ncpus'}) {

is enough to fix this issue by forcing the number of slots to be calculated again (I'm not aware if there are side effects, though).

Douglas A. Augusto <daaugusto>
Fri 07 Nov 2014 11:17:43 AM UTC, comment #4:

Fixed in [ee6d07d].

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 06 Nov 2014 01:02:26 AM UTC, comment #3:

Maybe related to:

cp hosts.slf- hosts.slf; parallel --delay 0.5 echo {}/: '>>' hosts.slf ::: {9..20} & seq 250 | parallel --slf hosts.slf 'hostname; sleep 1.{}'|wc -l

Ole Tange <tange>
Project AdministratorIn charge of this item.
Wed 05 Nov 2014 08:06:23 PM UTC, comment #2:

The problem is that if either (1) there are long running jobs or (2) the ssh login file changes frequently, then one may end up with tons of processes running on a same machine, bringing it down (that happened to me!).

Unfortunately, the probability of occurring (2) grows with the number of server entries in the ssh login file and connection failures (ssh login file's updating frequency). If just one server entry is changed then GNU Parallel will duplicate the number of jobs on every single machine.

Douglas A. Augusto <daaugusto>
Wed 05 Nov 2014 07:25:25 PM UTC, comment #1:

Having looked into the problem the issue is temporary: When the host.slf file is changed, GNU Parallel "forgets" all the jobs running.

So when the current running jobs are finished, GNU Parallel will again respect the limits.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 03 Nov 2014 05:36:35 PM UTC, original submission:

Hi,

There is a bug regarding the GNU Parallel ("Shellshock" version) feature that allows the dynamic modification of the servers list ("include/exclude servers on-the-fly", #42983). I received some complaints from the administrator here at the computer lab and then I ssh-ed into some remote machines to see what was happening: there were tons of processes running on a single machine, far more than the amount of specified slots! Not only that, but also stopping GNU Parallel on the local machine didn't stop the inconvenient remote processes.

Every time the ssh login file is reloaded, a number of jobs are immediately launched on all servers, regardless of their load (number of slots currently used).

Suppose we have the following entries in the ssh login file

1/server1.net
5/server5.net

and that there are 1 and 5 jobs currently running on server1.net and server5.net, respectively.

If this file is reread for whatever reason, then GNU Parallel will launch 1 more job on server1.net (thus a total of 2 jobs) and 5 more jobs on server5.net, totaling 10 jobs there. After that, when starting new jobs GNU Parallel will not check if the total number of running jobs on a host is less than its capacity. Thus, there will be 5 + the number of unfinished jobs running on server5.net, for example.

It seems that after rereading an ssh login file, GNU Parallel simply forgets/loses track of the number of jobs still running on each machine.

Unfortunately, for any large set of reasonably long running jobs, when the ssh login file changes frequently (which is common on unreliable machines/networks) GNU Parallel will end up launching a zillion jobs on each server, effectively rendering them inoperative.

Steps to reproduce:

1) Create a 'hosts.slf' file containing some hosts in it. For instance:

1/:
1/server1.net
1/server2.net
1/server3.net

2) Run the following GNU Parallel's command:

for i in $(seq 1 1000); do perl -e 'print rand(100)'; echo; done | ./parallel --slf hosts.slf 'echo $(hostname): {}; sleep {}'

3) Monitor how many 'sleep' instances are being executing on each host:

watch -n 1 "ps aux|grep 'sleep [0-9]\+'"

4) Add or remove a host in 'hosts.slf' (e.g., comment entry '1/:') and save the file. Wait until the file 'hosts.slf' is reread (when a job finishes).

5) Go to step 3 and observe there will be an additional 'sleep' instance on each host, thus not obeying the maximum number of allowed jobs (in our case it is one).

Douglas A. Augusto <daaugusto>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Posted a comment)
  • -unavailable- added by daaugusto (Voted in favor of this item)
  • -unavailable- added by daaugusto (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 47 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 07 Nov 2014 11:17:43 AM UTCtangeStatusNone=>Fixed
      Assigned toNone=>tange
      Open/ClosedOpen=>Closed
    Mon 03 Nov 2014 05:42:45 PM UTCdaaugustoCarbon-Copy-=>Added daaugusto

    Back to the top


    Powered by Savane 3.1-cleanup1