bugGNU Parallel - Bugs: bug #43518, GNU Parallel doesn't obey servers'...

 
 

bug #43518: GNU Parallel doesn't obey servers' jobs capacity when an ssh login file is reloaded

Submitter:  Douglas A. Augusto <daaugusto>
Submitted:  Mon 03 Nov 2014 05:36:35 PM UTC
Votes: 47
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 15 Nov 2014 10:05:26 PM UTC, comment #7: 

You're right, it seems to be a fine compromise. Thanks again for your work on extending GNU Parallel to handle the on-the-fly inclusion/exclusion of remote servers.

Douglas A. Augusto <daaugusto>
Tue 11 Nov 2014 06:15:29 PM UTC, comment #6: 

It is on purpose, but maybe it is a bad choice.

I have chosen to re-use the already computed ncpu if a machine comes back. The reason for this is that it takes time to recompute the number of CPUs, and I would think it would normally be the case that the number of CPUs of a server does not change.

The only situation where this does the wrong thing is when you start out by forcing a number of CPUs and then later asks GNU Parallel to auto-detect it.

If it proves to be a problem in a real-life situation, then it will be changed.

Ole Tange <tange>
Group administrator
Tue 11 Nov 2014 05:39:19 PM UTC, comment #5: 

Hi Tange, thank you very much; it seems to be working like a charm! There is one minor issue, however. If I had this server entry:

   10/:

and I changed to

   :

Then GNU Parallel will consider that the server ':' has still 10 slots instead of trying to guess how many slows it actually has. I think that just commenting out the line (2312)

   if($sshlogin->{'ncpus'}) {

is enough to fix this issue by forcing the number of slots to be calculated again (I'm not aware if there are side effects, though).

Douglas A. Augusto <daaugusto>
Fri 07 Nov 2014 11:17:43 AM UTC, comment #4: 

Fixed in [ee6d07d].

Ole Tange <tange>
Group administrator
Thu 06 Nov 2014 01:02:26 AM UTC, comment #3: 

Maybe related to:

cp hosts.slf- hosts.slf; parallel --delay 0.5  echo {}/: '>>' hosts.slf ::: {9..20} & seq 250 | parallel  --slf hosts.slf 'hostname; sleep 1.{}'|wc -l

Ole Tange <tange>
Group administrator
Wed 05 Nov 2014 08:06:23 PM UTC, comment #2: 

The problem is that if either (1) there are long running jobs or (2) the ssh login file changes frequently, then one may end up with tons of processes running on a same machine, bringing it down (that happened to me!).

Unfortunately, the probability of occurring (2) grows with the number of server entries in the ssh login file and connection failures (ssh login file's updating frequency). If just one server entry is changed then GNU Parallel will duplicate the number of jobs on every single machine.

Douglas A. Augusto <daaugusto>
Wed 05 Nov 2014 07:25:25 PM UTC, comment #1: 

Having looked into the problem the issue is temporary: When the host.slf file is changed, GNU Parallel "forgets" all the jobs running.

So when the current running jobs are finished, GNU Parallel will again respect the limits.

Ole Tange <tange>
Group administrator
Mon 03 Nov 2014 05:36:35 PM UTC, original submission:  

Hi,

There is a bug regarding the GNU Parallel ("Shellshock" version) feature that allows the dynamic modification of the servers list ("include/exclude servers on-the-fly", #42983). I received some complaints from the administrator here at the computer lab and then I ssh-ed into some remote machines to see what was happening: there were tons of processes running on a single machine, far more than the amount of specified slots! Not only that, but also stopping GNU Parallel on the local machine didn't stop the inconvenient remote processes.

Every time the ssh login file is reloaded, a number of jobs are immediately launched on all servers, regardless of their load (number of slots currently used).

Suppose we have the following entries in the ssh login file

   1/server1.net
   5/server5.net

and that there are 1 and 5 jobs currently running on server1.net and server5.net, respectively.

If this file is reread for whatever reason, then GNU Parallel will launch 1 more job on server1.net (thus a total of 2 jobs) and 5 more jobs on server5.net, totaling 10 jobs there. After that, when starting new jobs GNU Parallel will not check if the total number of running jobs on a host is less than its capacity. Thus, there will be 5 + the number of unfinished jobs running on server5.net, for example.

It seems that after rereading an ssh login file, GNU Parallel simply forgets/loses track of the number of jobs still running on each machine.

Unfortunately, for any large set of reasonably long running jobs, when the ssh login file changes frequently (which is common on unreliable machines/networks) GNU Parallel will end up launching a zillion jobs on each server, effectively rendering them inoperative.

Steps to reproduce:

1) Create a 'hosts.slf' file containing some hosts in it. For instance:

   1/:
   1/server1.net
   1/server2.net
   1/server3.net

2) Run the following GNU Parallel's command:

   for i in $(seq 1 1000); do perl -e 'print rand(100)'; echo; done | ./parallel --slf hosts.slf 'echo $(hostname): {}; sleep {}'

3) Monitor how many 'sleep' instances are being executing on each host:

   watch -n 1 "ps aux|grep 'sleep [0-9]\+'"

4) Add or remove a host in 'hosts.slf' (e.g., comment entry '1/:') and save the file. Wait until the file 'hosts.slf' is reread (when a job finishes).

5) Go to step 3 and observe there will be an additional 'sleep' instance on each host, thus not obeying the maximum number of allowed jobs (in our case it is one).

Douglas A. Augusto <daaugusto>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)
  • -email is unavailable- added by daaugusto (Voted in favor of this item)
  • -email is unavailable- added by daaugusto (Submitted the item)
  •  

    There are 47 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-11-07 tange StatusNone Fixed
        Assigned toNone tange
        Open/ClosedOpen Closed
    2014-11-03 daaugusto Carbon-Copy- Added daaugusto

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code