bugGNU Parallel - Bugs: bug #36456, spawn sshs slower to avoid htting...

 
 

bug #36456: spawn sshs slower to avoid htting MaxStartup

Submitter:  Ole Tange <tange>
Submitted:  Sat 12 May 2012 11:38:00 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 02 Jul 2013 07:48:24 PM UTC, comment #4: 

Probably fixed in 20130630 alpha.

Ole Tange <tange>
Group administrator
Mon 06 Aug 2012 11:04:04 PM UTC, comment #3: 

Using --joblog we can run ssh to all hosts in parallel twice and get the timings with ms accuracity.

It requires --plain to work as we need to be able to ignore all --profile/$PARALLEL etc.


Ole Tange <tange>
Group administrator
Mon 06 Aug 2012 11:03:56 PM UTC, comment #2: 

Auto master connection does not solve this.

https://lists.gnu.org/archive/html/parallel/2012-08/msg00012.html

Ole Tange <tange>
Group administrator
Tue 22 May 2012 11:30:29 PM UTC, comment #1: 

We also need to see if this can be fixed using auto master connection

Ole Tange <tange>
Group administrator
Sat 12 May 2012 11:38:00 PM UTC, original submission:  

sshd has MaxStartup which limits how many ssh connections that can be in login state (typically 10).

To avoid hitting that limit GNU Parallel should wait a little between spawning another ssh to the same host. The time to wait could be the time it takes to run:

  ssh remote echo foo

as that will be a full login. It should be run twice and the first be ignored, as DNS may account for the most of the delay.

Maybe a better wait time is the above time / 10 (for MaxStartup) * 2 (for 100% error margin).

To time this use:

  use Time::HiRes qw ( time );

time() without hires will give time in seconds and if added 1 second it will give a pessimistic time.

Experiments:

time ssh m echo foo
0.086s
sleep 0.004;ssh m echo & # This works

time ssh l echo foo
5.1s
sleep 0.1;ssh l echo & # This works

time ssh n echo
0.198s
sleep 0.02;ssh n echo & # This works

time ssh n2 echo
0.211s
sleep 0.03;ssh n2 echo & # This works

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-07-02 tange StatusNone Fixed
        Open/ClosedOpen Closed
    2012-08-06 tange Assigned toNone tange

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code