bugGNU Parallel - Bugs: bug #34372, filter out dead remote hosts...

 
 

bug #34372: filter out dead remote hosts before running on remotes

Submitter:  Ole Tange <tange>
Submitted:  Fri 23 Sep 2011 11:20:29 PM UTC
   
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 02 Jul 2013 07:49:59 PM UTC, comment #3: 

Fixed in 20130630 alpha.

Ole Tange <tange>
Group administrator
Mon 06 Aug 2012 11:08:57 PM UTC, comment #2: 

--plain might help here.

Ole Tange <tange>
Group administrator
Sun 03 Jun 2012 08:53:27 PM UTC, comment #1: 

--filter-hosts implemented. Not working if set in profile.

Ole Tange <tange>
Group administrator
Fri 23 Sep 2011 11:20:29 PM UTC, original submission:  

https://lists.gnu.org/archive/html/bug-parallel/2011-09/msg00007.html

> How about this idea:
>
> If a job fails on remote.example.com, try "ssh remote.example.com
> true". If that fails too, then mark this machine as dead and do not
> use this machine again in this run. Retry the job on another host.
>
> That would deal with servers going offline after starting. It would
> not discover if the server went online at a later time. This could be
> done with a timeout on how long servers should be considered dead
> (probably 1 hour or so):
>
> If a job fails on remote.example.com, try "ssh remote.example.com
> true". If that fails too, then mark this machine as dead FOR THE NEXT
> HOUR and do not use this machine again DURING THAT TIME.


That seems like a sensible approach. I might be inclined to do the ssh true as the first action and do the connection pruning straightaway. The -M option should minimise the waste if the connection succeeds. You'll have a better idea of what the average GNU parallel user is doing. If people are mainly distributing jobs that take significant processing then the overhead of this initial check will probably be offset by having a more flexible platform. You could always include some sort of speed mode flag that doesn't attempt to do this if the user want to squeeze as much performance out of GNU parallel as possible.


Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-07-02 tange StatusConfirmed Fixed
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code