bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #34372, filter out dead remote hosts...

 
 

bug #34372: filter out dead remote hosts before running on remotes

Submitted by:  Ole Tange <tange>
Submitted on:  Fri 23 Sep 2011 11:20:29 PM UTC  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 02 Jul 2013 07:49:59 PM UTC, comment #3:

Fixed in 20130630 alpha.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Aug 2012 11:08:57 PM UTC, comment #2:

--plain might help here.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sun 03 Jun 2012 08:53:27 PM UTC, comment #1:

--filter-hosts implemented. Not working if set in profile.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Fri 23 Sep 2011 11:20:29 PM UTC, original submission:

https://lists.gnu.org/archive/html/bug-parallel/2011-09/msg00007.html

> How about this idea:
>
> If a job fails on remote.example.com, try "ssh remote.example.com
> true". If that fails too, then mark this machine as dead and do not
> use this machine again in this run. Retry the job on another host.
>
> That would deal with servers going offline after starting. It would
> not discover if the server went online at a later time. This could be
> done with a timeout on how long servers should be considered dead
> (probably 1 hour or so):
>
> If a job fails on remote.example.com, try "ssh remote.example.com
> true". If that fails too, then mark this machine as dead FOR THE NEXT
> HOUR and do not use this machine again DURING THAT TIME.


That seems like a sensible approach. I might be inclined to do the ssh true as the first action and do the connection pruning straightaway. The -M option should minimise the waste if the connection succeeds. You'll have a better idea of what the average GNU parallel user is doing. If people are mainly distributing jobs that take significant processing then the overhead of this initial check will probably be offset by having a more flexible platform. You could always include some sort of speed mode flag that doesn't attempt to do this if the user want to squeeze as much performance out of GNU parallel as possible.

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 02 Jul 2013 07:49:59 PM UTCtangeStatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup