bugGNU Parallel - Bugs: bug #42983, include/exclude servers on-the-fly

 
 

bug #42983: include/exclude servers on-the-fly

Submitted by:  Ole Tange <tange>
Submitted on:  Thu 14 Aug 2014 07:27:58 AM UTC  
Votes:  51  
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  Ole Tange <tange>
Open/Closed:  Closed

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 24 Aug 2014 03:21:31 PM UTC, comment #2: 

Fixed without daemon in [6e61256].

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sun 17 Aug 2014 03:42:53 PM UTC, comment #1: 

So you are basically suggesting a daemon that keeps the slf updated.

Daemon:

forever {
  nice parallel --nonall -j0 -k --slf original.slf --tag echo | remove
final tab > tmp.slf
  if diff tmp.slf original.slf:
    mv tmp.slf tmp2.slf
  sleep 10
}

Parallel:

sub init {
  cp original.slf tmp2.slf
  start daemon
}

if tmp2.slf changed:
  @new = grep { not $existing{$_} } @slf
  @back = grep { $existing{$_} and $existing{$_}->jobslots == 0 } @slf
  @removed = grep { not in @slf } keys %existing

  for @new: add_host
  for @back: reset_jobslots
  for @removed: remove_host

sub add_host {
  do as normal
}

sub reset_jobslots {
  jobslots = original_jobslots
}

sub remove_host {
  set jobslots = 0
}

sub cleanup {
  kill daemon
  rm tmp.slf tmp2.slf
}

It is starting to look more and more doable.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 14 Aug 2014 07:27:58 AM UTC, original submission:  

> I am wondering if it is possible to include/exclude servers on-the-fly. This
> feature would be analogous to the way in which the load can be dynamically
> configured by simply changing a file ("procfile").


--sshloginfile already takes a file, so it will be natural to re-read that. Probably using this method:

    When a job finishes, and it is more than 1 second since we checked last:
      Check if the file has changed modification time. If yes: re-read it.

It ought to be fairly easy to add new servers. It will probably screw up the timings in --eta/--bar, but that is hardly a serious problem.

Removing is, however, a completely different ballgame: What do you do about jobs currently running on the servers? Also there is no infrastructure to say: Don't start new jobs on this server and remove it when the last job completes. The easiest is probably to add a 'dont start new jobs' flag to the server-object, and leave the data structure in place. It will, however, cost a few cycles to skip the server every time a new job is started.

--filter-hosts does the "removal" by not adding the host in the first place.

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mitchwyle (Voted in favor of this item)
  • -email is unavailable- added by daaugusto (Voted in favor of this item)
  • -email is unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 51 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-08-24 tange StatusNone => Fixed
        Assigned toNone => tange
        Open/ClosedOpen => Closed
    2014-08-22 mitchwyle Carbon-Copy- => Added mitchwyle
    2014-08-20 daaugusto Carbon-Copy- => Added daaugusto

    Back to the top


    Powered by Savane 3.7