bugGNU Parallel - Bugs: bug #42983, include/exclude servers on-the-fly

 
 

bug #42983: include/exclude servers on-the-fly

Submitter:  Ole Tange <tange>
Submitted:  Thu 14 Aug 2014 07:27:58 AM UTC
Votes: 51
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 24 Aug 2014 03:21:31 PM UTC, comment #2: 

Fixed without daemon in [6e61256].

Ole Tange <tange>
Group administrator
Sun 17 Aug 2014 03:42:53 PM UTC, comment #1: 


So you are basically suggesting a daemon that keeps the slf updated.

Daemon:

forever {
  nice parallel --nonall -j0 -k --slf original.slf --tag echo | remove
final tab > tmp.slf
  if diff tmp.slf original.slf:
    mv tmp.slf tmp2.slf
  sleep 10
}


Parallel:

sub init {
  cp original.slf tmp2.slf
  start daemon
}

if tmp2.slf changed:
  @new = grep { not $existing{$_} } @slf
  @back = grep { $existing{$_} and $existing{$_}->jobslots == 0 } @slf
  @removed = grep { not in @slf } keys %existing

  for @new: add_host
  for @back: reset_jobslots
  for @removed: remove_host

sub add_host {
  do as normal
}

sub reset_jobslots {
  jobslots = original_jobslots
}

sub remove_host {
  set jobslots = 0
}

sub cleanup {
  kill daemon
  rm tmp.slf tmp2.slf
}

It is starting to look more and more doable.

Ole Tange <tange>
Group administrator
Thu 14 Aug 2014 07:27:58 AM UTC, original submission:  


> I am wondering if it is possible to include/exclude servers on-the-fly. This
> feature would be analogous to the way in which the load can be dynamically
> configured by simply changing a file ("procfile").


--sshloginfile already takes a file, so it will be natural to re-read that. Probably using this method:

    When a job finishes, and it is more than 1 second since we checked last:
      Check if the file has changed modification time. If yes: re-read it.

It ought to be fairly easy to add new servers. It will probably screw up the timings in --eta/--bar, but that is hardly a serious problem.

Removing is, however, a completely different ballgame: What do you do about jobs currently running on the servers? Also there is no infrastructure to say: Don't start new jobs on this server and remove it when the last job completes. The easiest is probably to add a 'dont start new jobs' flag to the server-object, and leave the data structure in place. It will, however, cost a few cycles to skip the server every time a new job is started.

--filter-hosts does the "removal" by not adding the host in the first place.

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mitchwyle (Voted in favor of this item)
  • -email is unavailable- added by daaugusto (Voted in favor of this item)
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 51 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-08-24 tange StatusNone Fixed
        Assigned toNone tange
        Open/ClosedOpen Closed
    2014-08-22 mitchwyle Carbon-Copy- Added mitchwyle
    2014-08-20 daaugusto Carbon-Copy- Added daaugusto

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code