bugGNU Parallel - Bugs: bug #42983, include/exclude servers on-the-fly


bug #42983: include/exclude servers on-the-fly

Submitted by:  Ole Tange <tange>
Submitted on:  Thu 14 Aug 2014 07:27:58 AM UTC  
Votes:  51  
Category:  None Severity:  1 - Wish
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  Ole Tange <tange>
Open/Closed:  Closed

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Sun 24 Aug 2014 03:21:31 PM UTC, comment #2: 

Fixed without daemon in [6e61256].

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sun 17 Aug 2014 03:42:53 PM UTC, comment #1: 

So you are basically suggesting a daemon that keeps the slf updated.


forever {
  nice parallel --nonall -j0 -k --slf original.slf --tag echo | remove
final tab > tmp.slf
  if diff tmp.slf original.slf:
    mv tmp.slf tmp2.slf
  sleep 10


sub init {
  cp original.slf tmp2.slf
  start daemon

if tmp2.slf changed:
  @new = grep { not $existing{$_} } @slf
  @back = grep { $existing{$_} and $existing{$_}->jobslots == 0 } @slf
  @removed = grep { not in @slf } keys %existing

  for @new: add_host
  for @back: reset_jobslots
  for @removed: remove_host

sub add_host {
  do as normal

sub reset_jobslots {
  jobslots = original_jobslots

sub remove_host {
  set jobslots = 0

sub cleanup {
  kill daemon
  rm tmp.slf tmp2.slf

It is starting to look more and more doable.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 14 Aug 2014 07:27:58 AM UTC, original submission:  

> I am wondering if it is possible to include/exclude servers on-the-fly. This
> feature would be analogous to the way in which the load can be dynamically
> configured by simply changing a file ("procfile").

--sshloginfile already takes a file, so it will be natural to re-read that. Probably using this method:

    When a job finishes, and it is more than 1 second since we checked last:
      Check if the file has changed modification time. If yes: re-read it.

It ought to be fairly easy to add new servers. It will probably screw up the timings in --eta/--bar, but that is hardly a serious problem.

Removing is, however, a completely different ballgame: What do you do about jobs currently running on the servers? Also there is no infrastructure to say: Don't start new jobs on this server and remove it when the last job completes. The easiest is probably to add a 'dont start new jobs' flag to the server-object, and leave the data structure in place. It will, however, cost a few cycles to skip the server every time a new job is started.

--filter-hosts does the "removal" by not adding the host in the first place.

Ole Tange <tange>
Project AdministratorIn charge of this item.


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by mitchwyle (Voted in favor of this item)
  • -email is unavailable- added by daaugusto (Voted in favor of this item)
  • -email is unavailable- added by tange (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 51 encouragements so far.

    Only logged-in users can vote.




    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-08-24 tange StatusNone => Fixed
        Assigned toNone => tange
        Open/ClosedOpen => Closed
    2014-08-22 mitchwyle Carbon-Copy- => Added mitchwyle
    2014-08-20 daaugusto Carbon-Copy- => Added daaugusto

    Back to the top

    Powered by Savane 3.7