bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #42983, include/exclude servers on-the-fly

 
 

bug #42983: include/exclude servers on-the-fly

Submitted by:  Ole Tange <tange>
Submitted on:  Thu 14 Aug 2014 07:27:58 AM UTC  
Votes:  51  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 24 Aug 2014 03:21:31 PM UTC, comment #2:

Fixed without daemon in [6e61256].

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sun 17 Aug 2014 03:42:53 PM UTC, comment #1:

So you are basically suggesting a daemon that keeps the slf updated.

Daemon:

forever {
nice parallel --nonall -j0 -k --slf original.slf --tag echo | remove
final tab > tmp.slf
if diff tmp.slf original.slf:
mv tmp.slf tmp2.slf
sleep 10
}

Parallel:

sub init {
cp original.slf tmp2.slf
start daemon
}

if tmp2.slf changed:
@new = grep { not $existing{$_} } @slf
@back = grep { $existing{$_} and $existing{$_}->jobslots == 0 } @slf
@removed = grep { not in @slf } keys %existing

for @new: add_host
for @back: reset_jobslots
for @removed: remove_host

sub add_host {
do as normal
}

sub reset_jobslots {
jobslots = original_jobslots
}

sub remove_host {
set jobslots = 0
}

sub cleanup {
kill daemon
rm tmp.slf tmp2.slf
}

It is starting to look more and more doable.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 14 Aug 2014 07:27:58 AM UTC, original submission:

> I am wondering if it is possible to include/exclude servers on-the-fly. This
> feature would be analogous to the way in which the load can be dynamically
> configured by simply changing a file ("procfile").


--sshloginfile already takes a file, so it will be natural to re-read that. Probably using this method:

When a job finishes, and it is more than 1 second since we checked last:
Check if the file has changed modification time. If yes: re-read it.

It ought to be fairly easy to add new servers. It will probably screw up the timings in --eta/--bar, but that is hardly a serious problem.

Removing is, however, a completely different ballgame: What do you do about jobs currently running on the servers? Also there is no infrastructure to say: Don't start new jobs on this server and remove it when the last job completes. The easiest is probably to add a 'dont start new jobs' flag to the server-object, and leave the data structure in place. It will, however, cost a few cycles to skip the server every time a new job is started.

--filter-hosts does the "removal" by not adding the host in the first place.

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mitchwyle (Voted in favor of this item)
  • -unavailable- added by daaugusto (Voted in favor of this item)
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 51 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 24 Aug 2014 03:21:31 PM UTCtangeStatusNone=>Fixed
      Assigned toNone=>tange
      Open/ClosedOpen=>Closed
    Fri 22 Aug 2014 05:15:09 PM UTCmitchwyleCarbon-Copy-=>Added mitchwyle
    Wed 20 Aug 2014 11:25:55 PM UTCdaaugustoCarbon-Copy-=>Added daaugusto

    Back to the top


    Powered by Savane 3.1-cleanup1