bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #51813, User defined limit function

 
 

bug #51813: User defined limit function

Submitted by:  None
Submitted on:  Mon 21 Aug 2017 01:40:24 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 07 Sep 2017 08:36:45 PM UTC, comment #5:

Implemented in [1e62be24].

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 26 Aug 2017 10:00:29 PM UTC, comment #4:

io_file=$(tempfile)
export io_file
io_limit() {
limit=$1
# Do the measurement in the background
(tmp=$(tempfile);
LANG=C iostat -x 1 2 > $tmp;
mv $tmp $io_file) &
perl -e 'for(reverse <>) {
/Device:/ and last;
/(\S+)$/ and $max = $max > $1 ? $max : $1; }
exit ($max < '$limit')' $io_file
}
export -f io_limit
parallel --limit 'io_limit 10'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 26 Aug 2017 09:48:10 PM UTC, comment #3:

io_limit() {
limit=$1
# This is terrible slow
LANG=C iostat -x 1 2 |
perl -e 'for(reverse <>) {
/Device:/ and last;
/(\S+)$/ and $max = $max > $1 ? $max : $1; }
exit ($max < '$limit')'
}
export -f io_limit

parallel --limit 'io_limit 10'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 26 Aug 2017 09:25:23 PM UTC, comment #2:

load_limit() {
limit=$1
ps ax -o state,command |
grep -E '^[DOR].[^[]' |
wc -l |
# 0 = over limit
# 1 = below limit
perl -ne 'exit ($_ < '$limit')'
}
export -f load_limit

parallel --limit 'load_limit 5'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 26 Aug 2017 09:09:02 PM UTC, comment #1:

mem_limit() {
# 1 = OK
# 0 = wait
# 2 = kill
limit=$1
awk '/^((Swap)?Cached|MemFree|Buffers):/{ sum += $2}
END {
if (sum*1024 < '$limit'/2) { exit 2; }
# $free < 1G ? 0 : 1
else { exit (sum*1024 < '$limit') }}' /proc/meminfo
}
export -f mem_limit

parallel --limit 'mem_limit 1000000000'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 21 Aug 2017 01:40:24 PM UTC, original submission:

GNU Parallel has --memfree and --load.

How about having a generic limit function?

If this function returns 1, start another job.

If it returns 0, do not start another job (wait).

If it returns -1, kill the youngest job (similar to --memfree with less than 50%)

Can --load and --memfree be fit into that framework?

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 07 Sep 2017 08:36:45 PM UTCtangeStatusNone=>Fixed
      Assigned toNone=>tange
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1