bugGNU Parallel - Bugs: bug #51813, User defined limit function

 
 

bug #51813: User defined limit function

Submitter:  None
Submitted:  Mon 21 Aug 2017 01:40:24 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 07 Sep 2017 08:36:45 PM UTC, comment #5: 

Implemented in [1e62be24].

Ole Tange <tange>
Group administrator
Sat 26 Aug 2017 10:00:29 PM UTC, comment #4: 


io_file=$(tempfile)
export io_file
io_limit() {
  limit=$1
  # Do the measurement in the background
  (tmp=$(tempfile);
   LANG=C iostat -x 1 2 > $tmp;
   mv $tmp $io_file) &
  perl -e 'for(reverse <>) {
      /Device:/ and last;
      /(\S+)$/ and $max = $max > $1 ? $max : $1; }
      exit ($max < '$limit')' $io_file
}
export -f io_limit
parallel --limit 'io_limit 10'

Ole Tange <tange>
Group administrator
Sat 26 Aug 2017 09:48:10 PM UTC, comment #3: 

io_limit() {
  limit=$1
  # This is terrible slow
  LANG=C iostat -x 1 2 |
    perl -e 'for(reverse <>) {
      /Device:/ and last;
      /(\S+)$/ and $max = $max > $1 ? $max : $1; }
      exit ($max < '$limit')'
}   
export -f io_limit

parallel --limit 'io_limit 10'

Ole Tange <tange>
Group administrator
Sat 26 Aug 2017 09:25:23 PM UTC, comment #2: 

load_limit() {
    limit=$1
    ps ax -o state,command |
      grep -E '^[DOR].[^[]' |
      wc -l |
      # 0 = over limit
      # 1 = below limit
      perl -ne 'exit ($_ < '$limit')'
}
export -f load_limit

parallel --limit 'load_limit 5'

Ole Tange <tange>
Group administrator
Sat 26 Aug 2017 09:09:02 PM UTC, comment #1: 

mem_limit() {
    # 1 = OK
    # 0 = wait
    # 2 = kill
    limit=$1
    awk '/^((Swap)?Cached|MemFree|Buffers):/{ sum += $2}
         END {
           if (sum*1024 < '$limit'/2) { exit 2; }
           # $free < 1G ? 0 : 1
           else { exit (sum*1024 < '$limit') }}' /proc/meminfo
}
export -f mem_limit

parallel --limit 'mem_limit 1000000000'

Ole Tange <tange>
Group administrator
Mon 21 Aug 2017 01:40:24 PM UTC, original submission:  

GNU Parallel has --memfree and --load.

How about having a generic limit function?

If this function returns 1, start another job.

If it returns 0, do not start another job (wait).

If it returns -1, kill the youngest job (similar to --memfree with less than 50%)

Can --load and --memfree be fit into that framework?



Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-09-07 tange StatusNone Fixed
        Assigned toNone tange
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code