bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #37010, Problems if ulimit -n is > 1024

 
 

bug #37010: Problems if ulimit -n is > 1024

Submitted by:  Ole Tange <tange>
Submitted on:  Wed 01 Aug 2012 01:28:27 PM UTC  
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  Ole Tange <tange>
Open/Closed:  Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 01 Dec 2013 04:43:42 PM UTC, comment #6:

In 20131122 I do not get an error anymore, but GNU Parallel does not complete if there are less than 16 file handles available.

I do not consider that a problem, because if you have less than 16 file handles available you probably should not be running GNU Parallel anyway.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Aug 2012 07:20:14 PM UTC, comment #5:

Can this somehow be made into a test?

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo > /tmp/{#} ;echo {} $?;wc < /tmp/{#}'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Aug 2012 05:42:10 PM UTC, comment #4:

All of the stalled processes have 'Cannot spawn any jobs'

338 parallel: Cannot spawn any jobs. Raising ulimit -u may help.

338 /bin/bash: line 1: 18944 Done seq 200

338 18945 Exit 255 | src/parallel -j0 echo

338 18946 Terminated | wc

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo | wc;echo {}'

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 true;echo {} $?'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Aug 2012 05:03:22 PM UTC, comment #3:

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo | wc;echo {}'

Processes blocks now and then with:

$ ps auxwww|grep para

tange 8985 0.0 0.0 12332 1444 pts/13 S+ 18:58 0:00 /bin/bash -c ulimit -u 338; seq 200 | src/parallel -j0 echo | wc;echo 338

$ pstree -p 8985

bash(8985)───wc(10474)

If you kill the wc, it continues. So the parallel process is long gone, but maybe some signal is dangling?

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 02 Aug 2012 08:50:57 AM UTC, comment #2:

seq $((86+$(ps aux |grep tange|wc -l))) -1 1 | parallel -j1 -u -k --tag 'ulimit -u {};echo {}; seq 200 | parallel -j0 true'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 02 Aug 2012 08:30:00 AM UTC, comment #1:

seq 200 -1 1 | parallel -k --tag 'ulimit -u {}; seq 200 | parallel -j0 true'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Wed 01 Aug 2012 01:28:27 PM UTC, original submission:

seq 2000 -1 1 | parallel -k --tag 'ulimit -n {}; seq 2000 | parallel -j0 true'

1961 Can't call method "set_max_jobs_running" on an undefined value at /usr/local/bin/parallel line 2391.
1961 parallel: Warning: Only enough available processes to run 246 jobs in parallel.
open3: fork failed: Resource temporarily unavailable at /usr/local/bin/parallel line 3674

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-12-01 tange StatusIn Progress => Fixed
        Open/ClosedOpen => Closed
    2012-09-19 tange StatusConfirmed => In Progress
    2012-08-07 tange StatusNone => Confirmed
        Assigned toNone => tange

    Back to the top


    Powered by Savane 3.3