bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #37010, Problems if ulimit -n is > 1024

 
 

bug #37010: Problems if ulimit -n is > 1024

Submitted by:  Ole Tange <tange>
Submitted on:  Wed 01 Aug 2012 01:28:27 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 01 Dec 2013 04:43:42 PM UTC, comment #6:

In 20131122 I do not get an error anymore, but GNU Parallel does not complete if there are less than 16 file handles available.

I do not consider that a problem, because if you have less than 16 file handles available you probably should not be running GNU Parallel anyway.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Aug 2012 07:20:14 PM UTC, comment #5:

Can this somehow be made into a test?

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo > /tmp/{#} ;echo {} $?;wc < /tmp/{#}'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Aug 2012 05:42:10 PM UTC, comment #4:

All of the stalled processes have 'Cannot spawn any jobs'

338 parallel: Cannot spawn any jobs. Raising ulimit -u may help.

338 /bin/bash: line 1: 18944 Done seq 200

338 18945 Exit 255 | src/parallel -j0 echo

338 18946 Terminated | wc

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo | wc;echo {}'

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 true;echo {} $?'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Aug 2012 05:03:22 PM UTC, comment #3:

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo | wc;echo {}'

Processes blocks now and then with:

$ ps auxwww|grep para

tange 8985 0.0 0.0 12332 1444 pts/13 S+ 18:58 0:00 /bin/bash -c ulimit -u 338; seq 200 | src/parallel -j0 echo | wc;echo 338

$ pstree -p 8985

bash(8985)───wc(10474)

If you kill the wc, it continues. So the parallel process is long gone, but maybe some signal is dangling?

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 02 Aug 2012 08:50:57 AM UTC, comment #2:

seq $((86+$(ps aux |grep tange|wc -l))) -1 1 | parallel -j1 -u -k --tag 'ulimit -u {};echo {}; seq 200 | parallel -j0 true'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 02 Aug 2012 08:30:00 AM UTC, comment #1:

seq 200 -1 1 | parallel -k --tag 'ulimit -u {}; seq 200 | parallel -j0 true'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Wed 01 Aug 2012 01:28:27 PM UTC, original submission:

seq 2000 -1 1 | parallel -k --tag 'ulimit -n {}; seq 2000 | parallel -j0 true'

1961 Can't call method "set_max_jobs_running" on an undefined value at /usr/local/bin/parallel line 2391.
1961 parallel: Warning: Only enough available processes to run 246 jobs in parallel.
open3: fork failed: Resource temporarily unavailable at /usr/local/bin/parallel line 3674

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 01 Dec 2013 04:43:42 PM UTCtangeStatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed
    Wed 19 Sep 2012 11:56:44 AM UTCtangeStatusConfirmed=>In Progress
    Tue 07 Aug 2012 09:30:39 AM UTCtangeStatusNone=>Confirmed
      Assigned toNone=>tange

    Back to the top


    Powered by Savane 3.1-cleanup1