bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #37010, Problems if ulimit -n is > 1024

 
 

bug #37010: Problems if ulimit -n is > 1024

Submitted by:  Ole Tange <tange>
Submitted on:  Wed 01 Aug 2012 01:28:27 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 01 Dec 2013 04:43:42 PM UTC, comment #6:

In 20131122 I do not get an error anymore, but GNU Parallel does not complete if there are less than 16 file handles available.

I do not consider that a problem, because if you have less than 16 file handles available you probably should not be running GNU Parallel anyway.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Aug 2012 07:20:14 PM UTC, comment #5:

Can this somehow be made into a test?

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo > /tmp/{#} ;echo {} $?;wc < /tmp/{#}'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Aug 2012 05:42:10 PM UTC, comment #4:

All of the stalled processes have 'Cannot spawn any jobs'

338 parallel: Cannot spawn any jobs. Raising ulimit -u may help.

338 /bin/bash: line 1: 18944 Done seq 200

338 18945 Exit 255 | src/parallel -j0 echo

338 18946 Terminated | wc

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo | wc;echo {}'

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 true;echo {} $?'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 06 Aug 2012 05:03:22 PM UTC, comment #3:

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo | wc;echo {}'

Processes blocks now and then with:

$ ps auxwww|grep para

tange 8985 0.0 0.0 12332 1444 pts/13 S+ 18:58 0:00 /bin/bash -c ulimit -u 338; seq 200 | src/parallel -j0 echo | wc;echo 338

$ pstree -p 8985

bash(8985)───wc(10474)

If you kill the wc, it continues. So the parallel process is long gone, but maybe some signal is dangling?

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 02 Aug 2012 08:50:57 AM UTC, comment #2:

seq $((86+$(ps aux |grep tange|wc -l))) -1 1 | parallel -j1 -u -k --tag 'ulimit -u {};echo {}; seq 200 | parallel -j0 true'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 02 Aug 2012 08:30:00 AM UTC, comment #1:

seq 200 -1 1 | parallel -k --tag 'ulimit -u {}; seq 200 | parallel -j0 true'

Ole Tange <tange>
Project AdministratorIn charge of this item.
Wed 01 Aug 2012 01:28:27 PM UTC, original submission:

seq 2000 -1 1 | parallel -k --tag 'ulimit -n {}; seq 2000 | parallel -j0 true'

1961 Can't call method "set_max_jobs_running" on an undefined value at /usr/local/bin/parallel line 2391.
1961 parallel: Warning: Only enough available processes to run 246 jobs in parallel.
open3: fork failed: Resource temporarily unavailable at /usr/local/bin/parallel line 3674

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 01 Dec 2013 04:43:42 PM UTCtangeStatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed
    Wed 19 Sep 2012 11:56:44 AM UTCtangeStatusConfirmed=>In Progress
    Tue 07 Aug 2012 09:30:39 AM UTCtangeStatusNone=>Confirmed
      Assigned toNone=>tange

    Back to the top


    Powered by Savane 3.1-cleanup