bugGNU Parallel - Bugs: bug #37010, Problems if ulimit -n is > 1024

 
 

bug #37010: Problems if ulimit -n is > 1024

Submitter:  Ole Tange <tange>
Submitted:  Wed 01 Aug 2012 01:28:27 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 01 Dec 2013 04:43:42 PM UTC, comment #6: 

In 20131122 I do not get an error anymore, but GNU Parallel does not complete if there are less than 16 file handles available.

I do not consider that a problem, because if you have less than 16 file handles available you probably should not be running GNU Parallel anyway.

Ole Tange <tange>
Group administrator
Mon 06 Aug 2012 07:20:14 PM UTC, comment #5: 

Can this somehow be made into a test?

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo > /tmp/{#} ;echo {} $?;wc < /tmp/{#}'


Ole Tange <tange>
Group administrator
Mon 06 Aug 2012 05:42:10 PM UTC, comment #4: 

All of the stalled processes have 'Cannot spawn any jobs'

338 parallel: Cannot spawn any jobs. Raising ulimit -u may help.

338 /bin/bash: line 1: 18944 Done                    seq 200

338      18945 Exit 255                | src/parallel -j0 echo

338      18946 Terminated              | wc

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo | wc;echo {}'

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 true;echo {} $?'

Ole Tange <tange>
Group administrator
Mon 06 Aug 2012 05:03:22 PM UTC, comment #3: 

seq 500 -1 1 | parallel --tag -k 'ulimit -u {}; seq 200 | src/parallel -j0 echo | wc;echo {}'

Processes blocks now and then with:

$ ps auxwww|grep para

tange     8985  0.0  0.0  12332  1444 pts/13   S+   18:58   0:00 /bin/bash -c ulimit -u 338; seq 200 | src/parallel -j0 echo | wc;echo 338

$ pstree -p 8985

bash(8985)───wc(10474)

If you kill the wc, it continues. So the parallel process is long gone, but maybe some signal is dangling?

Ole Tange <tange>
Group administrator
Thu 02 Aug 2012 08:50:57 AM UTC, comment #2: 

seq $((86+$(ps aux |grep tange|wc -l))) -1 1 | parallel -j1 -u -k --tag 'ulimit -u {};echo {}; seq 200 | parallel -j0 true'

Ole Tange <tange>
Group administrator
Thu 02 Aug 2012 08:30:00 AM UTC, comment #1: 

seq 200 -1 1 | parallel -k --tag 'ulimit -u {}; seq 200 | parallel -j0 true'

Ole Tange <tange>
Group administrator
Wed 01 Aug 2012 01:28:27 PM UTC, original submission:  

seq 2000 -1 1 | parallel -k --tag 'ulimit -n {}; seq 2000 | parallel -j0 true'

1961 Can't call method "set_max_jobs_running" on an undefined value at /usr/local/bin/parallel line 2391.
1961 parallel: Warning: Only enough available processes to run 246 jobs in parallel.
open3: fork failed: Resource temporarily unavailable at /usr/local/bin/parallel line 3674

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-12-01 tange StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2012-09-19 tange StatusConfirmed In Progress
    2012-08-07 tange StatusNone Confirmed
        Assigned toNone tange

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code