bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #46966, memory leak with --pipe

 
 

bug #46966: memory leak with --pipe

Submitted by:  None
Submitted on:  Mon 25 Jan 2016 08:08:20 AM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Works For Me
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 28 Jan 2016 10:14:22 PM UTC, comment #1:

I tested your script. I found no leak.

Be aware that GNU Parallel uses one block of memory for each job.

So if you run 10 jobs in parallel with a block size of 1G you should expect GNU Parallel to use up to 10 GB RAM.

If you still believe there is a leak, please follow http://www.gnu.org/software/parallel/man.html#REPORTING-BUGS when re-opening this bug.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Mon 25 Jan 2016 08:08:20 AM UTC, original submission:

When using --pipe with large amount of data (10th of GB), parallel forks sub-processes that using more and more memory without clearing it.

The following command causes this issue:

And the corresponding processes with the increasing memory usage:

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Posted a comment)
  • -unavailable- added by wabu
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 28 Jan 2016 10:14:22 PM UTCtangeStatusNone=>Works For Me
      Assigned toNone=>tange
      Open/ClosedOpen=>Closed
    Mon 25 Jan 2016 08:18:06 AM UTCwabuCarbon-Copy-=>Added wabu

    Back to the top


    Powered by Savane 3.1-cleanup1