bugfindutils - Bugs: bug #31961, Out-of-control memory usage when...

 
 

bug #31961: Out-of-control memory usage when run against a large directory

Submitter:  None
Submitted:  Thu 23 Dec 2010 07:52:10 PM UTC
   
 
Category:  find Severity:  3 - Normal
Item Group:  None Status:  Works For Me
Privacy:  Public Assigned to:  jay
Originator Name:  Alex Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  4.4.0
Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 11 May 2011 08:29:10 AM UTC, comment #3: 

No response. Closing as obsolete.   Please reopen if you provide the information I asked about.

James Youngman <jay>
Group administrator
Sat 02 Apr 2011 09:49:09 PM UTC, comment #2: 

I was just taking a look at the state of this bug and I note that there has been no response.  In order to manage my workload I will close the bug unless there's an update.   If I let open bugs sit for a long time awaiting an update, they divert time from (and obscure) the bugs I can usefully work on.

Please respond.

James Youngman <jay>
Group administrator
Thu 23 Dec 2010 08:46:15 PM UTC, comment #1: 

The sizes of the files (as oppposed to directories)  is of course irrelevant; find never opens them, let alone reads them.

What is the depth of the directory hierarchy under this directory?     How many entries are there in each?  Do you get the same characteristics if you use the "oldfind" binary that's also generated when you build find (from source)?

Also, findutils-4.4.0 is quite old now, since it was released on  2008-03-15; could you try 4.4.2 (from ftp.gnu.org) or 4.5.9 (from alpha.gnu.org)?

James Youngman <jay>
Group administrator
Thu 23 Dec 2010 07:52:10 PM UTC, original submission:  

So I had this directory:

drwxrwxrwx 2 dan users 980M 2010-12-22 18:38 sessions2

The directories contents is small - just millions of tiny little files.

I ran

find sessions2 -type f -delete

but had to stop because both caused escalating memory usage. At one point it was using 65% of the system's memory.

Using simply

find sessions2 -print

printed the ".", then stopped printing, and the memory usage climbed and climbed and climbed.

I suspect that find is reading the directory's entire index into memory before doing anything.

I was able to empty the directory with this PHP script, with insignificant memory usage:

<?php
$dh = opendir($dir)) {
while (($file = readdir($dh)) !== false) {
  unlink($dir . '/' . $file);
}

closedir($dh);
?>

Could there perhaps be an option that allows the usage of find in these huge directories without reading the entire thing into memory? Perhaps disabling sorting and whatnot.

Thanks!

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-05-11 jay StatusNeed Info Works For Me
        Open/ClosedOpen Closed
    2011-04-02 jay StatusNone Need Info
        Assigned toNone jay

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code