bugfindutils - Bugs: bug #31961, Out-of-control memory usage when...

 
 

bug #31961: Out-of-control memory usage when run against a large directory

Submitted by:  None
Submitted on:  Thu 23 Dec 2010 07:52:10 PM UTC  
 
Category:  find Severity:  3 - Normal
Item Group:  None Status:  Works For Me
Privacy:  Public Assigned to:  James Youngman <jay>
Originator Name:  Alex Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  4.4.0
Fixed Release:  None

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Wed 11 May 2011 08:29:10 AM UTC, comment #3: 

No response. Closing as obsolete.   Please reopen if you provide the information I asked about.

James Youngman <jay>
Project AdministratorIn charge of this item.
Sat 02 Apr 2011 09:49:09 PM UTC, comment #2: 

I was just taking a look at the state of this bug and I note that there has been no response.  In order to manage my workload I will close the bug unless there's an update.   If I let open bugs sit for a long time awaiting an update, they divert time from (and obscure) the bugs I can usefully work on.

Please respond.

James Youngman <jay>
Project AdministratorIn charge of this item.
Thu 23 Dec 2010 08:46:15 PM UTC, comment #1: 

The sizes of the files (as oppposed to directories)  is of course irrelevant; find never opens them, let alone reads them.

What is the depth of the directory hierarchy under this directory?     How many entries are there in each?  Do you get the same characteristics if you use the "oldfind" binary that's also generated when you build find (from source)?

Also, findutils-4.4.0 is quite old now, since it was released on  2008-03-15; could you try 4.4.2 (from ftp.gnu.org) or 4.5.9 (from alpha.gnu.org)?

James Youngman <jay>
Project AdministratorIn charge of this item.
Thu 23 Dec 2010 07:52:10 PM UTC, original submission:  

So I had this directory:

drwxrwxrwx 2 dan users 980M 2010-12-22 18:38 sessions2

The directories contents is small - just millions of tiny little files.

I ran

find sessions2 -type f -delete

but had to stop because both caused escalating memory usage. At one point it was using 65% of the system's memory.

Using simply

find sessions2 -print

printed the ".", then stopped printing, and the memory usage climbed and climbed and climbed.

I suspect that find is reading the directory's entire index into memory before doing anything.

I was able to empty the directory with this PHP script, with insignificant memory usage:

<?php
$dh = opendir($dir)) {
while (($file = readdir($dh)) !== false) {
  unlink($dir . '/' . $file);
}

closedir($dh);
?>

Could there perhaps be an option that allows the usage of find in these huge directories without reading the entire thing into memory? Perhaps disabling sorting and whatnot.

Thanks!

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-05-11 jay StatusNeed Info => Works For Me
        Open/ClosedOpen => Closed
    2011-04-02 jay StatusNone => Need Info
        Assigned toNone => jay

    Back to the top


    Powered by Savane 3.5