bugfindutils - Bugs: bug #48298, Find performance problem on simple...

 
 

bug #48298: Find performance problem on simple conditions/large directories

Submitted by:  Yair <yairlenga>
Submitted on:  Thu 23 Jun 2016 10:29:01 AM UTC  
 
Category:  find Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  4.4.2 Fixed Release:  None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 23 Jun 2016 10:29:01 AM UTC, original submission:

The performance of 'find' is signficantly worst when executing simple conditions (-name ...) on large folders (>1000 files).

We are using 'find' to scan large trees (>100K files over ~300 folders). The directories are loaded incrementally over time. When the size of any folder crosses the 1000 files (approximately) performance becomes very slow.

The command is 'find . -name 'prefix.*'

We were expecting performance to be linear to the number of files. performance slowdown once we cross the threadshold was 5X, and kept going down.

The linear performance was expected since the "getdents" call return indication of folders. As a result, the whole query can be resolved by reading the directories, without having to get attributes on each an every file.

After running strace, we noticed that once a single directory has lot of files (somewhere between 700 and 1000, we could not identify exact point), find start to issue "fstat" call for each file.

In theory, the fstat call is only needed if the "find" command contain condition on the file. e.g, -type, -newer, ... If the test apply only to file name, no fstat is required.

Looking for other people advice/comment on this issue. Is this a find bug, or could this be related to the system configuraiton.

Yair <yairlenga>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by yairlenga (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    No changes have been made to this item

    Back to the top


    Powered by Savane 3.3