bugfindutils - Bugs: bug #48298, Find performance problem on simple...

 
 

bug #48298: Find performance problem on simple conditions/large directories

Submitter:  Yair <yairlenga>
Submitted:  Thu 23 Jun 2016 10:29:01 AM UTC
   
 
Category:  find Severity:  3 - Normal
Item Group:  None Status:  Need Info
Privacy:  Public Assigned to:  jay
Originator Name:  Open/Closed:  Open
Release:  4.4.2 Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 01 Sep 2019 07:51:12 AM UTC, comment #1: 

Findutils 4.7.0 was recently released and includes a newer version of the fts code.  Could you check whether this problem also exists in 4.7.0 please?

James Youngman <jay>
Group administrator
Thu 23 Jun 2016 10:29:01 AM UTC, original submission:  

The performance of 'find' is signficantly worst when executing simple conditions (-name ...) on large folders (>1000 files).

We are using 'find' to scan large trees (>100K files over ~300 folders). The directories are loaded incrementally over time. When the size of any folder crosses the 1000 files (approximately) performance becomes very slow.

The command is 'find . -name 'prefix.*'

We were expecting performance to be linear to the number of files. performance slowdown once we cross the threadshold was 5X, and kept going down.

The linear performance was expected since the "getdents" call return indication of folders. As a result, the whole query can be resolved by reading the directories, without having to get attributes on each an every file.

After running strace, we noticed that once a single directory has lot of files (somewhere between 700 and 1000, we could not identify exact point), find start to issue "fstat" call for each file.

In theory, the fstat call is only needed if the "find" command contain condition on the file. e.g, -type, -newer, ... If the test apply only to file name, no fstat is required.

Looking for other people advice/comment on this issue. Is this a find bug, or could this be related to the system configuraiton.

Yair <yairlenga>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by yairlenga (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-09-01 jay StatusNone Need Info
        Assigned toNone jay

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code