bugfindutils - Bugs: bug #57693, find wastefully calls stat for...

 
 

bug #57693: find wastefully calls stat for leaves

Submitter:  Vladimir Panteleev <vpanteleev>
Submitted:  Wed 29 Jan 2020 12:49:28 PM UTC
   
 
Category:  find Severity:  4 - Important
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 29 Jan 2020 12:49:28 PM UTC, original submission:  

Consider the invocation:

find /dir -mindepth 1 -maxdepth 1

The expected behavior is for find to print the full paths of all directory entries in /dir, which it does.

However, as far as I can see, this task should not require find to perform stat calls on the directory entries of /dir. Nevertheless, it does so.

In certain situations, a mere directory listing is much faster than also calling stat on every member. In my case, I am seeing a considerable performance difference when enumerating snapshots (~2000 total) on a btrfs filesystem located on a HDD. `ls /dir | cat` is almost instantaneous (here, the output is piped through `cat` so that `ls` doesn't attempt to colorize entries, which would require `stat` calls). However, the aforementioned `find` invocation, as well as just `ls`, takes several minutes.

The find manual states the following for the -D option:

    stat   Print messages as files are examined with the stat and lstat system calls.  The find program tries to minimise such calls.

However, I can observe that find does call stat, without even printing anything with `-D stat` prepended to its command line.

I can see that find calls stat by attaching to it, as it is running, with gdb, and examining its backtrace. With findutils 28f11d689dc61f9202de44078d67299419fbad26 and gnulib a7903da07d3d18c23314aa0815adbb4058fd7cec, here is one instance:

Thread 1 (process 277972):
#0  0x00007f2aa82bdddf in __fxstatat64 () from /usr/lib/libc.so.6
#1  0x00005557b564ff96 in fstatat (__flag=256, __statbuf=0x5557b6204a48, __filename=<optimized out>, __fd=<optimized out>) at /usr/include/sys/stat.h:477
#2  fts_stat (sp=sp@entry=0x5557b61cbf40, p=p@entry=0x5557b62049d0, follow=follow@entry=false) at fts.c:1827
#3  0x00005557b565208b in rpl_fts_read (sp=0x5557b61cbf40) at fts.c:1044
#4  0x00005557b5634012 in find (arg=0x7ffc358806ea "/mnt/2016-hdd-8t-raid/home") at ftsfind.c:561
#5  0x00005557b5633aea in process_all_startpoints (argv=<optimized out>, argc=<optimized out>) at ftsfind.c:625
#6  main (argc=<optimized out>, argv=<optimized out>) at ftsfind.c:734

It looks like stat is not being called by find directly, but rather the fts feature of gnulib, so it looks like there is possibly a second bug here (-D stat not reporting stat calls in gnulib).

Vladimir Panteleev <vpanteleev>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jay (Updated the item)
  • -email is unavailable- added by vpanteleev (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-05-19 jay Severity3 - Normal 4 - Important

    Back to the top

    Powered by Savane 3.13-54b4.
    Corresponding source code