bugfindutils - Bugs: bug #23542, Problem with performance, search...

 
 

bug #23542: Problem with performance, search it's to slow

Submitted by:  None
Submitted on:  Wed 11 Jun 2008 10:55:17 PM UTC  
 
Category: findSeverity: 3 - Normal
Item Group: Wrong resultStatus: Invalid
Privacy: PublicAssigned to: James Youngman <jay>
Originator Name: BernardoOriginator Email: -unavailable-
Open/Closed: ClosedRelease: 4.4.0
Fixed Release: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 31 Aug 2008 02:20:10 PM UTC, comment #4:

From what you said already, it looks like you do not have dir_index turned on. This is what it would look like if you did:

# tune2fs -l /dev/mirror_a/source | grep features
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file

I'm closing this bug because I think this boils down to a suboptimal configuration on your system, together with a large number of files in one directory.

Please reopen this bug or open a new bug, if you believe there is still a problem specifically with findutils.

James Youngman <jay>
Project AdministratorIn charge of this item.
Thu 12 Jun 2008 02:07:53 PM UTC, comment #3:

Hi, thanks for your answers.

Here is a strace results:

Process 10913 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
75.69 2.956670 3 856505 fstatat64
24.02 0.938296 71 13257 getdents64
0.29 0.011408 10 1193 brk
0.00 0.000035 0 3529 fstat64
0.00 0.000029 0 3536 6 open
0.00 0.000028 0 3525 fchdir
0.00 0.000007 0 3531 close
0.00 0.000000 0 6 read
0.00 0.000000 0 1 write
0.00 0.000000 0 1 execve
0.00 0.000000 0 6 6 access
0.00 0.000000 0 3 2 ioctl
0.00 0.000000 0 1 gettimeofday
0.00 0.000000 0 3 munmap
0.00 0.000000 0 2 uname
0.00 0.000000 0 1 mprotect
0.00 0.000000 0 2 rt_sigaction
0.00 0.000000 0 1 rt_sigprocmask
0.00 0.000000 0 1 getrlimit
0.00 0.000000 0 17 mmap2
0.00 0.000000 0 1 fcntl64
0.00 0.000000 0 1 futex
0.00 0.000000 0 1 set_thread_area
0.00 0.000000 0 1 set_tid_address
0.00 0.000000 0 1 set_robust_list
------ ----------- ----------- --------- --------- ----------------
100.00 3.906473 885126 14 total

And this is a tune2fs -l result:

Filesystem features: has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file

I don't how checking if the dir_index it's enabled or not.

Thanks, Bernardo.

Anonymous
Thu 12 Jun 2008 09:17:53 AM UTC, comment #2:

Obviously from the timing information you gave, we can see that almost all the time is used up in I/O wait rather than user or system CPU time.

Tools that are useful for finding out why your disk I/O are slow include iostat and strace. You can use strace for this task like this:

strace -c find /home/xxx/www -newer /home/xxx/flag > /dev/null

The result is a list of system calls with a summary of how much time is spent in each. You should read the strace documentation to understand how to understand the output.

Although you mention that your filesystems are ext3, you don't indicate whether the "dir_index" feature is both enabled and in use (because you've run e2fsck -D). I assume at this point that this will make a huge difference.

As the anonymous poster said, it's likely that the differences in runtime are affected by cache warmth but I would also guess that somethign else is going on since you didn't indicate there's a pattern to the run-time.

The fundutils source tarball also builds by default a binary called "oldfind" which uses a different algorithm. It might be worth comparing the runtimes of the two implementations.

Lastly a point unrelated to findutils. If you are serving static web content out of /home/xxx/www, you have a very poor configuration. Most Unix filesystems deal badly with very large directories, and you would be wise to avoid such an arangement. One wayt of dealing with this issue is to separate the URI view of your website from the filesystem layout of the HTML content by using URL rewriting. This technique will also allow you to maintain URI stability; see http://www.w3.org/Provider/Style/URI for more information.

James Youngman <jay>
Project AdministratorIn charge of this item.
Thu 12 Jun 2008 04:20:02 AM UTC, comment #1:

Can you clarify what you're basing 'too slow' against? I'm testing against a directory of six different kernel git trees (143010 files and 8685 directories) and on a desktop (7200/8MB/IDE) drive with both cold drive and system caches* am getting roughly 90 seconds' turnaround on 'find . -newer . > /dev/null'. That's with 4.4.0 on XFS-over-LVM-over-LUKS, but in this case I doubt the filesystem really matters too much.

Warm cache (immediate turnaround w/o flushing caches) returns for me in ~2 seconds. Depending on how much memory you have, how aggressive your particular kernel is at caching, and how much other activity you have going on reading your disks and pulling files into cache, you may or may not have a hit which would explain your variability.

  • dd if=/dev/random_source of=/dev/null bs=1k count=$3xmem && sync && echo 3 > /proc/sys/vm/drop_caches
Anonymous
Wed 11 Jun 2008 10:55:17 PM UTC, original submission:

Hi,

Hi, I use the find version 4.4.0 and have slow problems when use this comand (it delays 10~15 minutes) in a directory with more than 530.000 files:

find /home/xxx/www -newer /home/xxx/flag

where flag file has the time of the last search.

How can improving ? Sometimes (1 in 10) it's show the results in seconds (why sometimes), but in generally delays more than ten minutes.

I testing in two servers, one with two SATAII in RAID software mode and the other one SATAII. Both are Debian 2.6.2x i386 and EXT3 and the times it's similar in both.

Here four results running the find command with time:

real 6m11.324s
user 0m1.236s
sys 0m16.513s

real 12m56.431s
user 0m1.620s
sys 0m19.421s

real 13m38.488s
user 0m1.512s
sys 0m19.869s

real 13m27.678s
user 0m1.652s
sys 0m19.417s

Thanks, Bernardo.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jay (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 13 Apr 2009 12:05:23 AM UTCjayOpen/ClosedOpen=>Closed
    Sun 31 Aug 2008 02:20:10 PM UTCjayCategoryNone=>find
      Item GroupNone=>Wrong result
      StatusNone=>Invalid
      Assigned toNone=>jay

    Back to the top


    Powered by Savane 3.1-cleanup1