bugfindutils - Bugs: bug #26945, [wishlist] locate-d: dynamic...

 
 

bug #26945: [wishlist] locate-d: dynamic updates using dnotify, inotify or fsnotify

Submitted by:  Paul Wise <pabs>
Submitted on:  Fri 03 Jul 2009 07:53:04 AM UTC  
 
Category: locateSeverity: 1 - Wish
Item Group: NoneStatus: Postponed
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: NoneFixed Release: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Thu 11 Apr 2013 02:01:45 AM UTC, comment #6:

There is now an implementation of a system-wide filesystem monitor using fanotify:

‚Äčhttp://www.piware.de/2012/02/fatrace-report-system-wide-file-access-events/

Paul Wise <pabs>
Sun 11 Apr 2010 11:58:46 AM UTC, comment #5:

updatedb would probably be best rewritten. I'm going to mark the updatedb-related bus as "Postponed" in order to consider them all in a batch, when I get around to the rewrite.

(Leslie P. Polzer did start on a rewrite some time back as part of the Summer of Code but there are some updatedb-related issues his changes don't resolve yet, I think)

James Youngman <jay>
Project Administrator
Wed 08 Jul 2009 04:07:37 AM UTC, comment #4:

Eric Paris <eparis@redhat.com> wrote:

It could, although fanotify doesn't have good (really any?) handling of mv. I've been thinking about how to handle those but haven't had the time to try to code something, and I'm sure that the VFS people are going to give me a really rough time of it.

An fanotify listener can easily listen for all modify events and record what needs to be later re-indexed. Heck, I could probably code that in less than 200 lines (>50% of something like that code is in the lkml posting).

But I think you would still need to traverse the whole fs regularly to look for moves, which might defeat the purpose.....

I just haven't figured out a reasonable way to indicate mv. I probably should spend some time looking what other OS's do....

Paul Wise <pabs3@bonedaddy.net> wrote:

Do you mind if I copy this to the bug report?

I definitely think mv notifications are needed, some of the links I posted in the bug report might help you design them.

Eric Paris <eparis@redhat.com> wrote:

Feel free to say anything I say, although I just realized the ONLY thing that matters for updatedb is move, so really all I told you is "I can't do that right now"

I started looking at move tonight, I know I could do something on
sys_rename but I don't see how to do notification if an inode is moved by say the nfsd server instead of by a sys_call. I'll try to talk to the VFS maintainer and see if he has some thoughts.....

Paul Wise <pabs>
Sat 04 Jul 2009 09:07:49 AM UTC, comment #3:

I'm sending a mail to the fsnotify/fanotify author and will report back to this bug.

Paul Wise <pabs>
Sat 04 Jul 2009 09:03:29 AM UTC, comment #2:

I believe fsnotify will allow this:

http://lwn.net/Articles/311850/

If it doesn't, contacting the fsnotify/fanotify author and asking for the feature might be a good idea.

For inotify it appears that Linux allows the number of watches to be increased (see the -r option) when the user is root:

http://manpages.debian.net/man/1/inotifywatch

Other kernels have similar things:

http://en.wikipedia.org/wiki/FSEvents
http://www.opengroup.org/onlinepubs/9657099/
http://msdn.microsoft.com/en-us/library/aa365261(VS.85).aspx

It might be a good idea to look at FAM and gamin to find out how to do this on various kernels:

http://en.wikipedia.org/wiki/File_alteration_monitor
http://en.wikipedia.org/wiki/Gamin

Paul Wise <pabs>
Fri 03 Jul 2009 09:33:45 AM UTC, comment #1:

I believe the problem with this approach is that it requires an open fd per directory to be monitored. There are over 140,000 directories just on my desktop machine. If you can think of a way to do this with a daemon that uses less than (say) 100 open file descriptors, we should talk in detail about the design.

James Youngman <jay>
Project Administrator
Fri 03 Jul 2009 07:53:04 AM UTC, original submission:

It would be nice to not have to run updatedb every night, causing large I/O bandwidth reading every filename off the disk. One way to achieve this would be locate-d, a daemon which would listen to kernel-emitted filesystem events (from dnotify, inotify or fsnotify for Linux) and then update the database at a configurable interval. By storing timestamps of the events in the database, information about the evolution of the filesystem over time could be obtained. locate could be modified to query this daemon before checking the database. Such a feature (along with fixing the other issues fixed by other implementations) could make GNU findutils the default locate implementation on most GNU/Linux distros again.

Paul Wise <pabs>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jay (Posted a comment)
  • -unavailable- added by pabs (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 29 Sep 2013 07:56:18 PM UTCjaySeverity3 - Normal=>1 - Wish
    Sun 11 Apr 2010 11:58:46 AM UTCjayStatusNone=>Postponed

    Back to the top


    Powered by Savane 3.1-cleanup