bugfindutils - Bugs: bug #26945, [wishlist] locate-d: dynamic...

 
 

bug #26945: [wishlist] locate-d: dynamic updates using dnotify, inotify or fsnotify

Submitter:  Paul Wise <pabs>
Submitted:  Fri 03 Jul 2009 07:53:04 AM UTC
   
 
Category:  locate Severity:  1 - Wish
Item Group:  None Status:  Postponed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 11 Apr 2013 02:01:45 AM UTC, comment #6: 

There is now an implementation of a system-wide filesystem monitor using fanotify:

http://www.piware.de/2012/02/fatrace-report-system-wide-file-access-events/

Paul Wise <pabs>
Sun 11 Apr 2010 11:58:46 AM UTC, comment #5: 

updatedb would probably be best rewritten.   I'm going to mark the updatedb-related bus as "Postponed" in order to consider them all in a batch, when I get around to the rewrite.

(Leslie P. Polzer did start on a rewrite some time back as part of the Summer of Code but there are some updatedb-related issues his changes don't resolve yet, I think)

James Youngman <jay>
Group administrator
Wed 08 Jul 2009 04:07:37 AM UTC, comment #4: 

Eric Paris <eparis@redhat.com> wrote:

It could, although fanotify doesn't have good (really any?) handling of mv.  I've been thinking about how to handle those but haven't had the time to try to code something, and I'm sure that the VFS people are going to give me a really rough time of it.

An fanotify listener can easily listen for all modify events and record what needs to be later re-indexed.  Heck, I could probably code that in less than 200 lines (>50% of something like that code is in the lkml posting).

But I think you would still need to traverse the whole fs regularly to look for moves, which might defeat the purpose.....

I just haven't figured out a reasonable way to indicate mv.  I probably should spend some time looking what other OS's do....

Paul Wise <pabs3@bonedaddy.net> wrote:

Do you mind if I copy this to the bug report?

I definitely think mv notifications are needed, some of the links I posted in the bug report might help you design them.

Eric Paris <eparis@redhat.com> wrote:

Feel free to say anything I say, although I just realized the ONLY thing that matters for updatedb is move, so really all I told you is "I can't do that right now"

I started looking at move tonight, I know I could do something on
sys_rename but I don't see how to do notification if an inode is moved by say the nfsd server instead of by a sys_call.  I'll try to talk to the VFS maintainer and see if he has some thoughts.....

Paul Wise <pabs>
Sat 04 Jul 2009 09:07:49 AM UTC, comment #3: 

I'm sending a mail to the fsnotify/fanotify author and will report back to this bug.

Paul Wise <pabs>
Sat 04 Jul 2009 09:03:29 AM UTC, comment #2: 

I believe fsnotify will allow this:

http://lwn.net/Articles/311850/

If it doesn't, contacting the fsnotify/fanotify author and asking for the feature might be a good idea.

For inotify it appears that Linux allows the number of watches to be increased (see the -r option) when the user is root:

http://manpages.debian.net/man/1/inotifywatch

Other kernels have similar things:

http://en.wikipedia.org/wiki/FSEvents
http://www.opengroup.org/onlinepubs/9657099/
http://msdn.microsoft.com/en-us/library/aa365261(VS.85).aspx

It might be a good idea to look at FAM and gamin to find out how to do this on various kernels:

http://en.wikipedia.org/wiki/File_alteration_monitor
http://en.wikipedia.org/wiki/Gamin

Paul Wise <pabs>
Fri 03 Jul 2009 09:33:45 AM UTC, comment #1: 

I believe the problem with this approach is that it requires an open fd per directory to be monitored.   There are over 140,000 directories just on my desktop machine.  If you can think of a way to do this with a daemon that uses less than (say) 100 open file descriptors, we should talk in detail about the design.

James Youngman <jay>
Group administrator
Fri 03 Jul 2009 07:53:04 AM UTC, original submission:  

It would be nice to not have to run updatedb every night, causing large I/O bandwidth reading every filename off the disk. One way to achieve this would be locate-d, a daemon which would listen to kernel-emitted filesystem events (from dnotify, inotify or fsnotify for Linux) and then update the database at a configurable interval. By storing timestamps of the events in the database, information about the evolution of the filesystem over time could be obtained. locate could be modified to query this daemon before checking the database. Such a feature (along with fixing the other issues fixed by other implementations) could make GNU findutils the default locate implementation on most GNU/Linux distros again.

Paul Wise <pabs>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by pabs (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-09-29 jay Severity3 - Normal 1 - Wish
    2010-04-11 jay StatusNone Postponed

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code