bugfindutils - Bugs: bug #33207, updatedb (findutils-locate) has a...

 
 

bug #33207: updatedb (findutils-locate) has a memory leak on large nfs mounts, the index is not complete

Submitter:  None
Submitted:  Mon 02 May 2011 02:14:53 PM UTC
   
 
Category:  updatedb Severity:  3 - Normal
Item Group:  Wrong result Status:  Invalid
Privacy:  Public Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  4.4.2
Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 21 May 2011 06:58:55 AM UTC, comment #13: 

If other people running in the same situation:

My nfs-servers are quite old (Kernel 2.6.25) NFS-Clients until 2.6.34  worked nice together.

2.6.37 delivered with opensuse 11.4 did make this trouble, without comments in logfiles.

Then I switched to "HEAD"-kernels of opesnsuse 11.4, and now I get error-messages like this:
printserver kernel: [ 2685.444792] directory download/d contains a readdir loop.  Please contact your server vendor.  Offending cookie: 1213079068

On stdout it was be claimed there are to much symlinks.

After switching one spare nfs-server also to 2.6.37 the problems are gone.

Now I have to  to bring forward my nfs-server-updates  before I can continue  to update the nfs-clients.

In the hope, that the problem does not appear again, if both systems have the same kernel.

 





Anonymous
Fri 20 May 2011 11:57:53 PM UTC, comment #12: 

Yes, POSIX is explicit that if a directory is modified during reading, readdir() might return an entry zero times (the new entry was inserted prior to the point where readdir is traversing) or more than once (adding an entry reshuffled other entries, such that a previously read entry now falls after the point where readdir is traversing); and it is made worse by the fact that readdir is allowed to cache multiple entries rather than making a syscall on ever readdir() call.  That is, there is no atomicity with directory reading, and portable apps have to be prepared for the worst.

Eric Blake <ericb>
Group administrator
Fri 20 May 2011 11:50:29 PM UTC, comment #11: 

Thanks for letting me know.    The hint is right there in the fact that getdents is continuously returning the same number of entries, but I had assumed that, on your system, that was the number of entries that would fit into the buffer the caller supplied.   Essentially I just assumed you had a directory full of thousands of ~30 character file names.

I'm not sure offhand if POSIX implementations are allowed to return the same file name twice.  If yes, this would mean a workaround would be hard for glibc to implement.

Anyway, good to hear you figured out what was happening.   I'm going to mark this bug as invalid to signify that it is not a bug in findutils.

James Youngman <jay>
Group administrator
Thu 19 May 2011 12:26:15 PM UTC, comment #10: 

Please close this bug.

I am sure I have a kernel problem.

https://bugzilla.novell.com/show_bug.cgi?id=678123

claims to be solved, but not for me.

I will change my focus and starting to test with different kernels.




Anonymous
Wed 18 May 2011 02:20:32 PM UTC, comment #9: 

Bug 678123 - getdents64() misbehaves in NFS mounted directories


https://bugzilla.novell.com/show_bug.cgi?id=678123

could be my problem


Anonymous
Wed 18 May 2011 02:02:26 PM UTC, comment #8: 

I made the same test (but with less directories) local (opensuse11.4, findutils 4.4.2)

There was no problem. Also directories with a lot of files are successfully indexed.

I have a still a test running. (opensuse 11.3 has also Version 4.4.2 of findutils) over nfs. If that test does not fail, then the problem seems to me outside from findutils.

Then I have serious nfs/kernel-Problem.

2.6.34.7-0.7-default works with nfs and local

2.6.37.1-1.2-default has problems with find and sometimes also with ls on large dirs


Sorry for blaming the wrong.






Anonymous
Tue 17 May 2011 02:02:45 PM UTC, comment #7: 

strace of find:


getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
mremap(0x7f1be6d85000, 8392704, 16781312, MREMAP_MAYMOVE) = 0x7f1be4582000
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760


strace of ls


getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
brk(0x141f82000)                        = 0x141f82000
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
brk(0x141fa3000)                        = 0x141fa3000
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760


I am hopeless.

I afraid this is a different problem, because other nfs-clients show this dir without problems.

Anonymous
Tue 17 May 2011 08:43:23 AM UTC, comment #6: 

I went back to the big machine. My first conclusion.
The problem of incomplete index occurs on 64bit as well on 32bit.

The symptoms on the 64bit-machine are more. There is a lot of memory used, but not more than available, I don't see swapping-parties or out-of-memory-kills.

lsof | grep find

shows a lot of is own libs and two of the scanned directories.

one dir and one of its subdir

this subdir has 134610 files





(file #23415, file #23416)

Anonymous
Mon 16 May 2011 10:13:58 PM UTC, comment #5: 

Running a similar command here, I can find no evidence of memory growth:

$ ps axwwwv | egrep 'PID|find'
  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
 6867 pts/2    D+     0:05      1   213 12766  1468  0.0 /home/james/source/GNU/findutils/installed-64/bin/find-4.4.2 / ( -type d -regex \(^/mnt$\)\|\(^/cdrom$\)\|\(^/tmp$\)\|\(^/usr/tmp$\)\|\(^/var/tmp$\)\|\(^/var/spool$\)\|\(^/proc$\)\|\(^/media$\)\|\(^/sys$\) -prune ) -o -print0
 6916 pts/3    S+     0:00      0   109  7690   852  0.0 egrep PID|find

James Youngman <jay>
Group administrator
Mon 16 May 2011 12:59:47 PM UTC, comment #4: 

Sorry, for not exact following your instructions. Nevertheless I hope the infos are still usefull.


I changed to a smaller server. with normal RAM. Memory eating is relative, as long other processes are not disturbed.

But the main problem - incomplete index remains.

locate matches only hits from /nfs/data1 and /nfs/data2


I attached the ps-listing after start and some minutes later.

LANG=C df -i /nfs/data1 /nfs/data2
Filesystem     Inodes   IUsed   IFree IUse% Mounted on
xx/data1  183115776 4891655 178224121    3% /nfs/data1
xx/data2  183115776 7515404 175600372    5% /nfs/data2

This is not far from:

LANG=C locate -S
Database /var/lib/locatedb is in the GNU LOCATE02 format.
Locate database size: 156077870 bytes
All Filenames: 11973872
File names have a cumulative length of 877928437 bytes.
Of those file names,

        183476 contain whitespace,
        16 contain newline characters,
        and 71511 contain characters with the high bit set.
Compression ratio 82.22% (higher is better)


In a next try I will change the order of the netpaths and I try to follow with lsof.

(file #23409, file #23410)

Anonymous
Sat 14 May 2011 11:37:35 PM UTC, comment #3: 

Two other things that would be helpful in diagnosing this:

1. Check which directories the find process has open at two points in its execution, say 5 minutes apart.   You can do this with "lsof" or by examining /proc.   I'm mainly interested if all this memory consumption is a result of taking a long time to process one specific directory, or whether we have some kind of memory leak in which memory used for processing a directory is never freed even after we leave it.

2. Figure out if the old find implementation suffers from the same problem (they only have some of the code in common).   You can figure this out by specifying the option "--without-fts" to configure (this causes the non-fts based binary to be called "find" and the fts-based binary to be called "ftsfind").

James Youngman <jay>
Group administrator
Sat 14 May 2011 10:58:36 PM UTC, comment #2: 

a: May be the nfs-shares cause only the big amount.

b: Yes, I used "top" (mem-sorted) - "find" was the first, not sort.

Here is the output(good case):

LANG=C locate -S
Database /var/lib/locatedb is in the GNU LOCATE02 format.
Locate database size: 894160900 bytes
All Filenames: 76872632
File names have a cumulative length of 6408432554 bytes.
Of those file names,

        3809128 contain whitespace,
        35 contain newline characters,
        and 245038 contain characters with the high bit set.
Compression ratio 86.05% (higher is better)


c: the command line for find was created by the cronjob-script(made by you or by opensuse).
I will deliver it later, hoping the ps-command will not cutting it.

The nfs-shares are mounted from different servers.

d: A quite old system with 4.4.0 works. I have no versions between.


My experience with strace is very poor. I afraid the file will be very big and my conclusions will be very small. I give it a try within the next days.


Thomas

Anonymous
Sat 14 May 2011 09:54:31 PM UTC, comment #1: 

I think it's unlikely that this problem is NFS-specific.  

Are you sure it is find which is consuming all the memory?   Not sort?   How many files, in total, are involved?  (you can figure this out by running locate -S on a correct database)

What is the command line of the find process which is using up all the system's memory?   Do you get the same effect if you run the same find command on the NFS server directly, with the local filesystem paths (I suppose you will need to adjust the directory names)?

What is the newest findutils 4.4.x release which is not affected by this problem?

Does a system call trace (with e.g. strace on Linux or its equivalent on the NFS server) reveal any difference between the affected and unaffected releases, or any otherwise interesting behaviour?

James Youngman <jay>
Group administrator
Mon 02 May 2011 02:14:53 PM UTC, original submission:  

updatedb uses all RAM (128GB) but the result is useless, nearly empty



former versions did work - not eating all memory, resulting in a complete index

further information is in

https://bugzilla.novell.com/show_bug.cgi?id=689916





Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #23415:  ps-axuwwwf-output-110517100542.log added by None (2KiB - text/x-log)
file #23416:  top.txt added by None (1KiB - text/plain)
file #23409:  updatedb-find.txt added by None (2KiB - text/plain - process listing)
file #23410:  updatedb-find2.txt added by None (2KiB - text/plain - process listing)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ericb (Posted a comment)
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-05-20 jay StatusNone Invalid
        Open/ClosedOpen Closed
    2011-05-17 None Attached File- Added ps-axuwwwf-output-110517100542.log, #23415
        Attached File- Added top.txt, #23416
    2011-05-16 None Attached File- Added updatedb-find.txt, #23409
        Attached File- Added updatedb-find2.txt, #23410

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code