Sat 21 May 2011 06:58:55 AM UTC, comment #13:
If other people running in the same situation:
My nfs-servers are quite old (Kernel 2.6.25) NFS-Clients until 2.6.34 worked nice together.
2.6.37 delivered with opensuse 11.4 did make this trouble, without comments in logfiles.
Then I switched to "HEAD"-kernels of opesnsuse 11.4, and now I get error-messages like this:
printserver kernel: [ 2685.444792] directory download/d contains a readdir loop. Please contact your server vendor. Offending cookie: 1213079068
On stdout it was be claimed there are to much symlinks.
After switching one spare nfs-server also to 2.6.37 the problems are gone.
Now I have to to bring forward my nfs-server-updates before I can continue to update the nfs-clients.
In the hope, that the problem does not appear again, if both systems have the same kernel.
|
Tue 17 May 2011 02:02:45 PM UTC, comment #7:
strace of find:
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
mremap(0x7f1be6d85000, 8392704, 16781312, MREMAP_MAYMOVE) = 0x7f1be4582000
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
getdents64(4, /* 819 entries */, 32768) = 32760
strace of ls
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
brk(0x141f82000) = 0x141f82000
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
brk(0x141fa3000) = 0x141fa3000
getdents64(3, /* 819 entries */, 32768) = 32760
getdents64(3, /* 819 entries */, 32768) = 32760
I am hopeless.
I afraid this is a different problem, because other nfs-clients show this dir without problems.
|
Mon 16 May 2011 12:59:47 PM UTC, comment #4:
Sorry, for not exact following your instructions. Nevertheless I hope the infos are still usefull.
I changed to a smaller server. with normal RAM. Memory eating is relative, as long other processes are not disturbed.
But the main problem - incomplete index remains.
locate matches only hits from /nfs/data1 and /nfs/data2
I attached the ps-listing after start and some minutes later.
LANG=C df -i /nfs/data1 /nfs/data2
Filesystem Inodes IUsed IFree IUse% Mounted on
xx/data1 183115776 4891655 178224121 3% /nfs/data1
xx/data2 183115776 7515404 175600372 5% /nfs/data2
This is not far from:
LANG=C locate -S
Database /var/lib/locatedb is in the GNU LOCATE02 format.
Locate database size: 156077870 bytes
All Filenames: 11973872
File names have a cumulative length of 877928437 bytes.
Of those file names,
183476 contain whitespace,
16 contain newline characters,
and 71511 contain characters with the high bit set.
Compression ratio 82.22% (higher is better)
In a next try I will change the order of the netpaths and I try to follow with lsof.
(file #23409, file #23410)
|
Sat 14 May 2011 10:58:36 PM UTC, comment #2:
a: May be the nfs-shares cause only the big amount.
b: Yes, I used "top" (mem-sorted) - "find" was the first, not sort.
Here is the output(good case):
LANG=C locate -S
Database /var/lib/locatedb is in the GNU LOCATE02 format.
Locate database size: 894160900 bytes
All Filenames: 76872632
File names have a cumulative length of 6408432554 bytes.
Of those file names,
3809128 contain whitespace,
35 contain newline characters,
and 245038 contain characters with the high bit set.
Compression ratio 86.05% (higher is better)
c: the command line for find was created by the cronjob-script(made by you or by opensuse).
I will deliver it later, hoping the ps-command will not cutting it.
The nfs-shares are mounted from different servers.
d: A quite old system with 4.4.0 works. I have no versions between.
My experience with strace is very poor. I afraid the file will be very big and my conclusions will be very small. I give it a try within the next days.
Thomas
|
Sat 14 May 2011 09:54:31 PM UTC, comment #1:
I think it's unlikely that this problem is NFS-specific.
Are you sure it is find which is consuming all the memory? Not sort? How many files, in total, are involved? (you can figure this out by running locate -S on a correct database)
What is the command line of the find process which is using up all the system's memory? Do you get the same effect if you run the same find command on the NFS server directly, with the local filesystem paths (I suppose you will need to adjust the directory names)?
What is the newest findutils 4.4.x release which is not affected by this problem?
Does a system call trace (with e.g. strace on Linux or its equivalent on the NFS server) reveal any difference between the affected and unaffected releases, or any otherwise interesting behaviour?
|