bugfindutils - Bugs: bug #47261, find produces two different...

 
 

bug #47261: find produces two different results for the same command

Submitted by:  None
Submitted on:  Thu 25 Feb 2016 07:24:54 PM UTC  
 
Category: findSeverity: 4 - Important
Item Group: Wrong resultStatus: Works For Me
Privacy: PublicAssigned to: James Youngman <jay>
Originator Name: Len SchulwitzOriginator Email: -unavailable-
Open/Closed: OpenRelease: 4.4.2
Fixed Release: 4.6.0

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 24 May 2016 05:16:36 AM UTC, comment #10:

Thanks James for all your help. The -noleaf option does in fact remove the error for oldfind.

Len Schulwitz <schulwitz>
Mon 23 May 2016 07:35:22 PM UTC, comment #9:

Thanks for the update. Given what the error message says, I would guess that version 4.4.2 with -noleaf would avoid the problem. In any case, as the error message says, the bug is in the file system (which is producing an incorrect st_nlink value for a directory) rather than, it seems, in find.

If -noleaf doesn't mitigate the problem with findutils version 4.4.2, and you'd like me to help you further investigate, please re-open this bug. Thanks.

(I chose the bug status "Works for Me" to indicate that the problem lies in the filesystem driver. I normally use "Invalid" for this purpose but that didn't seem to appropriate here)

James Youngman <jay>
Project AdministratorIn charge of this item.
Fri 20 May 2016 08:32:46 AM UTC, comment #8:

Sorry it's been so long, but, it turns out the bug is not present when I compile it for 4.6.0, so problem solved!

Also, re-compiling for 4.4.2 still produces the bug with regular "find", but "oldfind" finds the file on the first try just fine, albeit with the following error: "WARNING: Hard link count is wrong for `/proc/fs' (saw only st_nlink=8 but we already saw 6 subdirectories): this may be a bug in your file system driver. Automatically turning on find's -noleaf option. Earlier results may have failed to include directories that should have been searched."

Also, in answer to your question, 4.4.2 find fails the first time, but succeeds every time after that. Also, the find problem occurs with -name and -regex in the same fashion.

Len Schulwitz <schulwitz>
Wed 02 Mar 2016 03:39:08 PM UTC, comment #7:

I wrote a script to determine what all the stat() calls are looking at, and it turns out my guess was wrong. There are a lot of files that find doesn't find, and I doubt it's because they're not in the filesystem. E.g., all of /etc is missing.

There are the expectable differences in /proc. But other than that, the first run ends at:

newfstatat(/sys/kernel/debug/tracing/events/task/task_newtask)
newfstatat(/sys/kernel/debug/tracing/events/task/task_rename)
newfstatat(/sys/kernel/debug/tracing/events/raw_syscalls)
newfstatat(/sys/kernel/debug/tracing/events/raw_syscalls/sys_enter)
newfstatat(/sys/kernel/debug/tracing/events/raw_syscalls/sys_exit)

but the second run continues with:

newfstatat(/sys/kernel/debug/tracing/options)
newfstatat(/sys/kernel/debug/tracing/instances)
newfstatat(/sys/kernel/debug/tracing/per_cpu)
newfstatat(/sys/kernel/debug/tracing/per_cpu/cpu0)
newfstatat(/sys/kernel/debug/tracing/trace_stat)
newfstatat(/sys/kernel/debug/usb)

and goes on to walk through /usr and /etc.

Looking at the end of the first run shows no obvious reason why find would terminate prematurely.

The script I used is:

#! /bin/perl

use strict;

my($debug) = 0;
my($cwd) = '';
my(@file);

while (<>) {
chomp;
print "\$_ = '$_' \n" if $debug;
if (m/^open\("(.*)", O_[^)]+\) += (\d+)$/) {
my($name) = $1;
my($fd) = $2;
$file[$fd] = &append($cwd, $name);
print "\$file[$fd] = '$file[$fd]'\n" if $debug;
} elsif (m/^openat\([^,]+, "(.*)", O_[^)]+\) += (\d+)$/) {
my($name) = $1;
my($fd) = $2;
$file[$fd] = &append($cwd, $name);
print "\$file[$fd] = '$file[$fd]'\n" if $debug;
} elsif (m/^fchdir\((\d+)\)/) {
my($fd) = $1;
$cwd = $file[$fd];
print "\$cwd = '$cwd'\n" if $debug;
} elsif (m/^newfstatat\([^,]+, "(.*)", {/) {
my($name) = $1;
$name = &append($cwd, $name);
$name = '/' if $name eq '';
print "newfstatat(", $name, ")\n";
}
}

sub append {
my($cwd, $name) = @_;

if ($name eq '.') {
$name = $cwd;
} elsif ($name eq '..') {
$name = $cwd;
$name =~ s%/[^/]*$%%;
} elsif ($name !~ m%^/%) {
$name = $cwd . '/' . $name;
} elsif ($name eq '/') {
$name = '';
}

return $name;
}

Dale Worley <worley>
Tue 01 Mar 2016 07:55:27 PM UTC, comment #6:

You can strip the memory addresses from the traces (which vary) by doing:

<find1.txt sed -e 's/0x[0-9a-f]/0x/g' >f1
<find2.txt sed -e 's/0x[0-9a-f]/0x/g' >f2

Then you can compare the traces directly with "diff -u f1 f2".

Looking at the results of that, I see that the major difference is that find2 has a huge block of fstat() calls, starting with "options", that are added after all the processing that is seen in find1. I suspect that some section of the filesystem was added or became visible between the two runs.

You could write a simple script to trace through all the open() and fchdir() calls to figure out what the full path of "options", etc. are, but I didn't put in the work.

Dale Worley <worley>
Fri 26 Feb 2016 04:11:56 PM UTC, comment #5:

Could it be a case of automounting kicking in? If your file is hidden behind a mount point, maybe the first run triggers the mount to automount but doesn't descend, and the second one is able to descend into the (now-mounted) location?

Eric Blake <ericb>
Project Administrator
Fri 26 Feb 2016 12:55:16 PM UTC, comment #4:

I wonder if something is special about /sys. What happens if you invoke find like this:

sudo find / \( -path /sys -prune \) , -iname 'firefox_binary.py'

Also do you definitely need to run "find" the first time to see the difference in behaviour? That is, if you boot, wait two minutes, and then run the find command, do you get a match for firefox_binary.py or not? To get the second behaviour (where we find the match) do you have to run the same find command, or can it be different in any way?

James Youngman <jay>
Project AdministratorIn charge of this item.
Fri 26 Feb 2016 12:50:48 PM UTC, comment #3:

It's also interesting to see the contents of the struct stat instances we get for the stat calls. Could you use "strace -v" rather than plan strace please?

James Youngman <jay>
Project AdministratorIn charge of this item.
Fri 26 Feb 2016 12:42:35 PM UTC, comment #2:

Some other things:

1. what file systems types are being used for the affected machine?

2. I assume that the same problem exists with -name as with -iname, but could you check? What about -regex?

3. You say the problem happens "the first time". If you try it a third (fourth, ...) time does it always succeed? Do you have to reboot the machine to get it back into a state where the problem can be reproduced?

James Youngman <jay>
Project AdministratorIn charge of this item.
Fri 26 Feb 2016 12:38:37 PM UTC, comment #1:

This is very interesting, and of course surprising.

Thanks for the comprehensive diagnositics. First of all, please try to reproduce the problem with the current stable release of find. You should be able to download version 4.6.0 from ftp.gnu.org/pub/gnu/findutils (you would need to build it yourself, I trust that this would not be a problem).

Given the nature of the problem, I'm still interested in tracking down the problem with version 4.4.2 even if it's not reproducible in 4.6.0.

Also, building 4.4.2 from source should generate two binaries. One is called find and is the one you will have used for your video etc. The other is called "oldfind". With version 4.4.2 some distributions include oldfind and some do not. I'd be interested in whether the problem is reproducible with oldfind (the two should be compatible and produce the same results but the internals are very different).

You can also get additional information about what find is doing with the -D option. Try for example:

oldfind -D search,stat,tree,opt / -iname 'firefox_binary.py'
find -D search,stat,tree,opt / -iname 'firefox_binary.py'

You can get additional information about the debug options like this:

find -D help

Another thing to check is whether something is getting fooled by an incorrect st_nlink count on a directory. Try passing the option -noleaf.

James Youngman <jay>
Project AdministratorIn charge of this item.
Thu 25 Feb 2016 07:24:54 PM UTC, original submission:

On a Raspberry Pi B+ running Raspbian jessie, I run this command: "sudo find / -iname 'firefox_binary.py'" to find a file I know exists. The first time I run it, find exits without any errors, and 0 results. The second time I run it, only seconds later, find locates the file. Here is a video documenting the behavior: youtube.com/watch?v=qDB_50RwWa8 In addition, I opened this up as a question at http://unix.stackexchange.com/questions/265571/how-is-it-that-the-same-find-command-can-give-two-different-results?noredirect=1#comment459672_265571 and it seems to have stumped all the unix gurus on there, who recommended that I submit a bug report. Here's the strace of the first run (fails): http://lenschulwitz.com/find1.txt and the strace of the second run (succeeds): http://lenschulwitz.com/find2.txt

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by schulwitz (Posted a comment)
  • -unavailable- added by worley (Posted a comment)
  • -unavailable- added by ericb (Posted a comment)
  • -unavailable- added by jay (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 23 May 2016 07:35:22 PM UTCjayStatusNeed Info=>Works For Me
      Fixed ReleaseNone=>4.6.0
    Fri 26 Feb 2016 12:38:37 PM UTCjaySeverity3 - Normal=>4 - Important
      StatusNone=>Need Info
      Assigned toNone=>jay

    Back to the top


    Powered by Savane 3.1-cleanup1