bugfindutils - Bugs: bug #47261, find produces two different...

 
 

bug #47261: find produces two different results for the same command

Submitter:  None
Submitted:  Thu 25 Feb 2016 07:24:54 PM UTC
   
 
Category:  find Severity:  4 - Important
Item Group:  Wrong result Status:  Works For Me
Privacy:  Public Assigned to:  jay
Originator Name:  Len Schulwitz Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  4.4.2
Fixed Release:  4.6.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 24 May 2016 05:16:36 AM UTC, comment #10: 

Thanks James for all your help.  The -noleaf option does in fact remove the error for oldfind.

Len Schulwitz <schulwitz>
Mon 23 May 2016 07:35:22 PM UTC, comment #9: 

Thanks for the update.   Given what the error message says, I would guess that version 4.4.2 with -noleaf would avoid the problem.   In any case, as the error message says, the bug is in the file system (which is producing an incorrect st_nlink value for a directory) rather than, it seems, in find.

If -noleaf doesn't mitigate the problem with findutils version 4.4.2, and you'd like me to help you further investigate, please re-open this bug.  Thanks.

(I chose the bug status "Works for Me" to indicate that the problem lies in the filesystem driver.  I normally use "Invalid" for this purpose but that didn't seem to appropriate here)

James Youngman <jay>
Group administrator
Fri 20 May 2016 08:32:46 AM UTC, comment #8: 

Sorry it's been so long, but, it turns out the bug is not present when I compile it for 4.6.0, so problem solved!

Also, re-compiling for 4.4.2 still produces the bug with regular "find", but "oldfind" finds the file on the first try just fine, albeit with the following error: "WARNING: Hard link count is wrong for `/proc/fs' (saw only st_nlink=8 but we already saw 6 subdirectories): this may be a bug in your file system driver.  Automatically turning on find's -noleaf option.  Earlier results may have failed to include directories that should have been searched."

Also, in answer to your question, 4.4.2 find fails the first time, but succeeds every time after that. Also, the find problem occurs with -name and -regex in the same fashion.

Len Schulwitz <schulwitz>
Wed 02 Mar 2016 03:39:08 PM UTC, comment #7: 

I wrote a script to determine what all the stat() calls are looking at, and it turns out my guess was wrong.  There are a lot of files that find doesn't find, and I doubt it's because they're not in the filesystem.  E.g., all of /etc is missing.

There are the expectable differences in /proc.  But other than that, the first run ends at:

newfstatat(/sys/kernel/debug/tracing/events/task/task_newtask)
newfstatat(/sys/kernel/debug/tracing/events/task/task_rename)
newfstatat(/sys/kernel/debug/tracing/events/raw_syscalls)
newfstatat(/sys/kernel/debug/tracing/events/raw_syscalls/sys_enter)
newfstatat(/sys/kernel/debug/tracing/events/raw_syscalls/sys_exit)

but the second run continues with:

newfstatat(/sys/kernel/debug/tracing/options)
newfstatat(/sys/kernel/debug/tracing/instances)
newfstatat(/sys/kernel/debug/tracing/per_cpu)
newfstatat(/sys/kernel/debug/tracing/per_cpu/cpu0)
newfstatat(/sys/kernel/debug/tracing/trace_stat)
newfstatat(/sys/kernel/debug/usb)

and goes on to walk through /usr and /etc.

Looking at the end of the first run shows no obvious reason why find would terminate prematurely.

The script I used is:

#! /bin/perl

use strict;

my($debug) = 0;
my($cwd) = '';
my(@file);

while (<>) {
    chomp;
    print "\$_ = '$_' \n" if $debug;
    if (m/^open\("(.*)", O_[^)]+\) += (\d+)$/) {
my($name) = $1;
my($fd) = $2;
$file[$fd] = &append($cwd, $name);
print "\$file[$fd] = '$file[$fd]'\n" if $debug;
    } elsif (m/^openat\([^,]+, "(.*)", O_[^)]+\) += (\d+)$/) {
my($name) = $1;
my($fd) = $2;
$file[$fd] = &append($cwd, $name);
print "\$file[$fd] = '$file[$fd]'\n" if $debug;
    } elsif (m/^fchdir\((\d+)\)/) {
my($fd) = $1;
$cwd = $file[$fd];
print "\$cwd = '$cwd'\n" if $debug;
    } elsif (m/^newfstatat\([^,]+, "(.*)", {/) {
my($name) = $1;
$name = &append($cwd, $name);
$name = '/' if $name eq '';
print "newfstatat(", $name, ")\n";
    }
}

sub append {
    my($cwd, $name) = @_;

    if ($name eq '.') {
$name = $cwd;
    } elsif ($name eq '..') {
$name = $cwd;
$name =~ s%/[^/]*$%%;
    } elsif ($name !~ m%^/%) {
$name = $cwd . '/' . $name;
    } elsif ($name eq '/') {
$name = '';
    }
   
    return $name;
}

Dale Worley <worley>
Tue 01 Mar 2016 07:55:27 PM UTC, comment #6: 

You can strip the memory addresses from the traces (which vary) by doing:

     <find1.txt sed -e 's/0x[0-9a-f]*/0x*/g' >f1
     <find2.txt sed -e 's/0x[0-9a-f]*/0x*/g' >f2

Then you can compare the traces directly with "diff -u f1 f2".

Looking at the results of that, I see that the major difference is that find2 has a huge block of fstat() calls, starting with "options", that are added after all the processing that is seen in find1.  I suspect that some section of the filesystem was added or became visible between the two runs.

You could write a simple script to trace through all the open() and fchdir() calls to figure out what the full path of "options", etc. are, but I didn't put in the work.

Dale Worley <worley>
Fri 26 Feb 2016 04:11:56 PM UTC, comment #5: 

Could it be a case of automounting kicking in? If your file is hidden behind a mount point, maybe the first run triggers the mount to automount but doesn't descend, and the second one is able to descend into the (now-mounted) location?

Eric Blake <ericb>
Group administrator
Fri 26 Feb 2016 12:55:16 PM UTC, comment #4: 

I wonder if something is special about /sys.   What happens if you invoke find like this:


sudo find / \( -path /sys -prune \) , -iname  'firefox_binary.py'

Also do you definitely need to run "find" the first time to see the difference in behaviour?  That is, if you boot, wait two minutes, and then run the find command, do you get a match for firefox_binary.py or not?   To get the second behaviour (where we find the match) do you have to run the same find command, or can it be different in any way?

James Youngman <jay>
Group administrator
Fri 26 Feb 2016 12:50:48 PM UTC, comment #3: 

It's also interesting to see the contents of the struct stat instances we get for the stat calls.   Could you use "strace -v" rather than plan strace please?

James Youngman <jay>
Group administrator
Fri 26 Feb 2016 12:42:35 PM UTC, comment #2: 

Some other things:

1. what file systems types are being used for the affected machine?

2. I assume that the same problem exists with -name as with -iname, but could you check?   What about -regex?

3. You say the problem happens "the first time".   If you try it a third (fourth, ...) time does it always succeed?  Do you have to reboot the machine to get it back into a state where the problem can be reproduced?

James Youngman <jay>
Group administrator
Fri 26 Feb 2016 12:38:37 PM UTC, comment #1: 

This is very interesting, and of course surprising.

Thanks for the comprehensive diagnositics.   First of all, please try to reproduce the problem with the current stable release of find.  You should be able to download version 4.6.0 from ftp.gnu.org/pub/gnu/findutils (you would need to build it yourself, I trust that this would not be a problem).

Given the nature of the problem, I'm still interested in tracking down the problem with version 4.4.2 even if it's not reproducible in 4.6.0.

Also, building 4.4.2 from source should generate two binaries.  One is called find and is the one you will have used for your video etc.  The other is called "oldfind".  With version 4.4.2 some distributions include oldfind and some do not.  I'd be interested in whether the problem is reproducible with oldfind (the two should be compatible and produce the same results but the internals are very different).

You can also get additional information about what find is doing with the -D option.  Try for example:

oldfind  -D search,stat,tree,opt / -iname 'firefox_binary.py'
find  -D search,stat,tree,opt / -iname 'firefox_binary.py'

You can get additional information about the debug options like this:

find -D help

Another thing to check is whether something is getting fooled by an incorrect st_nlink count on a directory.  Try passing the option -noleaf.





James Youngman <jay>
Group administrator
Thu 25 Feb 2016 07:24:54 PM UTC, original submission:  

On a Raspberry Pi B+ running Raspbian jessie, I run this command: "sudo find / -iname 'firefox_binary.py'" to find a file I know exists.  The first time I run it, find exits without any errors, and 0 results.  The second time I run it, only seconds later, find locates the file.  Here is a video documenting the behavior: youtube.com/watch?v=qDB_50RwWa8  In addition, I opened this up as a question at http://unix.stackexchange.com/questions/265571/how-is-it-that-the-same-find-command-can-give-two-different-results?noredirect=1#comment459672_265571 and it seems to have stumped all the unix gurus on there, who recommended that I submit a bug report.  Here's the strace of the first run (fails): http://lenschulwitz.com/find1.txt and the strace of the second run (succeeds): http://lenschulwitz.com/find2.txt

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by schulwitz (Posted a comment)
  • -email is unavailable- added by worley (Posted a comment)
  • -email is unavailable- added by ericb (Posted a comment)
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-05-23 jay StatusNeed Info Works For Me
        Fixed ReleaseNone 4.6.0
    2016-02-26 jay Severity3 - Normal 4 - Important
        StatusNone Need Info
        Assigned toNone jay

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code