This can be fixed from the AFS side; the stats do not need to take that long (we can avoid the throttling). That requires changes in the OpenAFS client that we're still discussing, though.

In the meantime, or to support older releases, if you want to work around this, the stat()s should go a lot faster if you hold open the directory while stat()ting the contents. You can try this out without modifying find if you just have another process opendir() and not close the dir.

Would it be acceptable for find to hold open the directory while examining the contents, perhaps only if we determine we are in AFS? This is of course only something I can say will probably work for OpenAFS; other client implementations may do different things.

Daniel:

I'm not sure there is anything to fix from the AFS side.  The problem that is being faced is that the process does not have the ability to read status data about the objects in the directory.  All AFS provides is the name.  Directories can be determined by the vnode having an odd value so the client is able to set that value for directory objects.  However, any vnode with an even value could be a mount point (which should appear as a directory), a file (which should not), or a symlink (which can appear as either depending on what it refers to).  The client therefore has no ability to set the type of an object in a status query when the process has no read permission.  Therefore, the d_type is set to DT_UNKNOWN.  The type truly is unknown.  To interpret this as "not a directory" would be incorrect.  In the AFS case, d_type == DT_UNKNOWN really means "access denied".

I suspect that the slow down you are seeing is caused by throttling on the file server when a client continuously sends FetchStatus requests which end in an Access Denied error.  After N RPCs that end in an error the file server begins to throttle the client to prevent it from DoS attacking the server.

Can you explain why you believe the issue to be timing related?

As for there not being public cells that export directories with "system:anyuser l" ACLs assigned, there are plenty of them.  Finding an example case would not be hard and even if it were hard, putting up a test volume with such a case in the grand.central.org or your-file-system.com or andrew.cmu.edu or ir.stanford.edu or athena.mit.edu cells would be trivial to do.

Jeffrey Altman

I'm not sure that an account or even a special system is required to be able to test this issue.  The /afs name space is globally public and most government and university deployments have significant quantities of data that is publicly readable by anonymous users.  Installing the openafs client package debian / ubuntu / rhel /fedora packages should be all that is required to setup a test system.

Yes, if I had access to an AFS system for development/testing, yes, I'd work on this.   Without it, no.  

It sounds brutal to baldly state it, but this problem has existed for years.   Yet still nobody has come forward volunteering to donate a patch and maintain it.   Nobody has come forward offering a suitable test/development system.   Given the absence of both those things, working on this hardly looks like a good time investment.

Mocking in a sufficiently realistic AFS emulation layer for development and regession testing is probably more work itself than just developing a fix, I would think.

I'm not blocking a fix here, I'm just pointing out that the prerequisites for a fix do not seem to be forthcoming.

No sign of a working solution so far; unless somebody has a useful suggestion in the mean time, I'll close this bug in a week or so.

If we're going to move forward with this bug, we need an approach that works with both oldfind and ftsfind.

Kevin Dalley checked this change in in 2000:

revision 1.6
date: 2000-04-05 08:26:24 +0100;  author: kevind;  state: Exp;  lines: +14 -11;

• configure.in: add Galition and Estonian languages.

Miscellaneous other fixes.

$ cvs -z3 diff -upN  -r1.6 -r1.7 configure.in
Index: configure.in
===================================================================
RCS file: /cvsroot/findutils/findutils/Attic/configure.in,v
retrieving revision 1.6
retrieving revision 1.7
diff -u -p -r1.6 -r1.7
--- configure.in        5 Apr 2000 07:26:24 -0000       1.6
+++ configure.in        12 Apr 2000 08:51:02 -0000      1.7
@@ -1,21 +1,21 @@
 dnl Process this file with autoconf to produce a configure script.
 AC_INIT(find/pred.c)
 AM_CONFIG_HEADER(config.h)
+AC_CANONICAL_HOST
 
 dnl Set of available languages.
 ALL_LINGUAS=" de es fr it ko nl pl pt_BR ru sv gl et"
 
-AM_INIT_AUTOMAKE(findutils, 4.1.4)
+AM_INIT_AUTOMAKE(findutils, 4.1.5)
 
 AC_SUBST(INCLUDES)dnl
 AC_ARG_ENABLE(id-cache,
 [  --enable-id-cache       cache all UIDs & GIDs; avoid if using NIS or Hesiod],
-  AC_DEFINE(CACHE_IDS))
-AC_ARG_WITH(afs,
-[  --with-afs              support -fstype afs],
-[  AC_DEFINE(AFS)
-  CPPFLAGS="$CPPFLAGS -I/usr/afsws/include"
-  LIBS="$LIBS -L/usr/afsws/lib -L/usr/afsws/lib/afs -lsys -lrx -llwp"])


I don't know what the motivation behind the change was, though.

> You would be open to a compile-time option to link against the AFS libraries? (

Yes, absolutely.   That's what configure is for.

> I was under the impression that this existed in past versions, and was removed at some point.)

No, it is still there:

#ifdef AFS
#include <netinet/in.h>
#include <afs/venus.h>
#if _STDC_
/* On SunOS 4, afs/vice.h defines this to rely on a pre-ANSI cpp.  */
#undef _VICEIOCTL
#define _VICEIOCTL(id)  ((unsigned int ) _IOW('V', id, struct ViceIoctl))
#endif
#ifndef _IOW
/* AFS on Solaris 2.3 doesn't get this definition.  */
#include <sys/ioccom.h>
#endif

static int
in_afs (char *path)
{
  static char space[2048];
  struct ViceIoctl vi;

  vi.in_size = 0;
  vi.out_size = sizeof (space);
  vi.out = space;

  if (pioctl (path, VIOC_FILE_CELL_NAME, &vi, 1)
      && (errno == EINVAL || errno == ENOENT))
        return 0;
  return 1;
}
#endif /* AFS */

static char *
file_system_type_uncached (const struct stat *statp, const char *path)
{
  struct mount_entry *entries, *entry;
  char *type;

  (void) path;
 
#ifdef AFS
  if (in_afs(path))
    {
      fstype_known = 1;
      return xstrdup("afs");
    }
#endif
...

> If we use libafs, we may as well go all the way and have find(1) check the permissions on each directory (if in AFS) and cue off of that.

You may find this hard to integrate with fts.  

However, if AFS volumes cannot contain anything that isn't AFS, you may not need to provide such integration.  Is it possible to mount a non-AFS filesystem at a mount point within the AFS filesystem?    What about bind mounts?

On the basis of an earlier conversation with Jeffrey Hutzelman, I would assume that on systems where AFS is available (i.e. AFS libraries were detected at configure time) we can use a pioctl() call to check whether we're inside an AFS volume each time we observe a change in st_dev.

Such an approach would be quite efficient since a pioctl call could occur a maximum of N*M times, where N is the number of mounted filesystems and M is the number of start-points listed on find's command line.   Even the O(N*M) figure should be reducible to O(N) with the use of a cache.

I get your first paragraph, but I don't understand what you're responding to in the second and third. Let me try this again:

We know what the behavior needs to be in order to avoid the 3-second-stat()s-on-AFS problem: if opendir() returns a d_type of DT_UNKNOWN for an entry, then give up trying to determine what it is (whether -L was given or not). However, we only want to do this on AFS, because some other filesystems can and do return DT_UNKNOWN and yet allow us to stat() the entry perfectly well.

The question, then, is how to make find(1) behave that way only when needed/appropriate. I see two viable options:


1. Add a new option---let's call it "-nostatdtu", i.e. "don't stat() entries with d_type=DT_UNKNOWN"---that goes right next to -noleaf in the man page. The user would then give this option when slow searches in AFS-space make it necessary.

Pros: find(1) remains filesystem-agnostic; this behavior can potentially be used to speed up searches on other filesystems with similar d_type behavior

Cons: The option is difficult to explain to ordinary users; user has to be aware of this option and when to use it


2. Have find(1) check each entry to see if it has a canonical path under /afs, and if so, change to the workaround behavior.

(The value of "/afs" would be hard-coded, not user-specified, as this is the non-aribitrary AFS mount point. The value might, at most, be a compile-time option.)

Pros: Transparent to the user; a hypothetical invocation of "find /" that goes into /afs and /brokenfs (with unreliable d_types) works correctly

Cons: Bad in the rare case that /afs isn't actually an AFS filesystem; find(1) is no longer filesystem-agnostic


It could be 1, or 2, or both, or 2 enabled with a compile-time option, or perhaps some other approach altogether. What would you like to see in a submitted patch?

Please see comment #2. The question is, "Should find(1) have an option that causes it to trust d_type=DT_UNKNOWN, or should it specifically check for AFS and do the same without user intervention when working inside that filesystem?"

I recognize that find(1) can't always trust the value of d_type, so I need to know what specifically should cause it to do so in situations when it is working under AFS. (I can't put together a patch until I know what behavior I'm implementing.)

Should there be an option, along the lines of -noleaf, that causes find(1) to believe what d_type says for unknown entries? Would you rather have find(1) autodetect AFS in some way to adapt its behavior on-the-fly? (IIRC, this would basically amount to checking whether the canonical path is under /afs; that's supposed to be the "official" method.)

d_type is DT_UNKNOWN only for non-directories. AFS "l" access does let you see if a name is a directory, so that you can look inside it (potentially with greater access permissions).

I could put together a patch; I just need to know what the acceptable behavior should be.

I have a script intended to generate a file listing of several users' home directories, stored in AFS. The listing should be fairly bare-bones: no file attributes of any kind, not even a distinction between regular files vs. symlinks vs. other file types, just names and the directories containing them.

Consistent with that, the script runs under a restricted permission set where it can only list directories. Where an AFS admin would have all "rlidwka" bits set, this script would only get "l" (list) access.

find(1) takes a very long time scanning directory trees under these conditions. Here is what "strace -r -p <find-pid>" says is going on:

...
     3.016894 lstat("Makefile", 0x7fff950fb590) = -1 EACCES (Permission denied)
     3.001573 newfstatat(AT_FDCWD, "abuzz_r3_linguistics.tex", 0x664278, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied)
     3.019501 lstat("abuzz_r3_linguistics.tex", 0x7fff950fb590) = -1 EACCES (Permission denied)
     3.001665 newfstatat(AT_FDCWD, "abuzz_r3.tex", 0x664398, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied)
     3.020538 lstat("abuzz_r3.tex", 0x7fff950fb590) = -1 EACCES (Permission denied)
     3.001657 newfstatat(AT_FDCWD, "abuzz_r3_status.tex", 0x664b48, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied)
     3.023643 lstat("abuzz_r3_status.tex", 0x7fff950fb590) = -1 EACCES (Permission denied)
     3.001544 newfstatat(AT_FDCWD, "abuzz_r3_misc.tex", 0x664c68, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied)
     3.019582 lstat("abuzz_r3_misc.tex", 0x7fff950fb590) = -1 EACCES (Permission denied)

Each of those failed stat calls is taking over three seconds! Per file! And this is with 4.4.0 compiled with d_type and leaf optimization enabled.

find(1) should not be stat()ing those files to begin with. I didn't give it a -type predicate, and it can already tell which ones are directories via d_type, so why go to the trouble?

As a potential (new) optimization, I vaguely recall that when doing readdir() for an AFS file with only "l" access, d_type would have an value corresponding to "unknown." In which case, find(1) can assume that a stat() is pointless because it would invariably fail. The thing to remember is that, with "l" access only, AFS will allow you to see the name, but will tenaciously withhold all other info, even file type (with the exception of directories). The fix is, effectively, to handle this situation in a more graceful way.