bugfindutils - Bugs: bug #24140, Painfully slow find(1) in...


bug #24140: Painfully slow find(1) in list-permission-only AFS paths

Submitted by:  Daniel Richard G. <iskunk>
Submitted on:  Tue 26 Aug 2008 07:03:01 AM UTC  
Category:  find Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  James Youngman <jay>
Originator Name:  Open/Closed:  Open
Release:  4.4.0 Fixed Release:  None

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Thu 23 Aug 2012 10:09:19 PM UTC, comment #27:

As you approach a testing state, please drop me a note. I have access to several AFS cells and can configure a subtree appropriate for testing rl vs. l permissions.

Steve Simmons <scs>
Wed 11 May 2011 09:47:45 AM UTC, comment #26:

Yes, I think an example VM would work. Which type would you be able to construct? I usually use VirtualBox-OSE or Xen.

James Youngman <jay>
Project AdministratorIn charge of this item.
Sat 13 Nov 2010 01:32:08 AM UTC, comment #25:

> Nobody has come forward offering a suitable test/development system.

I can provide a VM image with AFS installed, and an AFS account (although this is entirely not required for testing this issue). The VM image can of course be set up for anyone to download and use but obviously I'm not going to give away accounts willy-nilly.

I assume that will meet the requirement above.

Derrick Brashear <shadow>
Wed 14 Apr 2010 03:44:12 PM UTC, comment #24:

This can be fixed from the AFS side; the stats do not need to take that long (we can avoid the throttling). That requires changes in the OpenAFS client that we're still discussing, though.

In the meantime, or to support older releases, if you want to work around this, the stat()s should go a lot faster if you hold open the directory while stat()ting the contents. You can try this out without modifying find if you just have another process opendir() and not close the dir.

Would it be acceptable for find to hold open the directory while examining the contents, perhaps only if we determine we are in AFS? This is of course only something I can say will probably work for OpenAFS; other client implementations may do different things.

Andrew Deason <adeason>
Wed 14 Apr 2010 04:46:33 AM UTC, comment #23:

Hmm. So looking at d_type by itself yields normal directories, but not mount points, or directory symlinks. Symlinks aren't such a big deal, but I notice that even though mount points read as DT_UNKNOWN, you can still stat() them. Not that it does you much good when you're scanning zillions of entries.

(Part of the goal here is to allow find(1) to search through l-only-space quickly, so that it can find directories where it has read access. I suppose not recursing through mount points is the price to pay for avoiding stat() calls.)

I see that when the process has read access, you still get DT_UNKNOWN for all non-normal-directories. (You can't "fs examine", but you can stat() just fine.) Is this something that could be addressed on the AFS side? This would allow find(1) to determine that it has AFS read access to the directory without calling an ioctl, which would be splendid.

Thanks for confirming my suspicion with regards to throttling on the server side; I had wondered whether that was intentional.

I didn't mean to suggest that the issue is timing-related---only that network latency might make "ah, the server is dragging its feet on this stat() call" and "is everybody in my neighborhood using Bittorrent right now?" difficult to tell apart :-)

Finding a cell with a suitable path should be possible then, but I was making the point that even if a solution is developed with that, you won't end up with a nice, self-contained rig for doing regression testing after the fact. Emulating the behavior lets you dispense with all the infrastructure you'd otherwise need. (The only remaining question is to what extent are AFS ioctls and such needed, since a compile-time switch is obviously going to be a hurdle for most folks.)

Daniel Richard G. <iskunk>
Wed 14 Apr 2010 03:04:13 AM UTC, comment #22:


I'm not sure there is anything to fix from the AFS side. The problem that is being faced is that the process does not have the ability to read status data about the objects in the directory. All AFS provides is the name. Directories can be determined by the vnode having an odd value so the client is able to set that value for directory objects. However, any vnode with an even value could be a mount point (which should appear as a directory), a file (which should not), or a symlink (which can appear as either depending on what it refers to). The client therefore has no ability to set the type of an object in a status query when the process has no read permission. Therefore, the d_type is set to DT_UNKNOWN. The type truly is unknown. To interpret this as "not a directory" would be incorrect. In the AFS case, d_type == DT_UNKNOWN really means "access denied".

I suspect that the slow down you are seeing is caused by throttling on the file server when a client continuously sends FetchStatus requests which end in an Access Denied error. After N RPCs that end in an error the file server begins to throttle the client to prevent it from DoS attacking the server.

Can you explain why you believe the issue to be timing related?

As for there not being public cells that export directories with "system:anyuser l" ACLs assigned, there are plenty of them. Finding an example case would not be hard and even if it were hard, putting up a test volume with such a case in the grand.central.org or your-file-system.com or andrew.cmu.edu or ir.stanford.edu or athena.mit.edu cells would be trivial to do.

Jeffrey Altman

Jeffrey Altman <jaltman_openafs>
Tue 13 Apr 2010 11:34:55 PM UTC, comment #21:

Jeff, great to have your input here.

One wrinkle in the nature of this bug is that it occurs only when the AFS client has (l)ist but not (r)ead access in a directory. A suitable AFS path for testing thus might not be available. Moreover, because timing is one of the symptoms of this bug, having a remote AFS server in the loop may confound observations a bit. Though if you don't believe these will be issues, I'll go with what you say.

While you're here... do you see any prospect for this problematic AFS behavior to be fixed outright? The best outcome would be not to make any AFS-specific changes to findutils at all. Given that OpenAFS-1.5.x is overhauling a lot of things, I'd been wondering if there would be interest from the developers in addressing this issue from their end.

Daniel Richard G. <iskunk>
Tue 13 Apr 2010 09:41:33 PM UTC, comment #20:

I'm not sure that an account or even a special system is required to be able to test this issue. The /afs name space is globally public and most government and university deployments have significant quantities of data that is publicly readable by anonymous users. Installing the openafs client package debian / ubuntu / rhel /fedora packages should be all that is required to setup a test system.

Jeffrey Altman <jaltman_openafs>
Tue 13 Apr 2010 09:26:28 PM UTC, comment #19:

Providing an actual system to work on is a bit tricky, since AFS tends to be an institutional thing, and bureaucracy gets in the way. For my part, the installation I use is part of a corporate network---not only do I doubt you'd want to sign an NDA, it's not clear that I could even get things to that point if I wanted to.

Besides, even if there were a way to do that, that doesn't address regression testing. There's no way to put together a self-contained "make check" routine for this.

I'd be happy to put in the time to develop a minimal "AFS simulator," maybe something that can be LD_PRELOAD'ed. This could be bundled into the source tarball, and become part of normal regression testing. As long as you're willing to try that route.

(At the same time, I'll probably check in with some of the AFS guys, just to double-check whether this is something that they ever plan to change. Though my suspicion is that this was intended to be some sort of security/anti-DoS kind of measure...)

Daniel Richard G. <iskunk>
Tue 13 Apr 2010 04:20:18 PM UTC, comment #18:

Yes, if I had access to an AFS system for development/testing, yes, I'd work on this. Without it, no.

It sounds brutal to baldly state it, but this problem has existed for years. Yet still nobody has come forward volunteering to donate a patch and maintain it. Nobody has come forward offering a suitable test/development system. Given the absence of both those things, working on this hardly looks like a good time investment.

Mocking in a sufficiently realistic AFS emulation layer for development and regession testing is probably more work itself than just developing a fix, I would think.

I'm not blocking a fix here, I'm just pointing out that the prerequisites for a fix do not seem to be forthcoming.

James Youngman <jay>
Project AdministratorIn charge of this item.
Tue 13 Apr 2010 07:58:51 AM UTC, comment #17:

Hi James, good to hear from you again.

I'd still like to find a fix for this, because the epic slowness continues to be an issue in AFS.

Would it be helpful if you had access to an AFS system on which to test? While I can't provide a system, the predictable nature of the slowness-behavior would make it possible to emulate via some functions wrapping stat() and readdir(). This way, you could at least see the sort of thing I'm running into.

Daniel Richard G. <iskunk>
Thu 08 Apr 2010 09:37:37 AM UTC, comment #16:

No sign of a working solution so far; unless somebody has a useful suggestion in the mean time, I'll close this bug in a week or so.

James Youngman <jay>
Project AdministratorIn charge of this item.
Sun 12 Jul 2009 12:18:57 PM UTC, comment #15:

I don't have access to an AFS system, so I'm not able to work on this. I would consider applying contributed patches though of course.

For a patch to be usable, it would need to have a zero (or unmeasurable) performance cost for people not using AFS. Of course, if the change is significant, a copyright assignment would also be needed.

James Youngman <jay>
Project AdministratorIn charge of this item.
Sat 17 Jan 2009 07:32:08 PM UTC, comment #14:

If we're going to move forward with this bug, we need an approach that works with both oldfind and ftsfind.

James Youngman <jay>
Project AdministratorIn charge of this item.
Tue 30 Sep 2008 02:11:39 AM UTC, comment #13:

Okay, I've done some investigation on this.

First off, as you suspected, there's not much than can be done for ftsfind. I think the fts backend could be modified to handle AFS gracefully, but that goes beyond the scope of this project.

For oldfind, however, I believe there is a way. I took advantage of the d_type optimization and the "mode" argument to process_path(), to allow calling digest_mode() only if the eval tree requires it.

Here is a patch that is not intended to be applied to CVS, but merely illustrates a hack that works around the problem (without resorting to AFS calls, even). The oldfind(1) program built with this works perfectly in the problematic AFS directories. I'll explain it below:

--- find/find.c 10 Mar 2008 12:06:11 -0000 1.134
+++ find/find.c 29 Sep 2008 20:58:51 -0000
@@ -207,7 +207,7 @@
/* If no paths are given, default to ".". */
for (i = end_of_leading_options; i < argc && !looks_like_expression(argv[i], true); i++)
- process_top_path (argv[i], 0);
+ process_top_path (argv[i], S_IFDIR); /* assume path is directory */

/* If there were no path arguments, default to ".". */
@@ -1164,11 +1164,12 @@
/* Assume it is a non-directory initially. */
stat_buf.st_mode = 0;
state.rel_pathname = name;
- state.type = 0;
+ state.type = mode;
state.have_stat = false;
state.have_type = false;

- if (!digest_mode(mode, pathname, name, &stat_buf, leaf))
+ if ((eval_tree->need_stat || eval_tree->need_type) &&
+ !digest_mode(mode, pathname, name, &stat_buf, leaf))
return 0;

if (!S_ISDIR (state.type))
====END PATCH====

I modified process_path() to skip the call to digest_mode()---and the troublesome stat() call therein---unless a predicate requires it. This isn't a problem when recursing directories, because process_dir() passes in mode==S_IFDIR (thanks to xsavedir() grabbing the d_type info), and it's not a problem for leaf nodes, since those would just have mode==0 (which may well be enough to go on for the predicates).

It is a problem for the top-level call, because that passes in mode==0 unconditionally, so I hard-coded the assumption that the starting points are directories. In real life, the starting points would have to be stat()ed separately, as we obviously don't have d_type information for those.

I saw a lot of optimization logic, and can't be sure that the above isn't horribly breaking some common cases, so I'm hoping you can fill me in on what else might be going on here that I need to be aware of. I believe the main caveat is the assumption that "d_type==DT_UNKNOWN --> NOT_A_DIRECTORY" always holds true; if not, then we'd probably have to check for AFS so that we rely on that assumption only when we're in actual AFS space.

There's also room for improvement here, too. With this code tweak, "oldfind /path/to/afs" works fine, but "oldfind /path/to/afs -type f" yields a string of "Permission denied" error messages, one every ~3 seconds. Which could be handled more gracefully, but is in any event understandable. However, the same command with "-type d" yields the same behavior, when it really shouldn't. It shouldn't have to stat entries to tell if they are directories or not, when that is already known from the d_type information. (Admittedly, this is more ambitious a change.)

Daniel Richard G. <iskunk>
Wed 03 Sep 2008 09:29:01 AM UTC, comment #12:

Kevin Dalley checked this change in in 2000:

revision 1.6
date: 2000-04-05 08:26:24 +0100; author: kevind; state: Exp; lines: +14 -11;

  • configure.in: add Galition and Estonian languages.

Miscellaneous other fixes.

$ cvs -z3 diff -upN -r1.6 -r1.7 configure.in
Index: configure.in
RCS file: /cvsroot/findutils/findutils/Attic/configure.in,v
retrieving revision 1.6
retrieving revision 1.7
diff -u -p -r1.6 -r1.7
--- configure.in 5 Apr 2000 07:26:24 -0000 1.6
+++ configure.in 12 Apr 2000 08:51:02 -0000 1.7
@@ -1,21 +1,21 @@
dnl Process this file with autoconf to produce a configure script.

dnl Set of available languages.
ALL_LINGUAS=" de es fr it ko nl pl pt_BR ru sv gl et"

-AM_INIT_AUTOMAKE(findutils, 4.1.4)
+AM_INIT_AUTOMAKE(findutils, 4.1.5)

[ --enable-id-cache cache all UIDs & GIDs; avoid if using NIS or Hesiod],
-[ --with-afs support -fstype afs],
- CPPFLAGS="$CPPFLAGS -I/usr/afsws/include"
- LIBS="$LIBS -L/usr/afsws/lib -L/usr/afsws/lib/afs -lsys -lrx -llwp"])

I don't know what the motivation behind the change was, though.

James Youngman <jay>
Project AdministratorIn charge of this item.
Wed 03 Sep 2008 05:15:09 AM UTC, comment #11:

Interesting. I'll have a look at that. But to answer one question...

/tmp/foo.iso on /afs/example.com/user/skunk/blah type iso9660 (rw,loop=/dev/loop0)

Yes. And the same likely goes for bind mounts.

I seem to recall there was an --enable-afs (or similar) option in the configure script at one time, that would enable the AFS code. Did that go away for a lack of testing, or is there something else to know about the development history?

Daniel Richard G. <iskunk>
Tue 02 Sep 2008 10:11:45 AM UTC, comment #10:

> You would be open to a compile-time option to link against the AFS libraries? (

Yes, absolutely. That's what configure is for.

> I was under the impression that this existed in past versions, and was removed at some point.)

No, it is still there:

#ifdef AFS
#include <netinet/in.h>
#include <afs/venus.h>
#if _STDC_
/* On SunOS 4, afs/vice.h defines this to rely on a pre-ANSI cpp. */
#define _VICEIOCTL(id) ((unsigned int ) _IOW('V', id, struct ViceIoctl))
#ifndef _IOW
/* AFS on Solaris 2.3 doesn't get this definition. */
#include <sys/ioccom.h>

static int
in_afs (char *path)
static char space[2048];
struct ViceIoctl vi;

vi.in_size = 0;
vi.out_size = sizeof (space);
vi.out = space;

if (pioctl (path, VIOC_FILE_CELL_NAME, &vi, 1)
&& (errno == EINVAL || errno == ENOENT))
return 0;
return 1;
#endif /* AFS */

static char *
file_system_type_uncached (const struct stat statp, const char path)
struct mount_entry entries, entry;
char *type;

(void) path;

#ifdef AFS
if (in_afs(path))
fstype_known = 1;
return xstrdup("afs");

> If we use libafs, we may as well go all the way and have find(1) check the permissions on each directory (if in AFS) and cue off of that.

You may find this hard to integrate with fts.

However, if AFS volumes cannot contain anything that isn't AFS, you may not need to provide such integration. Is it possible to mount a non-AFS filesystem at a mount point within the AFS filesystem? What about bind mounts?

James Youngman <jay>
Project AdministratorIn charge of this item.
Mon 01 Sep 2008 11:12:26 PM UTC, comment #9:

You would be open to a compile-time option to link against the AFS libraries? (I was under the impression that this existed in past versions, and was removed at some point.)

If we use libafs, we may as well go all the way and have find(1) check the permissions on each directory (if in AFS) and cue off of that.

(I saw your bug-thread on the OpenAFS list from last May, and in particular the dirent.c test program which returns DT_UNKNOWN even for entries in read-accessible directories, a situation I have confirmed in my AFS installation. So a different approach is going to be needed at any rate.)

Daniel Richard G. <iskunk>
Mon 01 Sep 2008 04:16:34 PM UTC, comment #8:

On the basis of an earlier conversation with Jeffrey Hutzelman, I would assume that on systems where AFS is available (i.e. AFS libraries were detected at configure time) we can use a pioctl() call to check whether we're inside an AFS volume each time we observe a change in st_dev.

Such an approach would be quite efficient since a pioctl call could occur a maximum of N*M times, where N is the number of mounted filesystems and M is the number of start-points listed on find's command line. Even the O(N*M) figure should be reducible to O(N) with the use of a cache.

James Youngman <jay>
Project AdministratorIn charge of this item.
Mon 01 Sep 2008 04:10:07 PM UTC, comment #7:

I very much prefer to avoid an approach like -nostatdtu. You're right that it would in principle be quite like -noleaf. The problem is that people don't enable -noleaf even in cases where they should.

From a discussion I had with Jeffrey Altman <jaltman@secure-endpoints.com> :

<< The AFS file system can be mounted on any directory and there are some well known organizations that do mount the file system elsewhere or even in more than one place in the file system tree. >>

James Youngman <jay>
Project AdministratorIn charge of this item.
Mon 01 Sep 2008 04:08:05 AM UTC, comment #6:

I get your first paragraph, but I don't understand what you're responding to in the second and third. Let me try this again:

We know what the behavior needs to be in order to avoid the 3-second-stat()s-on-AFS problem: if opendir() returns a d_type of DT_UNKNOWN for an entry, then give up trying to determine what it is (whether -L was given or not). However, we only want to do this on AFS, because some other filesystems can and do return DT_UNKNOWN and yet allow us to stat() the entry perfectly well.

The question, then, is how to make find(1) behave that way only when needed/appropriate. I see two viable options:

1. Add a new option---let's call it "-nostatdtu", i.e. "don't stat() entries with d_type=DT_UNKNOWN"---that goes right next to -noleaf in the man page. The user would then give this option when slow searches in AFS-space make it necessary.

Pros: find(1) remains filesystem-agnostic; this behavior can potentially be used to speed up searches on other filesystems with similar d_type behavior

Cons: The option is difficult to explain to ordinary users; user has to be aware of this option and when to use it

2. Have find(1) check each entry to see if it has a canonical path under /afs, and if so, change to the workaround behavior.

(The value of "/afs" would be hard-coded, not user-specified, as this is the non-aribitrary AFS mount point. The value might, at most, be a compile-time option.)

Pros: Transparent to the user; a hypothetical invocation of "find /" that goes into /afs and /brokenfs (with unreliable d_types) works correctly

Cons: Bad in the rare case that /afs isn't actually an AFS filesystem; find(1) is no longer filesystem-agnostic

It could be 1, or 2, or both, or 2 enabled with a compile-time option, or perhaps some other approach altogether. What would you like to see in a submitted patch?

Daniel Richard G. <iskunk>
Mon 01 Sep 2008 12:58:02 AM UTC, comment #5:

To split a hair here, we're not talking about an option to trust d_type=DT_UNKNOWN. We're talking about how find will know to assume that DT_UNKNOWN means "it is impossible for this process to establish the nature of thie file at all".

But the answer to your question has to be that the user shall not be forced to provide a command-line mapping between path and AFS-filesystem-ness, for a number of reasons. The first and most obvious is usability, the second is that some fraction of users will not even know this is necessary, and the third is that it's either infeasible or very invasive to do this when the -L option is in effect.

Your second question I cannot answer since I don't know enough about AFS, sorry.

James Youngman <jay>
Project AdministratorIn charge of this item.
Sun 31 Aug 2008 05:18:10 PM UTC, comment #4:

Please see comment #2. The question is, "Should find(1) have an option that causes it to trust d_type=DT_UNKNOWN, or should it specifically check for AFS and do the same without user intervention when working inside that filesystem?"

I recognize that find(1) can't always trust the value of d_type, so I need to know what specifically should cause it to do so in situations when it is working under AFS. (I can't put together a patch until I know what behavior I'm implementing.)

Daniel Richard G. <iskunk>
Sun 31 Aug 2008 12:50:22 PM UTC, comment #3:

<blockquote>I vaguely recall that when doing readdir() for an AFS file with only "l" access, d_type would have an value corresponding to "unknown." In which case, find(1) can assume that a stat() is pointless because it would invariably fail.</blockquote>

That's not true in general for file systems other than AFS, though.

<blockquote>I didn't give it a -type predicate, and it can already tell which ones are directories via d_type, so why go to the trouble?</blockquote>

Because it can't be sure that the DT_UNKNOWN entries aren't really directories. It's possible that fts() could be changed to assume different semantics for DT_UNKNOWN on AFS and non-AFS filesystems, but who would take up the maintenance and testing burden for such a change?

Since I have no access to any AFS system for development, I'm not in a position to test any changes relating to it.

James Youngman <jay>
Project AdministratorIn charge of this item.
Tue 26 Aug 2008 02:53:24 PM UTC, comment #2:

Should there be an option, along the lines of -noleaf, that causes find(1) to believe what d_type says for unknown entries? Would you rather have find(1) autodetect AFS in some way to adapt its behavior on-the-fly? (IIRC, this would basically amount to checking whether the canonical path is under /afs; that's supposed to be the "official" method.)

d_type is DT_UNKNOWN only for non-directories. AFS "l" access does let you see if a name is a directory, so that you can look inside it (potentially with greater access permissions).

I could put together a patch; I just need to know what the acceptable behavior should be.

Daniel Richard G. <iskunk>
Tue 26 Aug 2008 10:18:42 AM UTC, comment #1:

AFAICS find has no way to distinguish "unknown because permissions do not allow you to see this information" from "unknown because d_type is not supported on this filesystem, or this part of this filesystem".

> it can already tell which ones are directories via d_type, so why go to the trouble?

As far as I can see from what you say, d_type is DT_UNKNOWN, so find can't tell which are directories.

But the long and the short of it is this - I don't have an AFS system, so I cannot test candidate patches. If you want findutils to handle this situation better, you will need to either send me a patch (that doesn't break anything else) or recruit someone else to do so.

Such a patch would need to address both find and oldfind.

James Youngman <jay>
Project AdministratorIn charge of this item.
Tue 26 Aug 2008 07:03:01 AM UTC, original submission:

I have a script intended to generate a file listing of several users' home directories, stored in AFS. The listing should be fairly bare-bones: no file attributes of any kind, not even a distinction between regular files vs. symlinks vs. other file types, just names and the directories containing them.

Consistent with that, the script runs under a restricted permission set where it can only list directories. Where an AFS admin would have all "rlidwka" bits set, this script would only get "l" (list) access.

find(1) takes a very long time scanning directory trees under these conditions. Here is what "strace -r -p <find-pid>" says is going on:

3.016894 lstat("Makefile", 0x7fff950fb590) = -1 EACCES (Permission denied)
3.001573 newfstatat(AT_FDCWD, "abuzz_r3_linguistics.tex", 0x664278, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied)
3.019501 lstat("abuzz_r3_linguistics.tex", 0x7fff950fb590) = -1 EACCES (Permission denied)
3.001665 newfstatat(AT_FDCWD, "abuzz_r3.tex", 0x664398, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied)
3.020538 lstat("abuzz_r3.tex", 0x7fff950fb590) = -1 EACCES (Permission denied)
3.001657 newfstatat(AT_FDCWD, "abuzz_r3_status.tex", 0x664b48, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied)
3.023643 lstat("abuzz_r3_status.tex", 0x7fff950fb590) = -1 EACCES (Permission denied)
3.001544 newfstatat(AT_FDCWD, "abuzz_r3_misc.tex", 0x664c68, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied)
3.019582 lstat("abuzz_r3_misc.tex", 0x7fff950fb590) = -1 EACCES (Permission denied)

Each of those failed stat calls is taking over three seconds! Per file! And this is with 4.4.0 compiled with d_type and leaf optimization enabled.

find(1) should not be stat()ing those files to begin with. I didn't give it a -type predicate, and it can already tell which ones are directories via d_type, so why go to the trouble?

As a potential (new) optimization, I vaguely recall that when doing readdir() for an AFS file with only "l" access, d_type would have an value corresponding to "unknown." In which case, find(1) can assume that a stat() is pointless because it would invariably fail. The thing to remember is that, with "l" access only, AFS will allow you to see the name, but will tenaciously withhold all other info, even file type (with the exception of directories). The fix is, effectively, to handle this situation in a more graceful way.

Daniel Richard G. <iskunk>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by scs (Posted a comment)
  • -email is unavailable- added by shadow (Posted a comment)
  • -email is unavailable- added by adeason (Posted a comment)
  • -email is unavailable- added by jaltman_openafs (Posted a comment)
  • -email is unavailable- added by jay
  • -email is unavailable- added by jay
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by iskunk (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-05-11 jay Assigned toNone => jay
    2011-05-11 jay StatusWont Fix => None
    2009-07-12 jay StatusNone => Wont Fix
    2008-09-01 jay Carbon-Copy- => Added jeffrey hutzelman
    2008-09-01 jay Carbon-Copy- => Added jeffrey altman

    Back to the top

    Powered by Savane 3.3