bugfindutils - Bugs: bug #46305, Doing "find -L . -type d...

 
 

bug #46305: Doing "find -L . -type d -delete" fails on symlinks to directories.

Submitter:  C <pfudd>
Submitted:  Tue 27 Oct 2015 01:23:12 AM UTC
 
Category:  find Severity:  4 - Important
Item Group:  Wrong result Status:  Wont Fix
Privacy:  Public Assigned to:  jay
Originator Name:  Open/Closed:  Open
Release:  4.4.2 Fixed Release:  None
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       

 

( Jump to the original submission )

Thu 02 Feb 2023 12:46:31 AM UTC, comment #11: 

comment #9:

> Oh, and the patch that tries unlink() if rmdir() failes with ENOTDIR.


That patch is here:

    [PATCH] find: fix -L and -delete removing symlinks to directories
    https://lists.gnu.org/archive/html/findutils-patches/2022-10/msg00002.html

However, in response to other patches, I was asked to submit an application to receive the paperwork for assigning copyright. I submitted it last November, but received no feedback, and the paperwork still hasn't arrived. I asked Craig about it, but I didn't receive a response. I think that process might need to be completed before the patch will be accepted, but it doesn't feel like it's progressing.

raf <raf>
Wed 01 Feb 2023 11:59:33 PM UTC, comment #10: 

comment #8:

> You bring up great points.
>
> I'd welcome a new, separate command-line option for a symlink-following delete, and the documentation change stating that -L is used during traversal only.


The patch I submitted that fixes this bug includes updated documentation on what -L does and doesn't do. So that doesn't have to wait until someone decides to implement a new command line option for symlink-following deletions.

raf <raf>
Wed 01 Feb 2023 05:01:14 AM UTC, comment #9: 

Oh, and the patch that tries unlink() if rmdir() failes with ENOTDIR.

Anonymous
Wed 01 Feb 2023 04:55:04 AM UTC, comment #8: 

You bring up great points.

I'd welcome a new, separate command-line option for a symlink-following delete, and the documentation change stating that -L is used during traversal only.

Anonymous
Tue 31 Jan 2023 02:47:25 AM UTC, comment #7: 

comment #6:

> It looks like the right thing to do is to delete the link and the thing the link points to, if the `-L` option (follow links) is enabled.


As I tried to explain earlier, I think that interpretation is incorrect and unhelpful. That interpretation leads to the problem being seen as very complicated and difficult and dangerous, and so very unlikely to ever be fixed.

My interpretation of the problem is simple, and lends itself to a trivial solution that would improve the user experience immediately (by replacing a failure to act and the accompanying error message with some useful, safe, and consistent behaviour).

I think it's perfectly reasonable to accept (and document the fact) that symlink-following (-L) applies to the search/traversal through the directory structure, but it does not apply to actual deletion of symlinks that were matched via searching.

When I have wanted to delete with -L, I certainly didn't want symlink-following to apply to the deletion. If it did, I'd want another option to disable it, although I'd want it disabled by default. Symlink-following deletion (if it exists at all) should be a new, separate command line option so as not to confuse that functionality with the functionality of -L. And I think it unlikely that most people would want symlink-following to apply to deletion if they thought sufficiently about the consequences of such behaviour.

I suggest accepting the patch I submitted that fixes this problem the simple way. And then, if enough users actually complain that the deletion should have been more aggressive by following the symlinks being deleted, that it could be explained to them that they really don't want that behaviour. Without knowing what other symlinks might point to the deleted symlink target, such an operation could often be a disaster. If they really do want that behaviour, a nested find/exec/find command would probably meet their needs.

Please remember that this bug report isn't about deletion not following symlinks. It's just about the failure and error message when find(1) tries to rmdir(2) a symlink whose target is a directory. When the symlink's target is a non-directory, the operation is unlink(2) instead and it works. It works by unlinking the symlink, not the target. The ability to unlink a symlink whose target is a directory would be consistent and useful behaviour. No user has complained that deleting a symlink to a file should also delete the target file, or that deleting a symlink to a directory should delete the target directory. That idea came up in the discussion of this bug report, but it is not what this bug report is about.

The patch could even be seen as a temporary stage that at least fixes the error, and adds the deletion of the symlink, and resolves this bug. The question of whether or not any user actually wants more aggressive symlink-following deletion can be left to a subsequent feature request or bug report about that actual issue. And if that feature request were ever implemented, it would still be backwards-compatible. So I don't see any compelling reason not to apply the patch and resolve this bug.

Just in case the unhelpful interpretation continues to prevail, and this bug is never fixed in find(1), and anyone else comes here hoping for the solution, they can use my "rawhide" file-finding program instead (https://raf.org/rawhide) which behaves in the way I have suggested.

raf <raf>
Mon 30 Jan 2023 06:52:08 PM UTC, comment #6: 

It looks like the right thing to do is to delete the link and the thing the link points to, if the `-L` option (follow links) is enabled.

Anonymous
Mon 24 Oct 2022 08:57:43 PM UTC, comment #5: 

(Wow, I'd forgotten all about this bug report!)

The interactions between find and symlinks have complicated implications, that's for sure.

As <berny> said:

> Finally, also rm(1) and rmdir(1) do not have an -L option - probably
> for a good reason.


There's a physical and a logical structure of a filesystem, as exemplified in the `pwd` command's man page:

> -L, --logical
>             use PWD from environment, even if it contains symlinks
> -P, --physical
>             avoid all symlinks


Removing (rm -rf) a physical directory tree is straightforward, and it's easy to reason about, mainly because it's a tree and therefore easy to say what things are in the set of things to delete.

Removing a logical directory structure (because we're following symlinks, we can't call it a tree) is straightforward too, at least from the point of view of the algorithm:  Recursively descend the directories, delete everything you find.

One problem: those darn symlinks can point to places you didn't really want to delete: if someone innocently created a symlink to `/` in one of the subdirectories then you're well on your way to an `rm -rf /` catastrophe.

It seems like the Unix philosophy is "If a tool isn't sharp enough to cut you badly, then it's not sharp enough to use."  To which some have responded "Sure, but don't put the sharp edges on the handle!" 

It looks a bit like the risk of `rm -rf /` is a sharp-edges-on-the-handle problem, as any user can create them, but it could be argued either way.  It's already in `find` and people have probably already written code that depends on it, so that train has left the station.

As for the original problem: when a symlinked thing is deleted, I would argue that the user intended to delete the symlink and the thing, not just the thing: when they specify `-delete`, they want that directory entry and all it represents to be gone.  This then prevents the 'directory not empty' and 'not a directory' problems seen in <raf>'s comment.  If there happen to be other symlinks that were outside the graph of things traversed, then they shall become broken.

One more complication:

$ date > file1
$ ln -s file1 file2
$ ln -s file2 file3
$ mkdir dir1
$ ln -s dir1 dir2
$ ln -s dir2 dir3
$ ls -l

total 4
drwxrwxr-x. 2 user user 40 Oct 24 13:40 dir1
lrwxrwxrwx. 1 user user  4 Oct 24 13:40 dir2 -> dir1
lrwxrwxrwx. 1 user user  4 Oct 24 13:40 dir3 -> dir2
-rw-rw-r--. 1 user user 29 Oct 24 13:37 file1
lrwxrwxrwx. 1 user user  5 Oct 24 13:37 file2 -> file1
lrwxrwxrwx. 1 user user  5 Oct 24 13:37 file3 -> file2

$ find -L file3 dir3 -delete
find: cannot delete ‘dir3’: Not a directory

$ ls -l
drwxrwxr-x. 2 user user 40 Oct 24 13:40 dir1
lrwxrwxrwx. 1 user user  4 Oct 24 13:40 dir2 -> dir1
lrwxrwxrwx. 1 user user  4 Oct 24 13:40 dir3 -> dir2
-rw-rw-r--. 1 user user 29 Oct 24 13:37 file1
lrwxrwxrwx. 1 user user  5 Oct 24 13:37 file2 -> file1

Currently it deletes only file3, whereas I'd argue it should delete everything.  To do that, it would need to manually follow symlink chains and delete the links and the final filesystem object.

Thanks

Anonymous
Sat 22 Oct 2022 10:01:26 PM UTC, comment #4: 

Hi,

I was just about to submit a bug report about this, and found this existing bug report. I think it's a shame that it has been set to WONTFIX. It's easy to fix, and fixing it would be in keeping with the principle of least astonishment.

Luckily, I noticed this in the notes:

> Please feel free to follow up if you disagree or you can see a
> simple solution. In the meantime I will work on some
> documentation updates to try to make things clearer.


So here goes. Here is my bug report with two (almost identical) solutions to choose from, followed by more commentary to address the existing discussion:

> "find -L /path -delete" fails to delete symlinks to directories
>
> When following symlinks with -H or -L, find sees a symlink to a
> directory as a directory, but it doesn't remember the fact that
> there was a symlink. So, when deleting, it uses rmdir rather
> than unlink (or their equivalents), which fails to remove the
> symlink, and that can break the overall deletion.
>
> Steps to repeat:
>
>   $ mkdir t dd dd/d
>   $ touch dd/d/f ff
>   $ ln -s ../dd t/ld
>   $ ln -s ../ff t/lf
>   $ find t dd ff -ls
>   ... t
>   ... t/lf -> ../ff
>   ... t/ld -> ../dd
>   ... dd
>   ... dd/d
>   ... dd/d/f
>   ... ff
>   $ find -L t -delete
>   find: cannot delete 't/ld': Not a directory
>   find: cannot delete 't': Directory not empty
>   $ find t dd ff -ls
>   ... t
>   ... t/ld -> ../dd
>   ... dd
>   ... ff
>   $ rm -r t dd ff
>
> Expected output:
>
>   $ find -L t -delete
>   $ find t dd ff -ls
>   find: 't': No such file or directory
>   ... dd
>   ... ff
>
> The solution is either to remember the fact that there's a symlink, or
> check for it with lstat just before each deletion of a directory (if
> -H or -L), and use unlink rather than rmdir when it's a symlink. If
> the existence of the symlink is already being remembered (e.g., for
> -xtype), then the first solution is probably easier. Otherwise, the
> second solution is probably easier.


Now for some more commentary:

I think it's a mistake to think that the symlink's ultimate target directory should be deleted when following symlinks, or that that is what the user is expecting when they use -L/-H and -delete together. There is (necessarily) a distinction between following symlinks for the purpose of file system traversal versus following symlinks for the purpose of deletion.

The first is what -L does, and what the user expects. The second is impossible (or ridiculously difficult/tedious/boring/errorprone outside the kernel, and probably very unwise (think long chains of symlinks and race conditions)). If any user does expect such behaviour, they just haven't thought about it enough. If they did, they'd realise that it's not an option. So that's no reason not to fix this.

And it's not just about directories. When following symlinks and deleting, if there's a symlink to a non-directory, it's the symlink that gets deleted, not the symlink's ultimate target. The example in the above bug report demonstrates that. The same should apply to symlinks to directories, but it doesn't. Consistency is a reason to fix this.

Unless this is fixed (by switching from rmdir to unlink when trying to delete a symlink), find fails to do what the user wants, and forces them to resort to a subsequent rm command. That's less than awesome.

The find(1) manual entry refers to "confusing behaviour", in relation to -L and -delete, but it's only confusing because it hasn't been explained to the user. That's easy to do as well. I've just written a find-alternative, and I've documented the behaviour like this (adapted for find terminology):

  When -delete is used with the -L or -H option,
  and a symlink to a directory is followed, the
  symlink's ultimate target directory's contents
  are searched, and any matches found there are
  deleted, but the target directory itself is
  never deleted. It isn't possible to delete a
  filesystem entry via a symlink to it. If the
  directory itself matches the search criteria,
  the symlink is deleted. Similarly, when a
  symlink to a non-directory is followed, and
  the symlink's ultimate target matches the search
  criteria, the symlink is deleted, not the
  ultimate target.

That should prevent any confusion arising (and probably be reassuring). It explains what the behaviour is and the reason for it being what it is. Clearly describing the behaviour should be enough to set the user's expectations (at least of those that read the documentation). You're welcome to incorporate/adapt the above text into find's manual entry, but it really belongs in the -delete section, rather than the -L section.

As for the issue of symlinks becoming broken during the deletion operation, I don't think that affects this at all. If find(1) thinks that it needs to delete a directory, because it's remembering something that is no longer true, it doesn't matter, because it's only the symlink itself that needs to be deleted anyway, and the symlink still exists and needs to be deleted, whether it's broken or not. If the comment in the manual entry about "confusing behaviour" was only about this, it could probably be removed. But perhaps it relates to something else.

Anyway, that's what I think. I hope you'll reconsider the WONTFIX status.

I'm submitting a patch that fixes it and documents and tests the behaviour. In pred_delete(), just after it switches to rmdir when unlink fails because it's a directory, I've added a check for rmdir failing because it's not a directory, and have it switch to unlink. That's all that's needed. It's a very small, simple change.

I assume that "Copyright-paperwork-exempt: Yes" because, even though the patch is more than 10 lines (but only if you include docs and tests), the 10 line code addition was copied and pasted from the 10 lines above, and 1 line of code and 1 line of comments were changed, so you could see it as only 1 or 2 new lines of code. If that's not how it works, let me know.

raf <raf>
Mon 16 Jul 2018 08:46:59 PM UTC, comment #3: 

With "-L bar -delete", find would have to delete 2 files:
a) the symlink target '../foo', and
b) the symlink itself 'bar/baz'.

I agree with James that trying to fix (or work around) this
would probably more ask for trouble than being of help.

From 'man find':

  -L   ... Actions that can cause  symbolic  links  to  become
       broken while find is executing (for example -delete)
       can give rise to confusing behaviour.

Depending on where the symlink points to, deleting possibly
the opposite way up the tree sounds too dangerous IMO; consider:
  ln -s ../../../../../../..  kllr-slink
or the 'symlink-to-root' example James gave.

Finally, also rm(1) and rmdir(1) do not have an -L option - probably
for a good reason.

Have a nice day,
Berny

Bernhard Voelker <berny>
Project Administrator
Thu 12 Jul 2018 03:45:53 AM UTC, comment #2: 

I think this is an actual bug, despite the general danger of running -delete together with -L.  In this setup:

$ mkdir foo
$ mkdir bar
$ ln -s ../foo bar/baz
$ find -L bar -delete
find: cannot delete ‘bar/baz’: Not a directory
find: cannot delete ‘bar’: Directory not empty

(No -type d necessary.)  strace shows that find tried to

unlinkat(AT_FDCWD, "bar/baz", AT_REMOVEDIR) = -1 ENOTDIR (Not a directory)

but "bar/baz" is not a directory so AT_REMOVEDIR is wrong.

Tavian Barnes <tavianator>
Sun 01 Nov 2015 09:23:19 PM UTC, comment #1: 

> The '-L' option makes '-type d' detect symlinks to directories, but '-delete' doesn't treat them as such.


I don't think that's what is happening.  Use of -delete automatically turns on -depth.  This means that the filesystem is visited in postorder.  For the command line " find -L . -type d -delete" this means that the -delete action takes place for children and some siblings before it takes place for some item.   In the case of symbolic links this will mean that, at least some of the time, the thing the link points to will have been deleted before the link is examined by find.

That means that, at the time bar (in your example) is examined, "-type d" is no longer true for it.  It's a broken symlink instead (-type l would match it) and so it is not deleted since -type d is false.

I don't think this problem can be "solved" by special-casing directories, as I'd imagine that analogous things apply to link counts or (sometimes) directory sizes.

I think then that find is working as designed and that changing this aspect of the design to change this behaviour would likely give rise to semantics that are both inconsistent and much harder to explain (because of the special-casing).

I'd recommend simply avoiding the use of -L with -delete (and in any case that combination risks causing the deletion of unexpected  parts of the tree where third parties can make symbolic links to / for example).

Please feel free to follow up if you disagree or you can see a simple solution.   In the meantime I will work on some documentation updates to try to make things clearer.

James Youngman <jay>
Project Administrator
Tue 27 Oct 2015 01:23:12 AM UTC, original submission:  

The '-L' option makes '-type d' detect symlinks to directories, but '-delete' doesn't treat them as such.

$ mkdir /tmp/foo
$ cd /tmp/foo
$ mkdir baz
$ ln -s baz bar
$ find -L . -type d
bar
baz
$ find -L . -type d -delete
./bar
find: cannot delete `./bar': Not a directory
./baz
$ ls -l
total 0
lrwxrwxrwx 1 root root 3 Oct 26 18:12 bar -> baz

I'd expect --delete to delete whatever it found.

Note: if 'bar' is the directory and 'baz' is the symlink, then no error message is generated, but baz remains as a broken symlink, since the directory was deleted before the symlink was found (assuming sorted directory traversal).

Also, http://ftp.gnu.org/gnu/findutils/ indicates that 4.4.2 is the latest version (from 2009), but this bug-reporting page indicates that 4.5.14 is the latest version.  Please link to the new download page on https://savannah.gnu.org/projects/findutils/, or expect bug reports about ancient versions of the code.  :-)

Thanks!

C <pfudd>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #35360:  sv-43605-base.sh added by jay (2KiB - application/x-shellscript)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by raf (Posted a comment)
  • -email is unavailable- added by berny (Posted a comment)
  • -email is unavailable- added by tavianator (Posted a comment)
  • -email is unavailable- added by jay (Updated the item)
  • -email is unavailable- added by pfudd (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-11-01 jay Attached File- Added sv-43605-base.sh, #35360
    2015-11-01 jay StatusNone Wont Fix
    2015-11-01 jay Severity3 - Normal 4 - Important
        Assigned toNone jay

    Back to the top

    Powered by Savane 3.10