bugfindutils - Bugs: bug #60383, [feature-request] let find read...

 
 

bug #60383: [feature-request] let find read files from stdin or file.

Submitter:  Paxsali <paxsali>
Submitted:  Mon 12 Apr 2021 03:11:02 PM UTC
   
 
Category:  find Severity:  1 - Wish
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  berny
Originator Name:  Open/Closed:  Closed
Release:  None Fixed Release:  4.9.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 01 Feb 2022 08:32:53 PM UTC, comment #19: 

IMO it's a bit moot to speculate about cmd failing to write the full file name without a reproducer,
and actually I don't think this is possible with a proper, atomic write(2).

Well, cmd could be constructed by a combination of programs with text processing etc.,
but the problem with abbreviated file names could also happen when passing the result directly as starting point.
Error handling is key, always.

For now I consider the discussion about -files0-from complete,
and I'm hereby closing this issue for the imminent release.

Discussion could of course continue here, or directly on the bug-findutils mailing list.

Bernhard Voelker <berny>
Group administrator
Sat 04 Dec 2021 01:55:08 PM UTC, comment #18: 


>  - cmd | find -files0-from - is potentially dangerous if cmd ends up being aborted for instance because it reaches some resource limits. For instance it was going to output /tmp/x/foo\0/tmp/x/bar\0 but got aborted just after it wrote /tmp/x/foo\0/, find will end up searching / and if it was with -mtime +30 -delete for instance, that could have disastrous consequences, maybe find should only consider fully delimited records.


About that, there's the same problem with xargs as in:


cmd | xargs -r0 rm -rf --


Could also end up removing the wrong things if cmd is interrupted. I verified none of GNU, FreeBSD, NetBSD require a terminating \0, as in printf x | xargs -r0 does output x\n just like printf 'x\0' | xargs -r0.

Stephane Chazelas <stephanechazelas>
Sat 04 Dec 2021 12:44:23 PM UTC, comment #17: 


> it seems just to ensure that the next argv[i] is treated as a starting point
> (regardless whether it looks like an option or not).
> Furthermore, -f seems to change the handling of the '--' argument.
>
> This seems to be a solution for the 80% case, but if one has file names
> like '--', then ...


It doesn't change the handling of --, the reference to -- in the BUGS section of the manual is a bit misleading.

-f is an option there, like -H or -L, except it's an option that takes arguments, it's not a predicate. So it has to be on the left hand side of --.

As -f takes a required argument (as -ffile or -f file as usual with getopt()), it will handle any arbitrary string including --. What follows is taken as the argument to -f whatever it is, whether it starts with - or looks like a predicate.

It's like for grep. grep -F "$string" won't work properly for values of $string that start with -. You need grep -F -e "$string", where "$string" is the argument to the -e option. With grep, you can also use grep -F -- "$string", where "--" marks the end of options.

But using -- doesn't help with find, because while "--" does mark the end of options, meaning that -L, -H, -f... can't be used afterwards, arguments starting with - (and [!()]) still cause problems as after that --, find still expects either file names or predicates, so -- doesn't help at all for find.

Note that I was not suggesting GNU find implements the -f option instead of the -files0-from predicate, just to add it for compatibility with BSDs and so we can eventually have a portable way to pass arbitrary file names to find.

I agree -files0-from has advantages over -f:

- the list can be fed slowly and find can start processing it as it comes
- no need to allocate and pass around a potentially huge list
- the list is not exposed in the output of ps
- not affected by E2BIG execve() limit

-f has advantages too though:

- for the very common case of invoking find on one directory, find -f "$dir" ... is less cumbersome than find -files0-from <(printf '%s\0' "$dir")
- stdin is preserved (compared to using cmd | find -files0-from -)
- cmd | find -files0-from - is potentially dangerous if cmd ends up being aborted for instance because it reaches some resource limits. For instance it was going to output /tmp/x/foo\0/tmp/x/bar\0 but got aborted just after it wrote /tmp/x/foo\0/, find will end up searching / and if it was with -mtime +30 -delete for instance, that could have disastrous consequences, maybe find should only consider fully delimited records.

For the E2BIG limit, note that BSDs support:


printf -- '-f%s\0' "${large_list[@]}" |
  xargs -r0 -J % find % ...


(here assuming a builtin printf)

That -J would be a welcome addition to GNU xargs as well.

Stephane Chazelas <stephanechazelas>
Fri 03 Dec 2021 06:31:57 PM UTC, comment #16: 

Looking at BSD's -f documentation - a one-liner - and implementation [1],
it seems just to ensure that the next argv[i] is treated as a starting point
(regardless whether it looks like an option or not).
Furthermore, -f seems to change the handling of the '--' argument.

This seems to be a solution for the 80% case, but if one has file names
like '--', then ...

And this is limited by the maximum command line length while -files0-from
allows an arbitrary number of starting points.

[1] https://github.com/freebsd/freebsd-src/tree/main/usr.bin/find/

Compared to GNU find's -files0-from option, the usefulness of -f seems
to be a bit limited, and the former is the more generic and superior
choice.

Therefore, I'd rather would like to see that POSIX would standardize
the -files0-from option (or a -0 shortcut for it).

Bernhard Voelker <berny>
Group administrator
Sun 28 Nov 2021 02:30:41 PM UTC, comment #15: 

Thanks Bernhard. I think it's a more useful API like that.

BTW, would you also consider adding BSD's -f to GNU find?

I don't think POSIX would agreed to specify -files0-from, like they rejected -print0 on the ground that most POSIX utilities can't cope with NULs anyway, but given that -f is already in BSDs, if it was added to GNU find, there's a good chance it would be accepted.

Timing could be good if we're quick as well as the next version of the standard is due soon, already in 3rd draft stage IIRC.

It would be a good opportunity to at last have a standard way to pass arbitrary file names to "find".

In BSDs, it's


find -f file predicates
find -f file1 -f file2 predicates
find -f file1 file2 predicates
find -f file1 -- file2 predicates


(files can be passed both with -f and without)

BSD finds don't allow things like "find -print" (no file specified). Still, I don't think that would be a problem for GNU find. It could accept all the above, but we just need to clarify that while "file -f file1 file2 -print" processes file1 and file2 and "find -print"  is the same as "find . -print", "find -f file1 -print" would not be the same as "find -f file1 . -print".

That means however that:

"file -f$array -print" (fish, rc, es syntax) or "find -f$^array -print" (zsh syntax), "find "${array[@]/#-f}" -print" (ksh/bash) would end up looking up files in "." (gives an error in BSDs). A caveat worth mentioning.

Note that in:

find -f file1 -H -f file2

The "-H" applies to both file1 and file2 in BSDs. Making it apply to only file2 could be a more useful API, though only marginally so, and potentially more surprising.

If you think adding -f is a good idea and would be willing to implement it, I can write up the request to POSIX to specify it.

It may be worth raising a separate savannah ticket for that. I'm happy to do that as well.

Stephane Chazelas <stephanechazelas>
Sat 27 Nov 2021 11:44:03 PM UTC, comment #14: 

Pushed (with a slightly different text in the documentation) at:

https://git.savannah.gnu.org/cgit/findutils.git/commit/?id=4ddb092de66a4186a097de5ab8390b2e7d167efb

Bernhard Voelker <berny>
Group administrator
Wed 24 Nov 2021 06:01:53 PM UTC, comment #13: 

Proposed patch attached.

(file #52339)

Bernhard Voelker <berny>
Group administrator
Sun 21 Nov 2021 03:33:46 PM UTC, comment #12: 

Thanks for the feedback and for testing some use cases.

Well, I decided to go the defensive way first because it's always
easier to extend functionality than to remove later.
My thought was that with the regular invocation without a starting point,
find(1) would implicitly assume '.' as the starting point.
In the -files0-from mode, doing the same would feel strange.
Therefore, outputting an error seemed the natural choice.

Of course, your request makes sense, but I wonder what would be
the best way to do it.

As you're mentioning the behavior of xargs(1), there seem 3 alternatives:

a) change -files0-from to silently ignore an empty file.
b) add an extra option like -ignore-files0-empty to tweak the behavior
   of the -files0-from option to ignore an empty input file.
c) add a sister option like -files0-from-ignore-empty (the name is too
   long, of course; it's to demonstrate).

Another choice could be the combination of a) and b), i.e., one would
need an additional option to get an error for an empty input file.

I have no strong preference, clearly a) is the easiest.
Any thoughts?

Bernhard Voelker <berny>
Group administrator
Sat 20 Nov 2021 01:05:43 PM UTC, comment #11: 

I very much welcome that new feature, as that at last fixes the long standing issue that you can't pass arbitrary file paths to "find" (such as "-my-file-", "(", "!" which you need to prefix with ./ as a work around, see also the (clunkier to use) -f option of BSDs for that).

Now, I can do (in Korn-like shells):


find -files0-from <(printf '%s\0' "${files[@]}") ... -print0 |
  find -files0-from - ...


And be sure none of the files in the $files array will be taken as a find option or predicate.

My only problem is that, when find is fed an empty list, we get a


find: file with starting points is empty: ‘(standard input)’


error (and a non zero exit status). I'd rather find not complained about it there, just like when it doesn't find any file or when using xargs -r0.

I can work around it using moreutils' ifne command as in:


(( ${#files[@]} == 0 )) ||
  find -files0-from <(printf '%s\0' "${files[@]}") ... |
    ifne find -files0-from - ...


But I'd rather avoid that dependency on a non-GNU utility.

What do you think? Should I raise it as a separate bug here?

Stephane Chazelas <stephanechazelas>
Sat 08 May 2021 12:16:34 PM UTC, comment #10: 
Bernhard Voelker <berny>
Group administrator
Thu 06 May 2021 10:27:38 PM UTC, comment #9: 

No news are good news.
Still I found a crash with certain compile options on GCC7.
Besides that, I fixed a couple of other issues (see below) and tested on:
openSUSE Factory (x86_64,i586,aarch64,armv7l,ppc,ppc64,ppc64le),
Solaris 11.3 SPARC, CentOS 7.8 on Power9, Debian Bullseye,
AIX 7.2 on Power8, MacOSX 11.2.3 (arm64), and OpenBSD-6.8.

The attached version 2 of the patch does:
- improve error diagnostics: use die().
- detect and prohibit cases where the given -files0-file FILE is associated with
  the same stream than standard input.
- avoid the "bad_cast" macro in the fallback "." case which was seen to crash
  with certain compiler options on GCC7.
- improve test coverage of code added/changed due to the -files0-from option.
- consistently use 'files0_from' as names inside the code.

Pushing soon.

(file #51391)

Bernhard Voelker <berny>
Group administrator
Thu 22 Apr 2021 08:10:59 PM UTC, comment #8: 

Here it is.
Please check, review, test.

One thing not yet handled: find prohibits the use of '-files0-from -'
together with the conflicting actions -ok and -okdir, but it doesn't
(yet?) prohibit other aliases of standard input like /dev/stdin:

  proggy | find -files0-from /dev/stdin -okdir smthng {} \;

Have fun,
Berny

(file #51309)

Bernhard Voelker <berny>
Group administrator
Wed 14 Apr 2021 11:01:26 PM UTC, comment #7: 

@James (#comment3): thanks for listing those restrictions; I also thought about them in this way.

Especially no. 1 (the seek position in the file) would maybe be tricky to to ensure.  But I think it's an extreme corner case to continue processing after find; e.g. count the entries in the list after 'find' has found e.g. the first file younger than 10 days and quits:


cat file \
  | { find -files0-from=- -mtime -10 -quit; \
      wc --files0-from=- -l; }


I'll start working on a patch - unless someone else already has something to show.

Bernhard Voelker <berny>
Group administrator
Wed 14 Apr 2021 04:39:12 PM UTC, comment #6: 

I tend to agree with Andreas.

James Youngman <jay>
Group administrator
Wed 14 Apr 2021 03:55:28 PM UTC, comment #5: 


comment #1:
[...]

> For a most-secure way, i.e., to avoid surprises with unusual file names,
> the input file should contain the entries separated by NUL characters.


I actually think if you are extending find in this non-portable way NUL separated input should be the only supported variant. We should not add new functionality that cannot handle "unsafe" filenames.

cu Andreas

Andreas Metzler <ametzler>
Wed 14 Apr 2021 01:13:47 PM UTC, comment #4: 

Ok now that discussion is off-topic, actually, but regarding this:

comment #3:

> We cannot use //x as a way to escape x, because in POSIX path names matching [^/] are `special` and have an implementation-defined significance, which `find` cannot know (find is intended to run also on non-GNU systems).


Interesting. I didn't know about that. But still, no show stopper in my estimation.

Another alternative would be prefixing it with a currenlty unused option/switch, e.g. "-e" like grep does for regular expressions starting with a dash, or bash's "echo" builtin command has with it's "-E" flag.

$ find -e -evil -e -pathnames -e -dont -e -scare -e -me -ls

and ...

$ find -e -e -ls

Paxsali <paxsali>
Wed 14 Apr 2021 09:20:30 AM UTC, comment #3: 

We cannot use //x as a way to escape x, because in POSIX path names matching [^/] are `special` and have an implementation-defined significance, which `find` cannot know (find is intended to run also on non-GNU systems).

The difficulties around -H/-L/-P could be resolved I think by making --file0 incompatible with -H.  That is, making it an error to specify both.

Other things we should document if we implement this option:

1. The seek position on the named file at the time find exits (normally, with -quit or in any other way) is not specified.
2. If --file0 is specified, no (other) starting points may be specified on the command line
3. If a file is listed more than once in the input file it is unspecified whether it is visited more than once (my motivation here is that we may need to worry about the interaction between this and symlink loop detection)
4. If the file is mutated during the operation of find, the result is unspecified.
5. The effect of empty records (i.e. \0\0) is unspecified.

By "unspecified" here I mean, may not work, may not do any specific thing, and this may change from platform to platform or findutils release to release.


James Youngman <jay>
Group administrator
Tue 13 Apr 2021 10:19:59 PM UTC, comment #2: 

There are more things to consider.

For instance, if a file in the input list is a directory, then the default settings ([-H] [-L] [-P]) need to apply, unless explicitly stated otherwise.

This means that if directories are elements of the input list, it should be treated equivalent as if they were given in the [starting-point...] section of paths and they will be recursively searched (unless one explicitly does not want that),
which must be explicitly mentioned in the manual, since it may confuse users otherwise, who might think the input list is absolute and nothing outside of it can be searched.

Btw TAR uses the same behavior with it's -T switch. If a file in the input list is a directory, it will be archived recursively, no matter if it's sub-files/dirs weren't elements of the provided input list (unless of course, one explicitly specifies --no-recursion).
It is my understanding that the same behavior is used/applied for this "new feature".

With regards to your concerns mentioned:

I agree and there is always the possibility to simply provide a new "mode of operation" for find, such that the named conflicting will not occur by design (your examples: -ok, -okdir).

About the "files or paths starting with -" problem I must admit I don't see why this should be a deal breaker for this particular feature-request.
The same basic problem exists for the current version without such a feature as you described it "as currently is" / "in general".

The easiest way to solve this issue, IMHO, is to use double slashes "//" at the beginning of a path spec to indicate a "literal" starting point, e.g.:

$ find -evil -pathnames -dont -scare -me -ls

When above "hypothetical find version" sees such path specs, it removes the "//" internally and treats the rest as a literal path specification.
This is just from the top of my head. It's off-topic, looks ugly, but it would work 100% as slashes and null bytes are the only special characters to consider.
Null bytes obviously you can't use, so you're stuck with doing something with slashes and since double slashes are "illogical", one could use that as a special prefix.
Maybe super-fancy new and cool programming languages will remove the null bytes restriction in the distant future... not that anybody misses null-bytes (where's my garlic and cross?).

Back to topic:

I take it as given that the implementation will consider such details as how to interpret the input list and provide multiple choices for it.
Null terminated lists should ofc be the minimun supported option, as they're the most powerful and safe ones, but I see no reason for not also supporting newline terminated input lists.

Paxsali <paxsali>
Tue 13 Apr 2021 09:37:42 PM UTC, comment #1: 

Thanks for the request, and the examples to illustrate the idea behind.

Actually, I suggested such option already in https://savannah.gnu.org/bugs/?58205 .
The discussion came from the problem when a given path name starts with a "-";
this screws up the argument parsing, and the only way out is to use absolute path names
or those relative to the current directory: './-somepath'.
It wasn't seen worth adding at that point.

Still, I see the issue that some other tool has pre-filtered path names and needs
to pass them to find(1) for further examination.  This is hard to achieve in a
secure and performant way.

For a most-secure way, i.e., to avoid surprises with unusual file names,
the input file should contain the entries separated by NUL characters.
The new option would be named '--files0-from=FILE' like in du(1) and wc(1)
from the GNU coreutils, and accept the special FILE name "-" to read from
standard input.


# Existing synopsis: pass starting points before expressions:
find [-H] [-L] [-P] [-D debugopts] [-Olevel] [--] [starting-point...] [expression]

# Alternative synopsis: read starting points from file:
find [-H] [-L] [-P] [-D debugopts] [-Olevel] -files0-from=FILE [--] [expression]

# e.g.
... | find -L -files0-from=- -type f ...


FWIW: using '-files0-from=-' would conflict with actions which require a
confirmation from the user: -ok and -okdir.  The implementation would/will
have to disallow this combination.

After all - and unless some other conflicts or drawbacks are raised here -,
I'm 60:40 for adding this option it.

Bernhard Voelker <berny>
Group administrator
Mon 12 Apr 2021 03:11:02 PM UTC, original submission:  

Hi all,

I was thinking about a very interesting, but admittedly niche use case.

What if I already have a narrowed list of files available, say from a previous command or stored in a file.

Now what I want is, that "find" instead of literally starting it's own filesystem search of the given paths given for files and folders etc...
...that "find" reads the list of input files from stdin or file.

Fantasy commandline:

$ find --files-from=input.txt -maxdepth 3 -type f -executable -ls

I just gave some dumb example from the top of my head.

In other words: "find" reads the files (and dirs) from input.txt (without scanning the filesystem!) and then further narrows them down by it's own options/directives, such as "-maxdepth 3" (relative from it's current working directory) and "-type f" and "-executable" and so forth.

Expected result: "find" prints in "-ls" (ls -dils) format all 1) regular files 2) that are at maximum 3 levels of depth starting from the CWD, and 3) are executable, but NOT from all files in the filesystem, starting from the CWD, and instead starting only from the files in the input list (input.txt).

More examples:

$  rg --files | find -c -name "*.c" -newer reference.c -used 7 -ls

This example reads files from ripgrep, which provides an already narrowed down scope of files in the filesystem that ignored SCM repositories and hidden files and such, and the result is further narrowed down by "find" specific syntax and options. In this case, any files output from "rg --files" are read by find via "-c" option (same as "--files-from -", read from stdin), if they are newer than "reference.c" and were accessed in the last 7 days since their last status change, then output that further narrowed list of files in "-ls" format.

Mind you, I can do "most" of those things using regular shell or "test" constructs, but not all.

I think that "find" has a very powerful set of tests and the killer feature is the way you can combine them in a compact sense.

$ find --files-from=input.txt -type f -executable -newer reference.c -ls

Above command unquestionably easier to do for most experienced find users than:

$ while IFS=$'\n' read file; do

> [[ -f "${file}" -a -x "${file}" -a "${file}" -nt "reference.c" ]] && ls -dils "${file}"
> done < input.txt


You could think of more complex examples, where the advantages of selection and action via "find" directives (options/tests) is far more powerful, precise, compact and easier to understand than what would be the equivalent of an arbitrarily complex shell script using standard "test" facilities.

"find" is, in fact, superior to "test" in many ways. It just offers far more possibilities for tests.

I was wondering what other users think about making use of "find's" directives on a given list of files, instead of the current behavior of recursing into the filesystem or given paths.

Please discuss/consider and thank you in advance.

Paxsali <paxsali>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #52339:  0001-find-allow-files0-from-input-file-to-be-empty.patch added by berny (4KiB - text/x-patch - [PATCH] find: allow -files0-from input file to be empty)
file #51391:  0001-find-add-files0-from-option.patch added by berny (31KiB - text/x-patch - [PATCHv2] find: add -files0-from option)
file #51309:  0001-find-add-files0-from-option.patch added by berny (28KiB - text/x-patch - [PATCH] find: add -files0-from option)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by stephanechazelas (Posted a comment)
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by berny (Posted a comment)
  • -email is unavailable- added by paxsali (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-02-01 berny Open/ClosedOpen Closed
        Fixed ReleaseNone 4.9.0
    2021-11-24 berny Attached File- Added 0001-find-allow-files0-from-input-file-to-be-empty.patch, #52339
    2021-05-08 berny StatusCode Review Fixed
    2021-05-06 berny Attached File- Added 0001-find-add-files0-from-option.patch, #51391
    2021-04-22 berny Attached File- Added 0001-find-add-files0-from-option.patch, #51309
        StatusNone Code Review
    2021-04-13 berny Severity3 - Normal 1 - Wish
        Assigned toNone berny

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code