bugfindutils - Bugs: bug #60383, [feature-request] let find read...


bug #60383: [feature-request] let find read files from stdin or file.

Submitted by:  Paxsali <paxsali>
Submitted on:  Mon 12 Apr 2021 03:11:02 PM UTC  
Category:  find Severity:  1 - Wish
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  Bernhard Voelker <berny>
Originator Name:  Open/Closed:  Open
Release:  None Fixed Release:  None

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Sat 08 May 2021 12:16:34 PM UTC, comment #10: 
Bernhard Voelker <berny>
Project AdministratorIn charge of this item.
Thu 06 May 2021 10:27:38 PM UTC, comment #9: 

No news are good news.
Still I found a crash with certain compile options on GCC7.
Besides that, I fixed a couple of other issues (see below) and tested on:
openSUSE Factory (x86_64,i586,aarch64,armv7l,ppc,ppc64,ppc64le),
Solaris 11.3 SPARC, CentOS 7.8 on Power9, Debian Bullseye,
AIX 7.2 on Power8, MacOSX 11.2.3 (arm64), and OpenBSD-6.8.

The attached version 2 of the patch does:
- improve error diagnostics: use die().
- detect and prohibit cases where the given -files0-file FILE is associated with
  the same stream than standard input.
- avoid the "bad_cast" macro in the fallback "." case which was seen to crash
  with certain compiler options on GCC7.
- improve test coverage of code added/changed due to the -files0-from option.
- consistently use 'files0_from' as names inside the code.

Pushing soon.

(file #51391)

Bernhard Voelker <berny>
Project AdministratorIn charge of this item.
Thu 22 Apr 2021 08:10:59 PM UTC, comment #8: 

Here it is.
Please check, review, test.

One thing not yet handled: find prohibits the use of '-files0-from -'
together with the conflicting actions -ok and -okdir, but it doesn't
(yet?) prohibit other aliases of standard input like /dev/stdin:

  proggy | find -files0-from /dev/stdin -okdir smthng {} \;

Have fun,

(file #51309)

Bernhard Voelker <berny>
Project AdministratorIn charge of this item.
Wed 14 Apr 2021 11:01:26 PM UTC, comment #7: 

@James (#comment3): thanks for listing those restrictions; I also thought about them in this way.

Especially no. 1 (the seek position in the file) would maybe be tricky to to ensure.  But I think it's an extreme corner case to continue processing after find; e.g. count the entries in the list after 'find' has found e.g. the first file younger than 10 days and quits:

cat file \
  | { find -files0-from=- -mtime -10 -quit; \
      wc --files0-from=- -l; }

I'll start working on a patch - unless someone else already has something to show.

Bernhard Voelker <berny>
Project AdministratorIn charge of this item.
Wed 14 Apr 2021 04:39:12 PM UTC, comment #6: 

I tend to agree with Andreas.

James Youngman <jay>
Project Administrator
Wed 14 Apr 2021 03:55:28 PM UTC, comment #5: 

comment #1:

> For a most-secure way, i.e., to avoid surprises with unusual file names,
> the input file should contain the entries separated by NUL characters.

I actually think if you are extending find in this non-portable way NUL separated input should be the only supported variant. We should not add new functionality that cannot handle "unsafe" filenames.

cu Andreas

Andreas Metzler <ametzler>
Wed 14 Apr 2021 01:13:47 PM UTC, comment #4: 

Ok now that discussion is off-topic, actually, but regarding this:

comment #3:

> We cannot use //x as a way to escape x, because in POSIX path names matching //[^/] are `special` and have an implementation-defined significance, which `find` cannot know (find is intended to run also on non-GNU systems).

Interesting. I didn't know about that. But still, no show stopper in my estimation.

Another alternative would be prefixing it with a currenlty unused option/switch, e.g. "-e" like grep does for regular expressions starting with a dash, or bash's "echo" builtin command has with it's "-E" flag.

$ find -e -evil -e -pathnames -e -dont -e -scare -e -me -ls

and ...

$ find -e -e -ls

Paxsali <paxsali>
Wed 14 Apr 2021 09:20:30 AM UTC, comment #3: 

We cannot use //x as a way to escape x, because in POSIX path names matching //[^/] are `special` and have an implementation-defined significance, which `find` cannot know (find is intended to run also on non-GNU systems).

The difficulties around -H/-L/-P could be resolved I think by making --file0 incompatible with -H.  That is, making it an error to specify both.

Other things we should document if we implement this option:

1. The seek position on the named file at the time find exits (normally, with -quit or in any other way) is not specified.
2. If --file0 is specified, no (other) starting points may be specified on the command line
3. If a file is listed more than once in the input file it is unspecified whether it is visited more than once (my motivation here is that we may need to worry about the interaction between this and symlink loop detection)
4. If the file is mutated during the operation of find, the result is unspecified.
5. The effect of empty records (i.e. \0\0) is unspecified.

By "unspecified" here I mean, may not work, may not do any specific thing, and this may change from platform to platform or findutils release to release.

James Youngman <jay>
Project Administrator
Tue 13 Apr 2021 10:19:59 PM UTC, comment #2: 

There are more things to consider.

For instance, if a file in the input list is a directory, then the default settings ([-H] [-L] [-P]) need to apply, unless explicitly stated otherwise.

This means that if directories are elements of the input list, it should be treated equivalent as if they were given in the [starting-point...] section of paths and they will be recursively searched (unless one explicitly does not want that),
which must be explicitly mentioned in the manual, since it may confuse users otherwise, who might think the input list is absolute and nothing outside of it can be searched.

Btw TAR uses the same behavior with it's -T switch. If a file in the input list is a directory, it will be archived recursively, no matter if it's sub-files/dirs weren't elements of the provided input list (unless of course, one explicitly specifies --no-recursion).
It is my understanding that the same behavior is used/applied for this "new feature".

With regards to your concerns mentioned:

I agree and there is always the possibility to simply provide a new "mode of operation" for find, such that the named conflicting will not occur by design (your examples: -ok, -okdir).

About the "files or paths starting with -" problem I must admit I don't see why this should be a deal breaker for this particular feature-request.
The same basic problem exists for the current version without such a feature as you described it "as currently is" / "in general".

The easiest way to solve this issue, IMHO, is to use double slashes "//" at the beginning of a path spec to indicate a "literal" starting point, e.g.:

$ find //-evil //-pathnames //-dont //-scare //-me -ls

When above "hypothetical find version" sees such path specs, it removes the "//" internally and treats the rest as a literal path specification.
This is just from the top of my head. It's off-topic, looks ugly, but it would work 100% as slashes and null bytes are the only special characters to consider.
Null bytes obviously you can't use, so you're stuck with doing something with slashes and since double slashes are "illogical", one could use that as a special prefix.
Maybe super-fancy new and cool programming languages will remove the null bytes restriction in the distant future... not that anybody misses null-bytes (where's my garlic and cross?).

Back to topic:

I take it as given that the implementation will consider such details as how to interpret the input list and provide multiple choices for it.
Null terminated lists should ofc be the minimun supported option, as they're the most powerful and safe ones, but I see no reason for not also supporting newline terminated input lists.

Paxsali <paxsali>
Tue 13 Apr 2021 09:37:42 PM UTC, comment #1: 

Thanks for the request, and the examples to illustrate the idea behind.

Actually, I suggested such option already in https://savannah.gnu.org/bugs/?58205 .
The discussion came from the problem when a given path name starts with a "-";
this screws up the argument parsing, and the only way out is to use absolute path names
or those relative to the current directory: './-somepath'.
It wasn't seen worth adding at that point.

Still, I see the issue that some other tool has pre-filtered path names and needs
to pass them to find(1) for further examination.  This is hard to achieve in a
secure and performant way.

For a most-secure way, i.e., to avoid surprises with unusual file names,
the input file should contain the entries separated by NUL characters.
The new option would be named '--files0-from=FILE' like in du(1) and wc(1)
from the GNU coreutils, and accept the special FILE name "-" to read from
standard input.

# Existing synopsis: pass starting points before expressions:
find [-H] [-L] [-P] [-D debugopts] [-Olevel] [--] [starting-point...] [expression]

# Alternative synopsis: read starting points from file:
find [-H] [-L] [-P] [-D debugopts] [-Olevel] -files0-from=FILE [--] [expression]

# e.g.
... | find -L -files0-from=- -type f ...

FWIW: using '-files0-from=-' would conflict with actions which require a
confirmation from the user: -ok and -okdir.  The implementation would/will
have to disallow this combination.

After all - and unless some other conflicts or drawbacks are raised here -,
I'm 60:40 for adding this option it.

Bernhard Voelker <berny>
Project AdministratorIn charge of this item.
Mon 12 Apr 2021 03:11:02 PM UTC, original submission:  

Hi all,

I was thinking about a very interesting, but admittedly niche use case.

What if I already have a narrowed list of files available, say from a previous command or stored in a file.

Now what I want is, that "find" instead of literally starting it's own filesystem search of the given paths given for files and folders etc...
...that "find" reads the list of input files from stdin or file.

Fantasy commandline:

$ find --files-from=input.txt -maxdepth 3 -type f -executable -ls

I just gave some dumb example from the top of my head.

In other words: "find" reads the files (and dirs) from input.txt (without scanning the filesystem!) and then further narrows them down by it's own options/directives, such as "-maxdepth 3" (relative from it's current working directory) and "-type f" and "-executable" and so forth.

Expected result: "find" prints in "-ls" (ls -dils) format all 1) regular files 2) that are at maximum 3 levels of depth starting from the CWD, and 3) are executable, but NOT from all files in the filesystem, starting from the CWD, and instead starting only from the files in the input list (input.txt).

More examples:

$  rg --files | find -c -name "*.c" -newer reference.c -used 7 -ls

This example reads files from ripgrep, which provides an already narrowed down scope of files in the filesystem that ignored SCM repositories and hidden files and such, and the result is further narrowed down by "find" specific syntax and options. In this case, any files output from "rg --files" are read by find via "-c" option (same as "--files-from -", read from stdin), if they are newer than "reference.c" and were accessed in the last 7 days since their last status change, then output that further narrowed list of files in "-ls" format.

Mind you, I can do "most" of those things using regular shell or "test" constructs, but not all.

I think that "find" has a very powerful set of tests and the killer feature is the way you can combine them in a compact sense.

$ find --files-from=input.txt -type f -executable -newer reference.c -ls

Above command unquestionably easier to do for most experienced find users than:

$ while IFS=$'\n' read file; do

> [[ -f "${file}" -a -x "${file}" -a "${file}" -nt "reference.c" ]] && ls -dils "${file}"
> done < input.txt

You could think of more complex examples, where the advantages of selection and action via "find" directives (options/tests) is far more powerful, precise, compact and easier to understand than what would be the equivalent of an arbitrarily complex shell script using standard "test" facilities.

"find" is, in fact, superior to "test" in many ways. It just offers far more possibilities for tests.

I was wondering what other users think about making use of "find's" directives on a given list of files, instead of the current behavior of recursing into the filesystem or given paths.

Please discuss/consider and thank you in advance.

Paxsali <paxsali>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #51391:  0001-find-add-files0-from-option.patch added by berny (31KiB - text/x-patch - [PATCHv2] find: add -files0-from option)
file #51309:  0001-find-add-files0-from-option.patch added by berny (28KiB - text/x-patch - [PATCH] find: add -files0-from option)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by berny (Posted a comment)
  • -email is unavailable- added by paxsali (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-05-08 berny StatusCode Review => Fixed
    2021-05-06 berny Attached File- => Added 0001-find-add-files0-from-option.patch, #51391
    2021-04-22 berny Attached File- => Added 0001-find-add-files0-from-option.patch, #51309
        StatusNone => Code Review
    2021-04-13 berny Severity3 - Normal => 1 - Wish
        Assigned toNone => berny

    Back to the top

    Powered by Savane 3.7