bugfindutils - Bugs: bug #64253, Suggestion - Add support for...


bug #64253: Suggestion - Add support for libmagic and xattr

Submitter:  None
Submitted:  Thu 25 May 2023 07:18:47 PM UTC
Category:  find Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  berny
Originator Name:  Jay Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  None
Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Mon 05 Jun 2023 07:55:13 AM UTC, comment #7: 

Fair enough. I've added it to rawhide. I'm just accepting that errors become the text being matched, and document the fact. But yeah, it is wierd. Debugging search criteria could require using %w/%W to look for error messages.

Maybe it was just designed for use by file(1) itself. But there are wrappers for libmagic in python etc., so it might have many clients. Maybe it was a deliberate choice so clients didn't need to worry about errors. The assumption might be that the user always sees them. But it would be nice if it were also possible for clients to be able to choose how they want to handle errors.

raf <raf>
Sat 03 Jun 2023 09:39:50 PM UTC, comment #6: 

No, I don't plan to add a -printf format for mime/magic.
The output doesn't sort in well in the other output formats anyway,
because it's quite verbose.

If I would think about adding such a format, I'd go starting to make
use of the "%{...}" syntax which is currently reserved for future; hence
a "%{magic}" and "%{mime}" would fit - not sure about rawhide, though.

Re. magic/mime implementation:

First of all, it's the first time find looks at file content, and
that processing (open/read/lookup/close) is by magnitudes slower than
the other tests in find(1).

Having played with it for some time now, I have major qualms to
add libmagic support:

a) file(1) has some more options than the -i (for mime output) option.
Of course, they're all available via flags in libmagic.
But it would be strange to have to add further flags or knobs
in find(1) to support these options as well.
But people will require it - "just that one little thing".
That are discussions we have to avoid.

b) error handling:
While libmagic has the flag MAGIC_ERROR to indicate an error
via return value NULL instead of placing the error string
into the magic result buffer, that does not work for
all cases, e.g. the simple open/EPERM case: we still get the error
message "regular file; no read permission" as magic string
instead of NULL.
Likewise file(1):

$ file -E /etc/sudoers; echo $?
/etc/sudoers: regular file, no read permission

I looked into file/libmagic code, and found various such places.
Also the library function magic_error does not indicate
an unreadable file as error.
We'd have to single out every such error by string matching,
which I'm not willing to do.  Proper error handling seems to
be tough with libmagic.  I'm not sure how and which other
projects are using libmagic, but the current state of error
handling doesn't work for how find(1) would need it.

It's not really find's business to look at the content of files,
and there are already ways to do the filtering with file(1) as shown
below (*): searching for magic strings or mime types can already be
done "the UNIX way" (i.e., one tool for one purpose).

Even if one likes to continue after the "magic check" with post-processing
via the find(1) command again, it is safe with the -files0-from option
for any kind of exotic file names incl. control or newline chars:

$ find -type f -size -40000c -mtime -10 -exec file -00 '{}' + \
    | sed -nz 'h;n;/^C source/{g;p}' \
    | find -files0-from - -printf "* %p\n  size: %s\n  inode: %i\n"
* ./find/defs.h
  size: 19707
  inode: 216534
* ./find/util.c
  size: 29571
  inode: 217027
* ./find/pred.c
  size: 37310
  inode: 152256

Obviously, that would be much easier if file(1) would provide options
to filter by certain magic/mime strings (as it does to exclude some tests).

I was quite enthusiastic about adding libmagic in the beginning,
but with the issues described above - above all the problematic error
handling -, I'm afraid I can't add libmagic support now.
I'm inclined to abandon or throw away my local work.

(*) The other day, maybe another one comes up with the idea that there's
a little library to get the content of cell A1 of a spreadsheet file,
or the title of a PDF file.  I don't believe it's a good idea to link
all those libraries, but instead encourage people to write tools which
fit well into UNIX pipes and transport remaining file names with
safe and Zero-terminated strings.

Bernhard Voelker <berny>
Group administrator
Sat 03 Jun 2023 03:44:39 AM UTC, comment #5: 

Are there going to be new corresponding -printf % format conversions? If so, what are the letters going to be (if they are single letters)? I'd like to use the same notation in my rawhide program if possible.

The only good choices for me are:

    %o %O
    %q %Q
    %w %W

They are the only pairs of letters I have left, and I think it makes sense to try to use an uppercase and lowercase version of the same letter for the (magic number) file type and mime type.

I have a preference for %w=magic and %W=mime, but only because "w" is the only available letter where I can think of a good mnemonic (i.e. "what"). Although "q" for "qualities" is OK too.

raf <raf>
Sat 27 May 2023 08:04:19 AM UTC, comment #4: 

Speaking for a distro which tends to expose a lot of this choice to users (Gentoo), it should be fine as long as there's explicit configure args for each.

Lots of other packages do stuff like this too. It's only really a problem if it's "automagic" (use-if-installed, no option to control it) or mandatory. Truly optional stuff with --with-x or --enable-x is fine (or --without/--disable, you get  the idea).

Also, thanks! This sounds great.

Sam James <thesamesam>
Sat 27 May 2023 07:54:52 AM UTC, comment #3: 

The request for xattr is duplicate to several others already.

Bernhard Voelker <berny>
Group administrator
Sat 27 May 2023 07:54:03 AM UTC, comment #2: 

I started working on -[i]magic/-[i]mime.

Obviously, this potentially pulls in - depending on the build configuration
of libmagic - additional dependencies to libzstd, liblzma, libbz2, and libz.
As find(1) is used in many bootstrapping scenarios, this might mean that
those environments may need/want to build two flavors of findutils:
a) one small one without libmagic for the bootstrapping part, and
b) a larger one with libmagic for the final system.
At least the downstream maintainer has a choice ...

Bernhard Voelker <berny>
Group administrator
Fri 26 May 2023 11:39:27 PM UTC, comment #1: 

Hi, As far as I know, all the versions of find that have an -xattr predicate only allow searching by the name of the extended attribute (or perhaps just the fact of their existence?).

I wrote a find alternative called rawhide (rh) that supports searching by extended attribute names and values (glob or regex). So if you can't wait for it in find, you can use rawhide. If the find developers think this is a good idea, they are welcome to plunder rawhide for its extended attribute code. It supports extended attributes on Linux, FreeBSD, macOS, Solaris, and Cygwin.

It doesn't support libmagic but I might add it. I wonder how useful it is. If it is useful, it would make queries much faster than a separate file(1) process per candidate file (but that works too). I'd like to think of some good examples first to motivate it.

I could be wrong, but I don't think a -mime predicate adds much value. Since mimetypes are determined by file name extension anyway, the same queries can be done with normal globbing, and I suspect the resulting find commands would often be shorter that way.

P.S. I don't think the emergence of .zip domains will have the effect on operating systems that you anticipate. The use of misleading double extensions has been around for years (e.g., "somethinginteresting.jpg                        .exe"). This tld development doesn't seem very different.

raf <raf>
Thu 25 May 2023 07:18:47 PM UTC, original submission:  

I've gone through past patches, bugs and suggestions and I was surprised I could not find any mention of the obvious idea of adding support for libmagic (magic | file | etc), so thought that it might be a useful feature to find, here are some ideas.

Also while I think about this, and with the growing use of extended attributes by applications, it may also make sense to think about including some sort of xattr filter too.
Most filesystems on which find is er..., found, have xattr capability and it has been present in almost all contemporary operating systems kernels for the best part of two decades.
man xattr

Homepage here:
file | magic | libmagic

Google have convinced ICANN that file extension like TLDs such as '.zip' are a good idea. This sets a ball rolling, others will folow. This means OSs will finally have to adapt and accept that deciding what a file is for, just based on a parts of the given file name is naive and will have to make use of the actual contents (as IMO as they should have a long time ago).  Very soon extensions mean little other than a hint to the human.

Features like these would allow searching of folders for type rather than extension, without extra levels of scripting.

Currently - without find : this is inefficient as we can't add filter without adding code and we are already spawning thousands of find instances.

for f in ./*; do
    if [[ $(file -b $f) == ".*PE32 executable.*" ]]; then
        my_command $f;

Currently - with find : We need xargs and sed and so have to worry about whitespace paths and filenames, we are also spawning several sub-commands.

find -type f |
 xargs file |
  sed -n 's/:.*PE32 executable.*/p' |
   xargs my_command

Conceptual new usage (syntax usage tbd)

# For libmagic
find . -magic ".*ELF.*x86_64.*" -not -path "./bin/*" -exec my_command  {} \;
find . -mime ".*application/x-dosexec.*" -not -path "./bin/*" -mv {} /Quaranteen/

# For xattr
find . -xattr 1 app.browser.url -xattr-substr 1 "http://download.org" -delete
find . -xattr 1 os.hash.blake2 -not -xattr-re 1 "^ERROR:bad hash.*" -exec hash whirlpool {} ;\



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by thesamesam (Posted a comment)
  • -email is unavailable- added by berny (Posted a comment)
  • -email is unavailable- added by raf (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.


    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-06-03 berny StatusIn Progress None
    2023-05-27 berny StatusNone In Progress
        Assigned toNone berny

    Back to the top

    Powered by Savane 3.12.
    Corresponding source code