bugfindutils - Bugs: bug #18554, Document construct -exec sh -c...


bug #18554: Document construct -exec sh -c 'foo "$@" bar' {} +

Submitted by:  Egmont Koblinger <egmont>
Submitted on:  Tue 19 Dec 2006 12:58:50 PM UTC  
Category:  documentation Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  James Youngman <jay>
Originator Name:  Open/Closed:  Closed
Release:  4.2.29 Fixed Release:  4.3.3

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Thu 01 Mar 2007 08:45:47 PM UTC, comment #16: 

I have made the indicated documentation change in both the 4.2.x CVS code and the 4.3.x code.

James Youngman <jay>
Project AdministratorIn charge of this item.
Fri 22 Dec 2006 08:35:40 PM UTC, comment #15: 

One other thing to throw in the mix:

POSIX has no requirements on -execdir.  Therefore, I argue that it might make sense to have -execdir be as friendly as possible, and allow the {} to appear anywhere, rather than only immediately before the +:

find startpoint -tests ... -execdir rsync {} /dest +

On the other hand, the current consistency between -exec and -execdir may be easier to explain in the documentation, so I won't be too upset if you ignore this particular suggestion.

Eric Blake <ericb>
Project Administrator
Fri 22 Dec 2006 05:11:47 PM UTC, comment #14: 

Reopened as a documentation problem.  

I (now) understand the dscussion about the non-security-problem.  This area is complex enough to deserve better findutils documentation.  Hence I reopened the bug as such.

James Youngman <jay>
Project AdministratorIn charge of this item.
Fri 22 Dec 2006 02:16:40 PM UTC, comment #13: 

Egmont, you raise an interesting point about the limit on the number of arguments.  There is nothing in POSIX about such a limit, but I don't think there is anything that forbids it either.

The -exec sh -c 'utility ... "$@" ...' sh {} + technique is well known and often recommended on comp.unix.shell, so if there is a potential problem with it on Linux I would say that is a (quality of implementation) problem for Linux, not a problem that POSIX needs to address. It could easily be mitigated by putting a smaller limit in find on the number of arguments it aggregates.  A limit of 32000 would leave plenty of room for additional fixed arguments to be passed to the utility.

Geoff Clare <geoffclare>
Fri 22 Dec 2006 01:23:37 PM UTC, comment #12: 

I agree that
find startpoint -tests ... -exec sh -c 'scp "$@" remote:/dest' sh {} +
has no security problems, because sh is not parsing the arguments.  The only time you have a security problem when passing arbitrary filenames to sh is when sh is allowed to parse metacharacters in those arguments, but here, the only metacharacters that sh can parse are contained in the -c argument, 'scp "$@" remote:/dest'.

Eric Blake <ericb>
Project Administrator
Fri 22 Dec 2006 01:06:13 PM UTC, comment #11: 

I can't see any security problem with that construct, either.

Egmont Koblinger <egmont>
Fri 22 Dec 2006 01:00:27 PM UTC, comment #10: 

In addition to ARG_MAX (# bytes of args + environ for exec(), according to linux/limits.h) the Linux kernel has another limit: the number of command line arguments (including the 0th) cannot exceed 32767. Is this a standard, too? Is find/xargs aware of this limit? (I can't see a constant defined for it in linux/limits.h.)

Using the inline shell script wrapper you've just shown, one can hit this limit. If you pass several options to scp, you may end up the scp command taking more arguments than sh gets. So even if "find" correctly takes care not to pass too many arguments to sh, it may eventually try to pass too many for scp, and there's nothing "find" could do against it.

I still can't understand POSIX people... Slightly breaking backwards compatibility is okay, but let's not introduce a useful new feature, let's rather create ugly workarounds for them, because that would break bw compatibility even a little bit more. Nevermind... :-)

How about allowing this feature only if POSIXLY_CORRECT is unset?

Egmont Koblinger <egmont>
Fri 22 Dec 2006 12:49:15 PM UTC, comment #9: 

James, I believe the way I suggested using "sh -c" is perfectly safe.  The security problem you are thinking of is associated with uses where the {} is part of the command, e.g.

find ... -exec sh -c 'ls -l {}' \;

which is not portable anyway.  (POSIX says {} only needs to be substituted if it is a separate argument.)

Geoff Clare <geoffclare>
Fri 22 Dec 2006 12:39:45 PM UTC, comment #8: 

I'm marking this feature request "Invalid" on the grounds that the requested change would be contrary to POSIX requirements (using the status "Won't Fix" seemed wrong since there is nothing really broken).  


James Youngman <jay>
Project AdministratorIn charge of this item.
Fri 22 Dec 2006 12:37:15 PM UTC, comment #7: 

Thanks for the (pretty much) official interpretation Geoff.

The findutils documentation would not include an example showing how to accomplish this with "sh -c" though, because of the disastrous security implications of passing untrusted data such as filenames to the shell.  

In fact I'd recommend that the POSIX revision you're talking about explicitly point out that this (along with almost any other use of "find ... -exec sh -c ...") is bad security practice.

(I know privileged operations are out of scope for POSIX, but I'd guess that a form of words can be found that preserves the useful guidance)

James Youngman <jay>
Project AdministratorIn charge of this item.
Fri 22 Dec 2006 11:54:39 AM UTC, comment #6: 

POSIX definitely only allows + to be special when it immediately follows {}.  I can see that because the text doesn't say "immediately follows" it could be misinterpreted, but the requirement is clear from the specified form of the primary:

-exec utility_name [argument ...] ;
-exec utility_name [argument ...] {} +

The reason it was done that way was to minimize the chances of breaking existing uses of -exec of the form:

find ... -exec utility [args]... {} args... + [args]... \;

The POSIX rationale gives an example command that would still be broken even with the rules as they are, but the hope was that such uses were very rare.

If there is a need to have arguments after the aggregated ones this can easily be accomplished using "sh -c":

find startpoint -tests ... -exec sh -c 'scp "$@" remote:/dest' sh {} +

The current draft of the next revision of POSIX.1 includes an example on the find page that shows this kind of usage.

Geoff Clare <geoffclare>
Wed 20 Dec 2006 05:58:20 PM UTC, comment #5: 

I'm just reading the standard you have linked, and I still can't see it explicitely stating that "{}" must immediately precede the "+" sign.

However, it says: "An argument containing only the two characters "{}" shall be replaced by the set of aggregated pathnames [...]" -- I'd definitely used other wording if only the last argument could have contained "{}". The current wording "_An_ argument" gives me a strong feeling that this can be any argument.

Later it says: "If more than one argument containing only the two characters "{}" is present, the behavior is unspecified." -- Yet again, this sentence wouldn't make sense if "{}" was allowed only at one particular place, right before the "+". IMHO this sentence also implies that "{}" can reside anywhere.

I agree that having more "{}" arguments before the "+" is a brain-damaged case and I don't really care how this is handled. I'm only taking about exactly one "{}" somewhere (anywhere) before the "+".

For rsync, you're right, I didn't know about its --from* options. But for scp there is no such option, so it's not really straightforward to do something equivavelent to
find startpoint -tests ... -exec scp {} remote:/dest +
Sure there are possible workarounds involving tricky options of xargs, or using an scp-wrapper script; but why make things so complicated if they could have been done much simpler?

Egmont Koblinger <egmont>
Wed 20 Dec 2006 05:22:18 PM UTC, comment #4: 

Sorry, the previous update was the victim of rich-text processing.  That last command should have been

printf "funny\nscary\n" | xargs -I X -L10 echo hello X world

(I originally used an underscopre but Savannah uses those to denote italics, wiki-style)

James Youngman <jay>
Project AdministratorIn charge of this item.
Wed 20 Dec 2006 05:20:11 PM UTC, comment #3: 

OK, I have now spent some time re-reading the standard at http://www.opengroup.org/onlinepubs/009695399/utilities/find.html.

The last thing before the "+" must be {}.   That {} will be replaced by an aggregated set of pathnames.     Hence there is no possibility to support something like "find . -exec foo {} bar +" while conforming to the standard.

This begs the question as to whether something like "find a b  -exec echo {} MARK {} +" should work, and if so how.  The standard states "If more than one argument containing only the two characters "{}" is present, the behavior is unspecified.".  So anything could happen, really.   What happens is that find issues an error message complaining of inappropriate usage and exits without doing anything.

So the result is that find supports "find .... -exec .... {} +" but the final + must be immediately preceded by {}.  While find could support a {} earlier in the command line, it doesn't currently support that, thet feature would not be useful to you as far as I can see, and the standard allows the current behaviour.

So, I think POSIX disallows the usage you would like, and the closest conforming usage pattern that find could support doesn't appear to help you. 

As far as I understand things though you want to be able to do something like this:

find startpoint -tests ... -exec rsync {} /dest +

The closest thing I can think of to what you want is this:

find a b -print0 | rsync --files-from=- --from0 /dest

Some versions of xargs support the use of both -I and -L in such a way that {} can expand to more than one filename.  GNU findutils doesn't support that.  That construct could perhaps be used to do what you want.  I suggest you join the findutils mailing list and start a discussion about what

printf "funny\nscary\n" | xargs -I   -L10 echo hello world

should do.  On other words, should -L reverse the effect of -I?  Please join the list before sending the mail, or you may miss some of the responses.

James Youngman <jay>
Project AdministratorIn charge of this item.
Tue 19 Dec 2006 07:31:59 PM UTC, comment #2: 

Yeah well... in the example, {} and + and next to each other. But I don't think we can count on the example, as it doesn't even mention that {} can be part of a longer argument, e.g. "exec utility_name [argument ...] foo{}bar +". So it is just an example, not a complete reference of all the valid features.

The description is a little bit unclear to me. "plus sign that follows {}" -- I guess that in "{} foo bar +" the + sign follows {}, whereas for example in "+ foo {}" it doesn't, it precedes (this is I guess what the last sentence of your quote could have referred to, which also means that the command line parsing algorithm described in my original post is wrong, a standalone + sign occuring before the first {} shouldn't be treated as special).

Does the English verb "follow" implies that it's the immediate successor? I don't think so, but I'm not so good in English to feel such nuances.

I extremely hate when slowly evolving standards set back innovation. And I also hate when standards use ambiguous wording. Anyway, there are far too many situations where current GNU utilities override or extend POSIX. I'm not thinking in POSIX, I'm thinking how useful or powerful a utility is. IMHO this feature wouldn't hurt anyone, but would make "find" more powerful.

Egmont Koblinger <egmont>
Tue 19 Dec 2006 06:14:04 PM UTC, comment #1: 

I just want to add that allowing stuff between {} and + would be a contradiction to POSIX/Susv3 afaict:

»-exec  utility_name  [argument ...] ;
-exec  utility_name  [argument ...]   {} +

[...] Only a plus sign that follows an argument containing the two characters "{}" shall punctuate the end of the primary expression. Other uses of the plus sign shall not be treated as special.«

cu andreas

Andreas Metzler <ametzler>
Tue 19 Dec 2006 12:58:50 PM UTC, original submission:  

Currently find refuses to have command line options between {} and +. This way the filenames found by "find" can only be put at the end of the argument list.

If you want to copy or move some files, the "cp" and "mv" commands just happen to have a --target-directory option so that you can use an unusual order, specify the target first and the source files later. I think these were added to support using cp or mv with xargs or find -exec {} +.

There are other utilities, such as scp, rsync and probably a whole lot more that do not have this option, they require the target directory to be the last option. Currently they can't really be used in combination with find.

So please add support for having extra arguments between {} and +. As I see, this wouldn't cause ambiguity or other problems when parsing the command line. When an -exec or -execdir is seen, first "find" should look for the next ';' or '+' argument. When it is found and it is a '+', it should check that there's exactly one '{}' before that. Checking whether the constructed command line would be too long is easy too; this time it doesn't matter if these extra args are at the end or at the beginning, they can be taken into account at once.

Egmont Koblinger <egmont>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by ericb (Posted a comment)
  • -email is unavailable- added by geoffclare (Posted a comment)
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by egmont (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2007-04-15 jay Open/ClosedOpen => Closed
        Fixed ReleaseNone => 4.3.3
    2007-03-01 jay StatusNone => Fixed
    2007-03-01 jay Summaryfeat req: -exec cmd {} more args + => Document construct -exec sh -c 'foo "$@" bar' {} +
    2006-12-22 ametzler Carbon-CopyRemoved 20807 => -
    2006-12-22 jay Categoryfind => documentation
        StatusInvalid => None
        Open/ClosedClosed => Open
    2006-12-22 jay Open/ClosedOpen => Closed
    2006-12-22 jay StatusNone => Invalid
    2006-12-20 jay Assigned toNone => jay

    Back to the top

    Powered by Savane 3.5