bugfindutils - Bugs: bug #64816, support ls' --quoting-style= option

 
 

bug #64816: support ls' --quoting-style= option

Submitter:  Christoph Anton Mitterer <calestyo>
Submitted:  Thu 26 Oct 2023 02:56:27 AM UTC
   
 
Category:  documentation Severity:  2 - Minor
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  4.9.0 Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 29 Oct 2023 10:55:41 PM UTC, comment #7: 


> well, you were asking for a GNU extension as well ...


Arguably, but at least in the GNU world, there is only one find implementation used (though there are other program like fd, which do something similar), while there are many different shells used.

Most notably, ash-based shells are used in many areas (e.g. Debian uses it as /bin/sh and thus for most scripts, busybox and klib-utils ship an ash-based sh).
And while bash does already support read -d '', ash-based shell's don't - and the POSIX revision that includes it isn't even out yet.


> we can suggest and add a lot of options and features,
> but to keep the code as compact as possible, we should
> encourage people to use what's there, i.e., use the GNU
> toolbox.  Therefore, before talking about what to add,
> let's take the time to talk about why to add it, and what
> kind of problem to solve with it.


Seems reasonable.

My main points for adding such feature would be:
1) Homogeneity with other GNU tools, especially coreutils whose ls provides already a way to quote the output like that.

find, as of now and not assuming -print0, uses various styles:
With -ls it seems to be a format that does quote, but is still different from what ls would do. e.g. find -ls':
./\360\221\231\243\360\221\231\244\360\221\231\246
./g\ h
vs ls':
𑙣𑙤𑙦
'g h'
etc.

I'd say it would make sense to align that to what ls does, which is IMO easier to read.


2) Usefulness of the printed data.

With -print and when connected to a terminal, it seems to do what ls did long ago (so called 'literal' style, despite it's not really literal) and use a ? instead of problematic characters.
Which is IMO quite useless, not only when there are multiple files which differ only in the non-printable character, but also when there's a file that would really use a ?.

If the default would change here, then sure, some people will complain (as they did with ls), but it should be only user human-readable output that changes - and if anyone fakes a terminal to be connected while still using the data in scripts... well... own fault.

With output not connected to a terminal output really seems to be literal, but that breaks as soon as files contains newlines, which - while uncommon - is still completely valid to have.

While I don't think the default should be changed in this case, it would still be nice to have a format that can be reused (which of course -print0 would also give).



3) Alternative to -print0
As said, right now it's not yet even possible to write a portable shell script which can read that. In command substitution NUL is not allowed (and would likely fail if there are more than one record) and read -d '' is so far not everywhere available.

Also using it sees more cumbersome to me than e.g. simply doing something like:
  ls $(find … --quoting-style=shell-escape)
(and yes it's clear that this particular example can also be made with -exec or xargs).

Christoph Anton Mitterer <calestyo>
Sun 29 Oct 2023 10:07:45 PM UTC, comment #6: 

P.S. I can see the need to continue processing in subsequent shell loops, and the problem is on the receiving side to take over arguments from NUL-separated input.

Meanwhile the problem seem to be solved there already:


$ ls -1
'????????????'
'a'$'\n''b'
'a\n'
'b\nc'
'c'$'\t''d'
'd\n\n'
'e'$'\r''f'
--foo
'g h'
'i'$'\n''j'
'k'$'\n\n'
'\n'
strange-files


$ find  -type f -print0 | while IFS=  read -rd '' file; do ls -logd -- "$file"; done
-rw-r--r-- 1 0 Oct 29 00:15 './a'$'\n''b'
-rw-r--r-- 1 0 Oct 29 00:16 './i'$'\n''j'
-rw-r--r-- 1 0 Oct 29 00:16 './g h'
-rw-r--r-- 1 0 Oct 29 00:16 ./strange-files
-rw-r--r-- 1 0 Oct 29 00:16 ./--foo
-rw-r--r-- 1 0 Oct 29 00:16 './e'$'\r''f'
-rw-r--r-- 1 0 Oct 29 01:01 './b\nc'
-rw-r--r-- 1 0 Oct 29 01:01 './d\n\n'
-rw-r--r-- 1 0 Oct 29 00:16 './k'$'\n\n'
-rw-r--r-- 1 0 Oct 29 01:01 './a\n'
-rw-r--r-- 1 0 Oct 29 00:16 './c'$'\t''d'
-rw-r--r-- 1 0 Oct 29 00:16 './????????????'
-rw-r--r-- 1 0 Oct 29 00:16 './\n'


Bernhard Voelker <berny>
Group administrator
Sat 28 Oct 2023 11:27:26 PM UTC, comment #5: 


> [...] most of the POSIX shell and utilities have no support for printing or reading NUL delimited, it's only GNU extensions [...]


well, you were asking for a GNU extension as well ...

But back to the requirement.

Of course, we can suggest and add a lot of options and features, but to keep the code
as compact as possible, we should encourage people to use what's there,
i.e., use the GNU toolbox.  Therefore, before talking about what to add,
let's take the time to talk about why to add it, and what kind of problem to
solve with it.  Eventually, there are already several other existing solutions
out there, and this would be a strong argument against adding new things.

TBH I'd not consider the for-loop example in the original report as sufficient as an
argument to add something in find/xargs, because there are several ways to
do such processing safely and efficient already.

So what is the concrete problem to be solved (which cannot already be solved
in the one or the other way)?

Bernhard Voelker <berny>
Group administrator
Sat 28 Oct 2023 04:56:25 PM UTC, comment #4: 


> Actually, NUL-terminated processing is the safest way of
> file name processing with tools in shells


Yes, but most of the POSIX shell and utilities have no support for printing or reading NUL delimited, it's only GNU extensions (well there will actually be some limited support in the next revision of POSIX, see https://austingroupbugs.net/view.php?id=243 ... but as far as I can see, this is only for xargs and read).

So once POSIX' read has -d (as specified now in POSIX and e.g. already used in bash) it will be possible to read a list of NUL delimited strings into shell variables.

But most other standard utilities like grep, sed, wc or shell language elements like command substitution do not support NUL delimited modes.
Thus many things will still not be possible.


> The point here is that the output should be safe no matter
> what malicious file an attacker might have placed, i.e., it
> should not disturb the terminal settings.


Isn't that also what I'm asking for with a --quoting-style= option?
If quoted properly, the output could be re-used as literal in the shell, without the need to read everything line by line with the future IFS='' read -r -d '' ... and even with that some things will still be impossible, e.g. re-use the read string in eval, since POSIX' printf won't have %q yet.

Christoph Anton Mitterer <calestyo>
Sat 28 Oct 2023 02:01:20 PM UTC, comment #3: 


>  [...] shells don't work very well with NULL delimited output.


I'm not sure I follow this statement.
Actually, NUL-terminated processing is the safest way of file name
processing with tools in shells, because the NUL character is the only one
(apart from a directory separator) which cannot be part of a file name.

Therefore, instead of going the route further down to use escaped strings
in find/xargs for further processing, the better way is IMO to give better
support of NUL-terminated in-/output to all tools.
The recent addition of the '-files0-from ...' option in find(1) was exactly
doing that.

The above is addressing the area of safely processing of file names in combination
with other tools via shell scripts/pipes, etc.

The discussion about 'find -ls' displaying information to the terminal and which
quoting style to be chosen best is a different one.
The point here is that the output should be safe no matter what malicious file
an attacker might have placed, i.e., it should not disturb the terminal settings.

Bernhard Voelker <berny>
Group administrator
Thu 26 Oct 2023 11:01:57 PM UTC, comment #2: 

Oh and just for the records:

If I'm not mistaken, bash's implementation of `%q` in its `printf` builtin can be found at:
https://github.com/bminor/bash/blob/ec8113b9861375e4e17b3307372569d429dec814/builtins/printf.def#L595-L644

with the actual work horses being:
https://github.com/bminor/bash/blob/ec8113b9861375e4e17b3307372569d429dec814/lib/sh/strtrans.c#L230-L317

or:

https://github.com/bminor/bash/blob/ec8113b9861375e4e17b3307372569d429dec814/lib/sh/shquote.c#L261-L312

(notice that these are unofficial git mirrors)


Also note, that for such functionality to be useful in the way I propose, it would be crucial that the escaping is done right, so that and arbitrary input string is quoted in a manner so that it can really be safely re-used as input (e.g. in `eval`) without any expansions/substitutions/etc. happening.

Christoph Anton Mitterer <calestyo>
Thu 26 Oct 2023 09:52:08 AM UTC, comment #1: 

Maybe it would also make sense to add the same functionality to xargs.

The idea is, that (POSIX compatible) shells don't work very well with NULL delimited output.

If some program produces such, it's output could be piped into xargs -0 ... and if that would now have an output mode that produces a string that is re-usable in the shell, one could rather easily get e.g. strings with trailing newlines into shell variables.

Something like:
eval "set -- $(  printf 'a\n\0b\nc\0d\n\n' | xargs -0 -- --quoting-style=shell-escape )"

where the printf simulates an arbitrary program that outputs byte strings NULL-delimited, which are then read by xargs and printed again but escaped for the shell to be re-used as input (without any command substitutions, etc. happening)... and in this example, set as the new positional parameters, which might be quite handy for e.g. shell functions.


[ Also notice, that this is not easily doable with shell command substitution ( $(... ) ) as that drops trailing newlines.
And yes I know, there are hacks around that (using sentinel values) but making the truly portable is something between tricky/fragile to not possible. ]


Just an idea though.

Christoph Anton Mitterer <calestyo>
Thu 26 Oct 2023 02:56:27 AM UTC, original submission:  

Hey.

The manpage says:
       -printf, -fprintf
              If the output is not going to a terminal, it is printed as-is.  Otherwise, …
              The directives %f,  %h,  %l,
              %p  and %P are quoted.  This quoting is performed in the same way as for GNU ls.


Let's see, I have a directory with lots of unusual filenames, which ls -1 prints like this:
# ls -1
'a'$'\n''b'
'c'$'\t''d'
'e'$'\r''f'
'g h'
'i'$'\n''j'
'k'$'\n\n'
strange-files
--foo
'\n'
????????????


# find . -mindepth 1 -maxdepth 1
./--foo
./a?b
./????????????
./\n
./g h
./c?d
./e?f
./i?j
./strange-files
./k??

This seems to be ls' old default quoting style of "literal" which was changed 2015 with https://github.com/coreutils/coreutils/commit/109b9220cead6e979d22d16327c4d9f8350431cc .
Nowadays it uses actually usable output ;-) (quoting style "shell-escape").

I think it would be most useful if find would provide a counterpart that allows setting the various quoting styles of ls.

Especially this should also work when output goes to a terminal.

So one could do e.g.
find … --quoting-style=shell-escape | \
while IFS='' read -r QUOTED_PATH; do
   eval 'PATH="${QUOTED_PATH}"'

   do_more_with "$PATH"
done

which should support even trailing newlines and the likes.


Of course, such escaping style would need to escape any characters considered special in POSIX shells,... which I guess ls' quoting does.

I'd further guess/hope that this also includes any extensions by bash.


Cheers,
Chris.

Christoph Anton Mitterer <calestyo>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jay (Updated the item)
  • -email is unavailable- added by berny (Posted a comment)
  • -email is unavailable- added by calestyo (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-05-19 jay Severity3 - Normal 2 - Minor

    Back to the top

    Powered by Savane 3.13-12ab.
    Corresponding source code