bugfindutils - Bugs: bug #39824, Add suggestion 'ls | tr \\n \\0 |...

 
 

bug #39824: Add suggestion 'ls | tr \\n \\0 | xargs -0 command' to xargs man page

Submitted by:  Edward J. Huff <ejhuff>
Submitted on:  Tue 20 Aug 2013 07:50:53 AM UTC  
 
Category: documentationSeverity: 3 - Normal
Item Group: NoneStatus: Works For Me
Privacy: PublicAssigned to: James Youngman <jay>
Originator Name: Edward HuffOpen/Closed: Closed
Release: NoneFixed Release: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 12 Jan 2014 08:47:27 PM UTC, comment #4:

As you point out

find ... -print0 | tr \\n\\0 \\1\\n | grep ... | tr \\1\\n \\n\\0 | xargs -0,

is cumbersome. But it's also pointless since find has -regex.

There are plenty of misconceptions about Unix filenames. One of them, as you point out, is that they don't/can't contain blanks. Of course, they can. But a much more pervasive misapprehension is that file names are text. They are not; instead they are byte sequences, in which system calls consider '/' to be special, and which are terminated by \0 when presented to the API.

The critical difference is that nothing on a Unix system requires or enforces that a directory name has the same character encoding as its parent. So there is no single locale configuration which will allow one to correctly process a general path name as text. This includes the "C" locale (as a trivial example, it fails to correctly collate characters beyond <DEL>). For this reason processing path names with grep or sed, while common and popular, is probably never going to be without problems, though I suppose one could invent a kind of they're-just-bytes locale definition.

Anyway, because this kind of use of tr perpetuates misunderstandings about what bytes values can exist in file names and what they mean, I wouldn't really want to include a "tr" command acting on file names in the documentation, except perhaps as a cautionary example.

James Youngman <jay>
Project AdministratorIn charge of this item.
Tue 20 Aug 2013 06:08:45 PM UTC, comment #3:

I agree that the proposed paragraph is not adequate, and have written more extensive revisions to the xargs man page, to be submitted immediately.

I will not be using |tr \\n \\0|xargs -0 anymore because |xargs -d\\n takes fewer keystrokes. I wasn't aware of that option until I started working on a revision to the man page.

In reply to


Solving the problem for whitespace-but-not-newlines only solves part of the problem. Examples in manpages should, as far as possible, not come with pitfalls.


It is very easy to create filenames containing blanks, tabs, quote marks, or backslashes using desktop GUI applications. Those applications refuse to create filenames containing newlines. Thus, failure to handle newlines is not so much of a pitfall for desktop users.

Since there are many useful tools for massaging file names that lack --null options (such as grep and sed), and since GUI users (who will be reading the man page to learn to use xargs) still need to use shell commands to deal with many filenames at once, none of which contain newlines, the manpage should not begin by suggesting that the default options are useful.


Lack of a -0 option isn't a problem in the case you suggest (since the program being invoked, xargs, has -0).


(Actually, the program is ls | grep). Now grep does have a
-Z option, but that is for the case when grep is reporting a list of files which contain an instance of the pattern. It would be nice if grep had an option to treat newlines like other whitespace and nulls like newlines, but it doesn't.

Well, ok, if I know that filenames do not contain \01 characters, I can write find ... -print0 | tr \\n\\0 \\1\\n | grep ... | tr \\1\\n \\n\\0 | xargs -0, but this is a lot of keystrokes when I happen to know the filenames do not contain newlines.


find . -maxdepth 1 -mindepth 1 -regex regexp -exec stat {} \+


requires more keystrokes and new users have no chance of coming up with it from memory. Efficiency is just not a problem on modern single-user systems.

Edward J. Huff <ejhuff>
Tue 20 Aug 2013 05:03:07 PM UTC, comment #2:

I don't think the suggested extra paragraph is an improvement, really. Solving the problem for whitespace-but-not-newlines only solves part of the problem. Examples in manpages should, as far as possible, not come with pitfalls.

Lack of a -0 option isn't a problem in the case you suggest (since the program being invoked, xargs, has -0). Even if it did lack that option, the "tr" approach is suboptimal compared to, for example

find . -maxdepth 1 -mindepth 1 -regex regexp -exec stat {} \+

James Youngman <jay>
Project AdministratorIn charge of this item.
Tue 20 Aug 2013 09:47:34 AM UTC, comment #1:

Ok, I see that tr \\n \\0 | xargs -0 is exactly equivalent to xargs -d\\n. Still, filenames containing blanks are easily created using gui save dialogs whereas newlines cannot be inserted that way.

So I will submit a patch to the documentation which will recommend always using -d\\n when -0 is not possible,
and will add -d\\n to all of the invocations which do not
use -0.

Edward J. Huff <ejhuff>
Tue 20 Aug 2013 07:50:53 AM UTC, original submission:

After this paragraph in the xargs man page:

Because Unix filenames can contain blanks
and newlines, this default behaviour is
often problematic; filenames containing
blanks and/or newlines are incorrectly
processed by xargs. In these situations
it is better to use the -0 option, which
prevents such problems. When using this
option you will need to ensure that the
program which produces the input for
xargs also uses a null character as a
separator. If that program is GNU find
for example, the -print0 option does this
for you.

Insert a new paragraph:

In the much more common case, Unix filenames
contain blank spaces but not newlines, and
furthermore, the input program has no -0 option.
In that case, you can ensure correct operation
by inserting tr \\n \\0 into the
pipeline just before xargs -0. For example,

ls | grep regexp | tr \\n \\0 | xargs -0 stat

to stat all filenames matching regexp.

Edward J. Huff <ejhuff>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #28896:  xargs.1 added by ejhuff (15kB - application/x-troff-man - A second draft. Better, I think.)
file #28887:  xargs.diff.txt added by ejhuff (7kB - text/plain - not actually a patch, just diff text of original vs. edited man page)
file #28884:  patch added by ejhuff (5kB - application/octet-stream - Suggested revisions to the man page in Fedora 19, which seems to be identical to the page on man7.org.)
file #28885:  xargs.txt added by ejhuff (16kB - text/plain - Suggested revisions to the man page in Fedora 19, which seems to be identical to the page on man7.org.)
file #28886:  xargs.1 added by ejhuff (14kB - application/x-troff-man - Suggested revisions to the man page in Fedora 19, which seems to be identical to the page on man7.org.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jay (Posted a comment)
  • -unavailable- added by ejhuff (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 10 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 12 Jan 2014 08:47:27 PM UTCjayStatusNone=>Works For Me
      Open/ClosedOpen=>Closed
      SummaryAdd suggestion \'ls | tr \\\\n \\\\0 | xargs -0 command\' to xargs man page=>Add suggestion 'ls | tr \\n \\0 | xargs -0 command' to xargs man page
    Wed 21 Aug 2013 11:47:44 AM UTCejhuffAttached File-=>Added xargs.1, #28896
    Tue 20 Aug 2013 06:39:24 PM UTCejhuffAttached File-=>Added xargs.diff.txt, #28887
    Tue 20 Aug 2013 06:34:35 PM UTCejhuffAttached File-=>Added patch, #28884
      Attached File-=>Added xargs.txt, #28885
      Attached File-=>Added xargs.1, #28886
    Tue 20 Aug 2013 05:03:07 PM UTCjayAssigned toNone=>jay
      SummaryAdd suggestion \'ls | tr \\\\n \\\\0 | xargs -0 command\' to xargs man page=>Add suggestion 'ls | tr \\n \\0 | xargs -0 command' to xargs man page

    Back to the top


    Powered by Savane 3.1-cleanup