bugfindutils - Bugs: bug #39824, Add suggestion 'ls | tr \\n \\0 |...

 
 

bug #39824: Add suggestion 'ls | tr \\n \\0 | xargs -0 command' to xargs man page

Submitter:  Edward J. Huff <ejhuff>
Submitted:  Tue 20 Aug 2013 07:50:53 AM UTC
   
 
Category:  documentation Severity:  3 - Normal
Item Group:  None Status:  Works For Me
Privacy:  Public Assigned to:  jay
Originator Name:  Edward Huff Open/Closed:  Closed
Release:  None Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 12 Jan 2014 08:47:27 PM UTC, comment #4: 

As you point out

find ... -print0 | tr \\n\\0 \\1\\n | grep ... | tr \\1\\n \\n\\0 | xargs -0,

is cumbersome.  But it's also pointless since find has -regex.

There are plenty of misconceptions about Unix filenames.   One of them, as you point out, is that they don't/can't contain blanks.  Of course, they can.   But a much more pervasive misapprehension is that file names are text.  They are not; instead they are byte sequences, in which system calls consider '/' to be special, and which are terminated by \0 when presented to the API.

The critical difference is that nothing on a Unix system requires or enforces that a directory name has the same character encoding as its parent.   So there is no single locale configuration which will allow one to correctly process a general path name as text.   This includes the "C" locale (as a trivial example, it fails to correctly collate characters beyond <DEL>).  For this reason processing path names with grep or sed, while common and popular, is probably never going to be without problems, though I suppose one could invent a kind of they're-just-bytes locale definition.

Anyway, because this kind of use of tr perpetuates misunderstandings about what bytes values can exist in file names and what they mean, I wouldn't really want to include a "tr" command acting on file names in the documentation, except perhaps as a cautionary example.

James Youngman <jay>
Group administrator
Tue 20 Aug 2013 06:08:45 PM UTC, comment #3: 

I agree that the proposed paragraph is not adequate, and have written more extensive revisions to the xargs man page, to be submitted immediately.

I will not be using |tr \\n \\0|xargs -0 anymore because |xargs -d\\n takes fewer keystrokes.  I wasn't aware of that option until I started working on a revision to the man page.

In reply to



Solving the problem for whitespace-but-not-newlines only solves part of the problem. Examples in manpages should, as far as possible, not come with pitfalls.


It is very easy to create filenames containing blanks, tabs, quote marks, or backslashes using desktop GUI applications.  Those applications refuse to create filenames containing newlines.  Thus, failure to handle newlines is not so much of a pitfall for desktop users.

Since there are many useful tools for massaging file names that lack --null options (such as grep and sed), and since GUI users (who will be reading the man page to learn to use xargs) still need to use shell commands to deal with many filenames at once, none of which contain newlines, the manpage should not begin by suggesting that the default options are useful.




Lack of a -0 option isn't a problem in the case you suggest (since the program being invoked, xargs, has -0).


(Actually, the program is ls | grep).  Now grep does have a
-Z option, but that is for the case when grep is reporting a list of files which contain an instance of the pattern.  It would be nice if grep had an option to treat newlines like other whitespace and nulls like newlines, but it doesn't.

Well, ok, if I know that filenames do not contain \01 characters, I can write find ... -print0 | tr \\n\\0 \\1\\n | grep ... | tr \\1\\n \\n\\0 | xargs -0, but this is a lot of keystrokes when I happen to know the filenames do not contain newlines.


find . -maxdepth 1 -mindepth 1 -regex regexp -exec stat {} \+



requires more keystrokes and new users have no chance of coming up with it from memory.  Efficiency is just not a problem on modern single-user systems.

Edward J. Huff <ejhuff>
Tue 20 Aug 2013 05:03:07 PM UTC, comment #2: 

I don't think the suggested extra paragraph is an improvement, really.   Solving the problem for whitespace-but-not-newlines only solves part of the problem.   Examples in manpages should, as far as possible, not come with pitfalls.

Lack of a -0 option isn't a problem in the case you suggest (since the program being invoked, xargs, has -0).   Even if it did lack that option, the "tr" approach is suboptimal compared to, for example

find . -maxdepth 1 -mindepth 1 -regex regexp -exec stat {} \+

James Youngman <jay>
Group administrator
Tue 20 Aug 2013 09:47:34 AM UTC, comment #1: 

Ok, I see that tr \\n \\0 | xargs -0 is exactly equivalent to xargs -d\\n.  Still, filenames containing blanks are easily created using gui save dialogs whereas newlines cannot be inserted that way.

So I will submit a patch to the documentation which will recommend always using -d\\n when -0 is not possible,
and will add -d\\n to all of the invocations which do not
use -0.

Edward J. Huff <ejhuff>
Tue 20 Aug 2013 07:50:53 AM UTC, original submission:  

After this paragraph in the xargs man page:

       Because Unix filenames can contain blanks
       and newlines, this default  behaviour  is
       often  problematic;  filenames containing
       blanks and/or  newlines  are  incorrectly
       processed  by xargs.  In these situations
       it is better to use the -0 option,  which
       prevents such problems.   When using this
       option you will need to ensure  that  the
       program  which  produces  the  input  for
       xargs also uses a  null  character  as  a
       separator.   If  that program is GNU find
       for example, the -print0 option does this
       for you.

Insert a new paragraph:

In the much more common case, Unix filenames
contain blank spaces but not newlines, and
furthermore, the input program has no -0 option.
In that case, you can ensure correct operation
by inserting tr \\n \\0 into the
pipeline just before xargs -0.  For example,

   ls | grep regexp | tr \\n \\0 | xargs -0 stat

to stat all filenames matching regexp.

Edward J. Huff <ejhuff>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #28896:  xargs.1 added by ejhuff (15KiB - application/x-troff-man - A second draft. Better, I think.)
file #28887:  xargs.diff.txt added by ejhuff (7KiB - text/plain - not actually a patch, just diff text of original vs. edited man page)
file #28884:  patch added by ejhuff (5KiB - application/octet-stream - Suggested revisions to the man page in Fedora 19, which seems to be identical to the page on man7.org.)
file #28885:  xargs.txt added by ejhuff (16KiB - text/plain - Suggested revisions to the man page in Fedora 19, which seems to be identical to the page on man7.org.)
file #28886:  xargs.1 added by ejhuff (14KiB - application/x-troff-man - Suggested revisions to the man page in Fedora 19, which seems to be identical to the page on man7.org.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by ejhuff (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-01-12 jay StatusNone Works For Me
        Open/ClosedOpen Closed
        SummaryAdd suggestion \'ls | tr \\\\n \\\\0 | xargs -0 command\' to xargs man page Add suggestion 'ls | tr \\n \\0 | xargs -0 command' to xargs man page
    2013-08-21 ejhuff Attached File- Added xargs.1, #28896
    2013-08-20 ejhuff Attached File- Added xargs.diff.txt, #28887
    2013-08-20 ejhuff Attached File- Added patch, #28884
        Attached File- Added xargs.txt, #28885
        Attached File- Added xargs.1, #28886
    2013-08-20 jay Assigned toNone jay
        SummaryAdd suggestion \'ls | tr \\\\n \\\\0 | xargs -0 command\' to xargs man page Add suggestion 'ls | tr \\n \\0 | xargs -0 command' to xargs man page

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code