bugfindutils - Bugs: bug #38356, xargs -P option creates wrong...


bug #38356: xargs -P option creates wrong result

Submitter:  Congshi Huang <congshi>
Submitted:  Wed 20 Feb 2013 05:52:40 PM UTC
Category:  xargs Severity:  3 - Normal
Item Group:  Wrong result Status:  Fixed
Privacy:  Public Assigned to:  jay
Originator Name:  Open/Closed:  Closed
Release:  4.4.2 Fixed Release:  4.5.12
* Mandatory Fields

Add a New Comment (Rich Markup)

Comment Type & Canned Response:


( Jump to the original submission )

Fri 05 Apr 2013 08:54:06 PM UTC, comment #8: 

Changesets [842a39c81] and [1611816e].

Ole Tange <tange>
Sun 24 Mar 2013 11:15:53 PM UTC, comment #7: 

I applied Bernhard's patch, with slightly different wording (and updates to NEWS and ChangeLog), and also a following patch which also updates the Texinfo documentaiton.

Both patches are now in git.

James Youngman <jay>
Project Administrator
Sat 23 Feb 2013 12:09:01 PM UTC, comment #6: 

"grep --line-buffered" doesn't help - the parallel grep processes
still write to stdout at the same time. Proper locking is needed.

man page patch attached.

Bernhard Voelker <berny>
Project Administrator
Sat 23 Feb 2013 11:10:23 AM UTC, comment #5: 

Try "grep --line-buffered", it might help avoiding output mangling.

I suppose xargs documentation could mention that it is duty of parallelized processes executed by xargs -P to care of potential output mangling.

Dmitry V. Levin <ldv>
Sat 23 Feb 2013 05:49:30 AM UTC, comment #4: 

It is acceptable if the line order is different due to the –P option. If the line is mangled as garbage, the option is questionable. If the results is useless, so called fast and efficient is meaningless.
The best way is to fix it if possible. If it cannot be fixed due to complexity or out of the scope, remove this option is reasonable. At least, some warning message should be in man page.

Congshi Huang <congshi>
Thu 21 Feb 2013 11:23:31 PM UTC, comment #3: 

xargs -P is not designed to make sure the output does not mix up. GNU Parallel, however, is. So use GNU Parallel instead and it will work just fine:

find . -type f \( -name "\.[cC]" -o -name "\.[hH]" -o -name "*\.[cC][pP][pP]" \) | parallel -P0 -I"{}" grep -Hn $1 $2 $3 $4 $5 $6 $7 $8 $9 "{}"

See: http://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel and http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Propose: Close - not a bug (but a problem due to the design).

Ole Tange <tange>
Thu 21 Feb 2013 02:49:50 PM UTC, comment #2: 

The problem is that all parallel instances of grep have the
output file open for writing at the same time, and they do
not care about locking it. Therefore, I do not consider this
an xargs bug as xargs can nothing do about it.

Instead, you could synchronize writing of grep's output by an
intermediate wrapper script which obtains an exclusive lock on
stdout before writing the result:

  matches=$( grep "$@" )
  flock 1
  printf "%s\n" "$matches"

Please note that you could also flock stdout before grep to
avoid the intermediate 'matches' variable, but this would
obvisouly lock out all other parallel grep processes until
this one has finished.
For larger program output, you may need to use a temporary
file instead of 'matches'.

Bernhard Voelker <berny>
Project Administrator
Wed 20 Feb 2013 06:32:43 PM UTC, comment #1: 

The script in Discussion are changed when submitting somehow. Please refer to attachment.

(file #27485, file #27486)

Congshi Huang <congshi>
Wed 20 Feb 2013 05:52:40 PM UTC, original submission:  

I searched and did not find related fix/bug

script ff
find . -type f \( -name "*\.[cC]" -o -name "*\.[hH]" -o -name "*\.[cC][pP][pP]" \) | xargs -I"{}" grep -Hn $1 $2 $3 $4 $5 $6 $7 $8 $9 "{}"

script ff0
find . -type f \( -name "*\.[cC]" -o -name "*\.[hH]" -o -name "*\.[cC][pP][pP]" \) | xargs -P0 -I"{}" grep -Hn $1 $2 $3 $4 $5 $6 $7 $8 $9 "{}"

cd to kernel source /usr/src/linux
and run the scripts as
    ff -w long > /dev/shm/1
    ff0 -w long > /dev/shm/0

The result is different and the line in /dev/shm/0 are mangled and the file size is smaller.


uname -a               
Linux test #1 SMP PREEMPT Tue Dec 11 15:28:02 PST 2012 i686 Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz GenuineIntel GNU/Linux

xargs --version
xargs (GNU findutils) 4.4.2
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Eric B. Decker, James Youngman, and Kevin Dalley.
Built using GNU gnulib version e5573b1bad88bfabcda181b9e0125fb0c52b7d3b

Congshi Huang <congshi>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #27509:  xargs-man-mangled-stdout.patch added by berny (1KiB - text/x-patch - [PATCH] xargs: document need to manage access to resources with -P)
file #27486:  ff0 added by congshi (143B - application/octet-stream)
file #27485:  ff added by congshi (139B - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by jay (Posted a comment)
  • -email is unavailable- added by ldv (Posted a comment)
  • -email is unavailable- added by tange (Posted a comment)
  • -email is unavailable- added by berny (Posted a comment)
  • -email is unavailable- added by congshi (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.





    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-09-24 jay Open/ClosedOpen Closed
        Fixed ReleaseNone 4.5.12
    2013-03-24 jay Assigned toNone jay
    2013-03-24 jay StatusNone Fixed
    2013-02-23 berny Attached File- Added xargs-man-mangled-stdout.patch, #27509
    2013-02-20 congshi Attached File- Added ff0, #27486
    2013-02-20 congshi Attached File- Added ff, #27485

    Back to the top

    Powered by Savane 3.9