bugfindutils - Bugs: bug #38356, xargs -P option creates wrong...

 
 

bug #38356: xargs -P option creates wrong result

Submitted by:  Congshi Huang <congshi>
Submitted on:  Wed 20 Feb 2013 05:52:40 PM UTC  
 
Category: xargsSeverity: 3 - Normal
Item Group: Wrong resultStatus: Fixed
Privacy: PublicAssigned to: James Youngman <jay>
Originator Name: Open/Closed: Closed
Release: 4.4.2Fixed Release: 4.5.12

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Fri 05 Apr 2013 08:54:06 PM UTC, comment #8:

Changesets [842a39c81] and [1611816e].

Ole Tange <tange>
Sun 24 Mar 2013 11:15:53 PM UTC, comment #7:

I applied Bernhard's patch, with slightly different wording (and updates to NEWS and ChangeLog), and also a following patch which also updates the Texinfo documentaiton.

Both patches are now in git.

James Youngman <jay>
Project AdministratorIn charge of this item.
Sat 23 Feb 2013 12:09:01 PM UTC, comment #6:

"grep --line-buffered" doesn't help - the parallel grep processes
still write to stdout at the same time. Proper locking is needed.

man page patch attached.

Bernhard Voelker <berny>
Sat 23 Feb 2013 11:10:23 AM UTC, comment #5:

Try "grep --line-buffered", it might help avoiding output mangling.

I suppose xargs documentation could mention that it is duty of parallelized processes executed by xargs -P to care of potential output mangling.

Dmitry V. Levin <ldv>
Sat 23 Feb 2013 05:49:30 AM UTC, comment #4:

It is acceptable if the line order is different due to the –P option. If the line is mangled as garbage, the option is questionable. If the results is useless, so called fast and efficient is meaningless.
The best way is to fix it if possible. If it cannot be fixed due to complexity or out of the scope, remove this option is reasonable. At least, some warning message should be in man page.

Congshi Huang <congshi>
Thu 21 Feb 2013 11:23:31 PM UTC, comment #3:

xargs -P is not designed to make sure the output does not mix up. GNU Parallel, however, is. So use GNU Parallel instead and it will work just fine:

find . -type f \( -name "\.[cC]" -o -name "\.[hH]" -o -name "*\.[cC][pP][pP]" \) | parallel -P0 -I"{}" grep -Hn $1 $2 $3 $4 $5 $6 $7 $8 $9 "{}"

See: http://www.gnu.org/software/parallel/man.html#differences_between_xargs_and_gnu_parallel and http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Propose: Close - not a bug (but a problem due to the design).

Ole Tange <tange>
Thu 21 Feb 2013 02:49:50 PM UTC, comment #2:

The problem is that all parallel instances of grep have the
output file open for writing at the same time, and they do
not care about locking it. Therefore, I do not consider this
an xargs bug as xargs can nothing do about it.

Instead, you could synchronize writing of grep's output by an
intermediate wrapper script which obtains an exclusive lock on
stdout before writing the result:

#!/bin/sh
matches=$( grep "$@" )
flock 1
printf "%s\n" "$matches"

Please note that you could also flock stdout before grep to
avoid the intermediate 'matches' variable, but this would
obvisouly lock out all other parallel grep processes until
this one has finished.
For larger program output, you may need to use a temporary
file instead of 'matches'.

Bernhard Voelker <berny>
Wed 20 Feb 2013 06:32:43 PM UTC, comment #1:

The script in Discussion are changed when submitting somehow. Please refer to attachment.

(file #27485, file #27486)

Congshi Huang <congshi>
Wed 20 Feb 2013 05:52:40 PM UTC, original submission:

I searched and did not find related fix/bug

script ff
find . -type f \( -name "\.[cC]" -o -name "\.[hH]" -o -name "*\.[cC][pP][pP]" \) | xargs -I"{}" grep -Hn $1 $2 $3 $4 $5 $6 $7 $8 $9 "{}"

script ff0
find . -type f \( -name "\.[cC]" -o -name "\.[hH]" -o -name "*\.[cC][pP][pP]" \) | xargs -P0 -I"{}" grep -Hn $1 $2 $3 $4 $5 $6 $7 $8 $9 "{}"

cd to kernel source /usr/src/linux
and run the scripts as
ff -w long > /dev/shm/1
and
ff0 -w long > /dev/shm/0

The result is different and the line in /dev/shm/0 are mangled and the file size is smaller.

Slackware 13.0.0.0.0

uname -a
Linux test 2.6.36.2-smp #1 SMP PREEMPT Tue Dec 11 15:28:02 PST 2012 i686 Intel(R) Xeon(R) CPU E5345 @ 2.33GHz GenuineIntel GNU/Linux

xargs --version
xargs (GNU findutils) 4.4.2
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Eric B. Decker, James Youngman, and Kevin Dalley.
Built using GNU gnulib version e5573b1bad88bfabcda181b9e0125fb0c52b7d3b

Congshi Huang <congshi>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #27509:  xargs-man-mangled-stdout.patch added by berny (1kB - text/x-patch - [PATCH] xargs: document need to manage access to resources with -P)
file #27486:  ff0 added by congshi (143B - application/octet-stream)
file #27485:  ff added by congshi (139B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jay (Posted a comment)
  • -unavailable- added by ldv (Posted a comment)
  • -unavailable- added by tange (Posted a comment)
  • -unavailable- added by berny (Posted a comment)
  • -unavailable- added by congshi (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 24 Sep 2013 10:06:01 PM UTCjayOpen/ClosedOpen=>Closed
      Fixed ReleaseNone=>4.5.12
    Sun 24 Mar 2013 11:15:53 PM UTCjayAssigned toNone=>jay
    Sun 24 Mar 2013 11:15:52 PM UTCjayStatusNone=>Fixed
    Sat 23 Feb 2013 12:06:24 PM UTCbernyAttached File-=>Added xargs-man-mangled-stdout.patch, #27509
    Wed 20 Feb 2013 06:32:43 PM UTCcongshiAttached File-=>Added ff0, #27486
    Wed 20 Feb 2013 06:32:42 PM UTCcongshiAttached File-=>Added ff, #27485

    Back to the top


    Powered by Savane 3.1-cleanup