bugfindutils - Bugs: bug #61664, Messy output using xargs -P

 
 

bug #61664: Messy output using xargs -P

Submitter:  Kirill Zhdanov <storenth>
Submitted:  Mon 13 Dec 2021 05:04:57 AM UTC
   
 
Category:  xargs Severity:  3 - Normal
Item Group:  Wrong result Status:  Invalid
Privacy:  Public Assigned to:  berny
Originator Name:  Open/Closed:  Closed
Release:  4.7.0 Fixed Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 14 Dec 2021 12:12:54 AM UTC, comment #2: 

P.S. One way to ensure that the launched processes are using different
resources is by passing the --process-slot-var=N option.

P.P.S. Furthermore, your example could avoid spawning so many grep(1)
processes (and therefore processing the input file again and again) by passing
the patterns as input via -f (depending on the real size of 'param-list.txt'):

xargs -I '{}' echo '^https?://(([[:alnum:][:punct:]]+)+)?{}=' < ./params-list.txt \
  | grep -oiEf - ./filtered_fetched_list.txt \
  | sort -u

or simpler:

sed 's|^|^https?://(([[:alnum:][:punct:]]+)+)?|; s|$|=|' < ./params-list.txt \
  | grep -oiEf - ./filtered_fetched_list.txt \
  | sort -u


I'll leave it as homework to the reader to find the combination
of both by e.g. passing <500 patterns to each grep process. ;-)

Bernhard Voelker <berny>
Group administrator
Mon 13 Dec 2021 11:27:05 PM UTC, comment #1: 

There's nothing xargs could do about it - it doesn't even know
that the launched processes write to stdout at the same time.

The man page puts this quite clearly:


Please  note  that it is up to the called processes to properly manage parallel access to shared resources.  For example, if more than one of them tries to print to stdout, the output will be produced in an indeterminate order (and very likely mixed
up) unless the processes collaborate in some way to prevent this.  Using some kind of locking scheme is one way to prevent such problems.  In general, using a locking scheme will help ensure correct output but reduce performance.  If you don't  want
to tolerate the performance difference, simply arrange for each process to produce a separate output file (or otherwise use separate resources).


It's simply a race: no one can guarantee which process will write next into
the same file descriptor, stdout.

As this is nothing xargs(1) can do about it, I'm hereby marking this as 'invalid'.

Bernhard Voelker <berny>
Group administrator
Mon 13 Dec 2021 05:04:57 AM UTC, original submission:  

Steps to reproduce:
1. Call the next line:
xargs -P10 -n1 -I {} grep -oiE "^https?://(([[:alnum:][:punct:]]+)+)?{}=" ./filtered_fetched_list.txt < ./params-list.txt | sort -u | head -n 10

Expected result: output entries contains http(s) start with string.
Actual behavior: output entries doesn’t contain http(s)
Workaround: using xargs with process count by default (without -P argument).

Kirill Zhdanov <storenth>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #52490:  xargs-p-issue.png added by storenth (423KiB - image/png)
file #52491:  params-list.txt added by storenth (19KiB - text/plain)
file #52492:  xargs-without-p.png added by storenth (483KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by berny (Posted a comment)
  • -email is unavailable- added by storenth (Submitted the item)
  • -email is unavailable- added by storenth
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-12-13 berny StatusNone Invalid
        Assigned toNone berny
        Open/ClosedOpen Closed
    2021-12-13 storenth Attached File- Added xargs-p-issue.png, #52490
        Attached File- Added params-list.txt, #52491
        Attached File- Added xargs-without-p.png, #52492
        Attached File- Added filtered_fetched_list.txt, #52493
        Carbon-Copy- Added storenth

    Back to the top

    Powered by Savane 3.12.
    Corresponding source code