bugGNU Parallel - Bugs: bug #63090, --compress --keep-order together...

 
 

bug #63090: --compress --keep-order together can cause thousands of processes to spawn

Submitter:  John <johncs>
Submitted:  Thu 22 Sep 2022 04:59:57 AM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 23 Apr 2023 08:19:00 AM UTC, comment #2: 

Now part of documentation.

Ole Tange <tange>
Group administrator
Wed 28 Sep 2022 11:55:06 PM UTC, comment #1: 

Yep. That is by design.

It should probably be mentioned under `--compress` and explained in `man parallel_design`.

Ole Tange <tange>
Group administrator
Thu 22 Sep 2022 04:59:57 AM UTC, original submission:  

The key ingredients seems to be --compress and --keep-order together, plus a task that takes awhile. Here is a simple reproduction:


# WARNING: This will cause thousands of processes to spawn
yes 0 | head -n 400 | xargs parallel --keep-order --compress sleep ::: 5


Try running `watch "ps | wc -l"` in another terminal window to see the number of processes explode.

Here's some version information about my environment:


$ parallel --version
GNU parallel 20220822
$ sw_vers
ProductName:        macOS
ProductVersion:        12.5.1
BuildVersion:        21G83


John <johncs>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Posted a comment)
  • -email is unavailable- added by johncs (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-04-23 tange StatusIn Progress Fixed
        Open/ClosedOpen Closed
    2022-09-28 tange StatusNone In Progress
        Assigned toNone tange

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code