bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #50738, --pipe --tee with > 251...

 
 

bug #50738: --pipe --tee with > 251 arguments

Submitted by:  Ole Tange <tange>
Submitted on:  Wed 05 Apr 2017 11:53:28 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 17 Apr 2017 08:57:26 PM UTC, comment #3:

This works nicely:

seq 1000000 | parallel --pipe --tee wc ::: {1..7500}

Each job starts 3 processes: bash, cat, wc. In total 22500 jobs.

Process table is limited to 32768.

Adjusting 'ulimit -u' makes it possible to do:

It takes 15 GB RAM.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 15 Apr 2017 08:56:07 AM UTC, comment #2:

ulimit is the problem. With ulimit raised you can run 3500 jobs on 5 GB RAM.

The few users that need this will have to ask to have the limit raised.

- sleep 10101010;
+ # 'exec sleep' takes less RAM than sleeping in perl
+ exec 'sleep', 10101;

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 06 Apr 2017 05:09:44 AM UTC, comment #1:

seq 3 | ppar --pipe --tee --argsep ,, -I II -v -N30 -- ppar --tee --pipe wc'\;' echo {} ::: II ,, {1..100}

Ole Tange <tange>
Project AdministratorIn charge of this item.
Wed 05 Apr 2017 11:53:28 PM UTC, original submission:

Can we do something similar to:

seq 3 | ppar --pipe --tee --argsep ,, -vX --dry-run ppar --tee --pipe cat ::: ,, {1..1000}

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 17 Apr 2017 08:59:39 PM UTCtangeStatusNone=>Fixed
      Assigned toNone=>tange
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1