bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #45841, Replacement string for total no of...

 
 

bug #45841: Replacement string for total no of jobs

Submitted by:  Ole Tange <tange>
Submitted on:  Sat 29 Aug 2015 01:56:42 AM UTC  
 
Category: NoneSeverity: 1 - Wish
Item Group: NoneStatus: Fixed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 18 Oct 2015 02:44:13 PM UTC, comment #4:

Fixed in [b244ddf7c]:

--plus activates {##}

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 17 Oct 2015 03:36:09 PM UTC, comment #3:

Partly supported:

ppar 'echo {= ($_=1+$Global::JobQueue->total_jobs());=}' ::: {1..30}

But it breaks down if not called first:

ppar 'echo {= $::G++ > 3 and ($_=1+$Global::JobQueue->total_jobs());=}' ::: {1..30}

Can the number of already read jobs be added to total_jobs?

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sun 13 Sep 2015 07:46:28 PM UTC, comment #2:

{
# Computer the number of records
# Problem: -N can take more than one record
# Solution: Divide by -N,-L,-l,-n
# Problem: {= =} may not call no_of_jobs(): {= $_ = rand() > 0.5 ? no_of_jobs() : 0 =}
# Solution: Record the number of jobs allready generated
my $multifilequeue = MultifileQueue->new(\@input_source_fh);
my @arg_records;
my $record;
# Get all records of args
while(defined($record = $multifilequeue->get())) {
push @arg_records, $record;
}
# Unget all records of args to compute all command lines
$multifilequeue->unget(@arg_records);
print $#arg_records;
}

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 29 Aug 2015 01:59:04 AM UTC, comment #1:

The number of jobs only change when using -X/--xargs.

Maybe have a way of generating all jobs with a dummy value of no-of-jobs. When the real no-of-jobs is known, discard all jobs, and generate them again.

It will cost the time to regenerate all jobs (expensive), but might work.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Sat 29 Aug 2015 01:56:42 AM UTC, original submission:

There is currently no way to do this.

The reason is that to compute the number of jobs GNU Parallel has to build the commands to be executed. And if you are using -X, then the number of jobs depended on the length of the replacement strings, so GNU Parallel does not know the number of jobs when it starts building the first command. Only when all commands are built, does GNU Parallel know how many there are.

To understand why this a problem look at this paradox. Which of these should be correct:

$ seq 1000 1010 | parallel -Xj1 -s 17 echo {###} {}
6 1000 1001
6 1002 1003
6 1004 1005
6 1006 1007
6 1008 1009
6 1010
Or:

$ seq 1000 1010 | parallel -Xj1 -s 17 echo {###} {}
11 1000
11 1001
11 1002
11 1003
11 1004
11 1005
11 1006
11 1007
11 1008
11 1009
11 1010
You can argue that both of them are correct.

Ole Tange <tange>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 18 Oct 2015 02:44:13 PM UTCtangeStatusNone=>Fixed
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1