bugGNU Parallel - Bugs: bug #45841, Replacement string for total no of...

 
 

bug #45841: Replacement string for total no of jobs

Submitter:  Ole Tange <tange>
Submitted:  Sat 29 Aug 2015 01:56:42 AM UTC
   
 
Category:  None Severity:  1 - Wish
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  tange
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 18 Oct 2015 02:44:13 PM UTC, comment #4: 

Fixed in [b244ddf7c]:

--plus activates {##}

Ole Tange <tange>
Group administrator
Sat 17 Oct 2015 03:36:09 PM UTC, comment #3: 

Partly supported:

ppar 'echo {= ($_=1+$Global::JobQueue->total_jobs());=}' ::: {1..30}

But it breaks down if not called first:

ppar 'echo {= $::G++ > 3 and ($_=1+$Global::JobQueue->total_jobs());=}' ::: {1..30}

Can the number of already read jobs be added to total_jobs?

Ole Tange <tange>
Group administrator
Sun 13 Sep 2015 07:46:28 PM UTC, comment #2: 

                                      
{
    # Computer the number of records                                                          
    # Problem: -N can take more than one record                                               
    #   Solution: Divide by -N,-L,-l,-n                                                       
    # Problem: {= =} may not call no_of_jobs(): {=  $_ = rand() > 0.5 ? no_of_jobs() : 0 =}   
    #   Solution: Record the number of jobs allready generated                                
    my $multifilequeue = MultifileQueue->new(\@input_source_fh);
    my @arg_records;
    my $record;
    # Get all records of args                                                                 
    while(defined($record = $multifilequeue->get())) {
        push @arg_records, $record;
    }
    # Unget all records of args to compute all command lines                                  
    $multifilequeue->unget(@arg_records);
    print $#arg_records;
}

Ole Tange <tange>
Group administrator
Sat 29 Aug 2015 01:59:04 AM UTC, comment #1: 

The number of jobs only change when using -X/--xargs.

Maybe have a way of generating all jobs with a dummy value of no-of-jobs. When the real no-of-jobs is known, discard all jobs, and generate them again.

It will cost the time to regenerate all jobs (expensive), but might work.

Ole Tange <tange>
Group administrator
Sat 29 Aug 2015 01:56:42 AM UTC, original submission:  

There is currently no way to do this.

The reason is that to compute the number of jobs GNU Parallel has to build the commands to be executed. And if you are using -X, then the number of jobs depended on the length of the replacement strings, so GNU Parallel does not know the number of jobs when it starts building the first command. Only when all commands are built, does GNU Parallel know how many there are.

To understand why this a problem look at this paradox. Which of these should be correct:

$ seq 1000 1010 | parallel -Xj1 -s 17 echo {###} {}
6 1000 1001
6 1002 1003
6 1004 1005
6 1006 1007
6 1008 1009
6 1010
Or:

$ seq 1000 1010 | parallel -Xj1 -s 17 echo {###} {}
11 1000
11 1001
11 1002
11 1003
11 1004
11 1005
11 1006
11 1007
11 1008
11 1009
11 1010
You can argue that both of them are correct.

Ole Tange <tange>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tange (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-10-18 tange StatusNone Fixed
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code