bugGNU Parallel - build and execute command lines from standard input in parallel - Bugs: bug #32196, --eta is not useful or accurate

 
 

bug #32196: --eta is not useful or accurate

Submitted by:  None
Submitted on:  Thu 20 Jan 2011 06:45:55 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: NoneStatus: Confirmed
Privacy: PublicAssigned to: Ole Tange <tange>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Wed 27 Apr 2011 02:24:41 PM UTC, comment #3:

The accurateness of --eta is now improved significantly.

Re-open the bug if you can give examples of --eta working badly.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Fri 21 Jan 2011 02:35:39 PM UTC, comment #2:

GNU parallel 20101222.

Anonymous
Thu 20 Jan 2011 11:08:02 PM UTC, comment #1:

You have not included which version of GNU Parallel you use. Please do that to make it easier to reproduce the error.

I cannot reproduce the error you see.

ETA works crap. Unfortunately that is well known.

Your idea of adding a total column is good.

Ole Tange <tange>
Project AdministratorIn charge of this item.
Thu 20 Jan 2011 06:45:55 PM UTC, original submission:

The docs say that --eta "forces GNU parallel to read all jobs before starting to find the number of jobs." It does not appear to do this, or if it does, the computed ETA is not accurate. In either case, the total number of jobs is never displayed, which would be useful.

In this example, on RHEL 5.5 x86_64 (perl 5.8.8), the ETA always says 0s, when parallel should easily be able to determine that jobs take 5 seconds:

-----------------------
$ time seq 0 5 | parallel -P2 --eta "echo {} > /dev/null; sleep 5"
Use of uninitialized value in scalar assignment at /usr/local/bin/parallel line 1635, <STDIN> line 1.
Use of uninitialized value in scalar assignment at /usr/local/bin/parallel line 1635, <STDIN> line 2.

Computers / CPU cores / Max jobs to run
1:local / 4 / 2

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s local:0/6/100%/2.7s

real 0m16.305s
user 0m0.135s
sys 0m2.010s
------------------------

Second, the average seconds to complete is not correct, which may be the root problem in computing the ETA. It currently appears to be wallclock time / total jobs, whereas it should be the sum of the wallclock times for each job / total jobs. I.e., each job should have a separate wallclock timer, and these should be summed. E.g., if 3 jobs take, in parallel, 1, 2 and 3 seconds, the average seconds should be (1 + 2 + 3) / 3 = 2, and not 3 / 2 = 1.5. "Seconds for a (serial) job" is what users are familiar with seeing and using as a rule of thumb and is more intuitive to reason with.

Putting these together, if GNU parallel knows the total number of jobs and the average time, the most useful progress display would be

completed jobs / total jobs / % complete / Avg. time

consistent with the familiar division-style usage "I've done 3/5 jobs."

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by tange (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 27 Apr 2011 02:24:41 PM UTCtangeOpen/ClosedOpen=>Closed
    Thu 20 Jan 2011 11:08:02 PM UTCtangeStatusNone=>Confirmed
      Assigned toNone=>tange

    Back to the top


    Powered by Savane 3.1-cleanup1