bugGNU Wget - Bugs: bug #24663, Download-rate calculations are...

 
 

bug #24663: Download-rate calculations are error-prone

Submitter:  Micah Cowan <micahcowan>
Submitted:  Mon 27 Oct 2008 06:52:46 PM UTC
   
 
Category:  Program Logic Severity:  2 - Minor
Priority:  3 - Low Status:  Needs Discussion
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.11.3 Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.15 Regression:  None
Work Required:  0 - Hours Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 01 Aug 2009 09:37:19 PM UTC, comment #1: 

So, this appears to be a misunderstanding of what that value is intended to represent, combined with unavoidable shortcomings in how it is calculated.

That value does not represent how much data was downloaded (which is represented in the left-hand column), but rather the rate (per second) of download corresponding to just that single row. This is calculated by taking how many bytes are represented by this row, and dividing it by the actual time it took to fetch those bytes.

This is susceptible to anomalies. Consider the case where 20k is downloaded, and then it takes a second or two, but then another 80k is downloaded (each row consisting of 50k). Well, the first row will show a download rate of maybe 25k a second, but the next row's data will then already be available in the system buffers, so when Wget tries to fetch it, it will obtain it virtually instantaneously. So, the bytes downloaded for row two will be 50k, but the amount of time taken will be fairly close to zero, resulting in an abnormally high number. This will be especially true on the very last row, which is likely to have less than 50k; so if ~12k is fetched instantaneously because it was already available, the download rate leaps to something huge.

A discussion should take place on what more appropriate download-rate calculation methods might exist.

Micah Cowan <micahcowan>
Mon 27 Oct 2008 06:52:46 PM UTC, original submission:  

http://article.gmane.org/gmane.comp.web.wget.general/8384


Hello.

During download with wget I've redirected output into file with the
following command:

$ LC_ALL=C wget -o output 'ftp://mirror.yandex.ru/gentoo-distfiles/distfiles/OOo_3.0.0rc4_20080930_LinuxIntel_langpack_en-GB.tar.gz'

I've set LC_ALL and LANG explicitly to be sure that this is not locale
related problem. The output I saw in output file was:

--2008-10-25 14:51:17--  ftp://mirror.yandex.ru/gentoo-distfiles/distfiles/OOo_3.0.0rc4_20080930_LinuxIntel_langpack_en-GB.tar.gz
           => `OOo_3.0.0rc4_20080930_LinuxIntel_langpack_en-GB.tar.gz.13'
Resolving mirror.yandex.ru... 77.88.19.68
Connecting to mirror.yandex.ru|77.88.19.68|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /gentoo-distfiles/distfiles ... done.
==> SIZE OOo_3.0.0rc4_20080930_LinuxIntel_langpack_en-GB.tar.gz ... 13633213
==> PASV ... done.    ==> RETR OOo_3.0.0rc4_20080930_LinuxIntel_langpack_en-GB.tar.gz ... done.
Length: 13633213 (13M)

     0K .......... .......... .......... .......... ..........  0%  131K 1m41s
    50K .......... .......... .......... .......... ..........  0%  132K 1m40s
   100K .......... .......... .......... .......... ..........  1%  135K 99s
   150K .......... .......... .......... .......... ..........  1%  132K 99s
   200K .......... .......... .......... .......... ..........  1%  130K 99s
   250K .......... .......... .......... .......... ..........  2% 45.9K 2m9s
   300K .......... .......... .......... .......... ..........  2% 64.3M 1m50s
[snip]
 13250K .......... .......... .......... .......... .......... 99%  131K 0s
 13300K .......... ...                                        100%  134K=1m41s

2008-10-25 14:52:58 (132 KB/s) - `OOo_3.0.0rc4_20080930_LinuxIntel_langpack_en-GB.tar.gz.13'
saved [13633213]

Note the line above snip:
   300K ..........  2% 64.3M 1m50s

This is impossible to download so much Mbytes as file is much less. I
don't know why sometimes this number jumps, but in some cases it cause
the following output at the end of download:

 13300K .......... ...              100% 26101G=1m45s

Obviously I don't have possibility to download with such high
(26101G=1m45s) speed. This is reproducible with wget 1.11.4.

--
Peter.


Micah Cowan <micahcowan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by micahcowan (Submitted the item)
  •  

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-08-01 micahcowan StatusNeeds Investigation Needs Discussion
        Planned Release1.12 1.15
        SummaryIncorrect download speed calculations (with --progress=dot?) Download-rate calculations are error-prone
    2008-10-27 micahcowan Priority5 - Normal 3 - Low

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code