bugGNU Wget - Bugs: bug #48232, Sometimes wget restarts download...

 
 

bug #48232: Sometimes wget restarts download from the beginning, even if the server supports resumed downloads

Submitter:  Evgeny Kapun <abacabadabacaba>
Submitted:  Wed 15 Jun 2016 03:06:05 PM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  trunk Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  trunk
Planned Release:  1.21 Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 25 Jan 2020 05:38:46 PM UTC, comment #3: 

Sorry - for some reason this was off our radar. Applied now to master and will be included in next release.

Tim Ruehsen <rockdaboot>
Group administrator
Fri 24 Jan 2020 01:57:26 PM UTC, comment #2: 

Why this bug still not fixed despite there is a patch ready? I'm on a unsteady connection and this annoys me a lot:

88300K .......... .......... .......... .......... .......... 98% 38.7K 14s
 88350K .......... .......... .....                            98% 7.03K=11m27s

2020-01-24 18:47:00 (129 KB/s) - Read error at byte 90496711/92306352 (Connection timed out). Retrying.

--2020-01-24 18:47:00--  (try: 5)  http://distfiles.gentoo.org/distfiles/fb/rustc-1.40.0-src.tar.xz
Connecting to distfiles.gentoo.org|64.50.233.100|:80...connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Retrying.

--2020-01-24 18:47:58--  (try: 6)  http://distfiles.gentoo.org/distfiles/fb/rustc-1.40.0-src.tar.xz
Connecting to distfiles.gentoo.org|64.50.233.100|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 92306352 (88M) [application/x-xz]
Saving to: ‘/media/Hdd/usr/portage/distfiles/rustc-1.40.0-src.tar.xz.__download__’

     0K .......... .......... .......... .......... ..........  0% 66.9K 22m26s
    50K .......... .......... .......... .......... ..........  0%  135K 16m47s

FUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU

Anonymous
Thu 16 Jun 2016 02:38:59 PM UTC, comment #1: 

See the attached patch. It fixes the issue on my side.

I'm specially afraid of having introduced an off-by-one error, so please remind me to write a test case for this next week. Just no time now.

(file #37503)

Ander Juaristi <ajuaristi>
Wed 15 Jun 2016 03:06:05 PM UTC, original submission:  

If the connection is interrupted during download, wget normally tries to continue the download from the same place where it left off. This only works if the server supports resumed downloads, otherwise, download restarts from the beginning. However, sometimes wget would restart the download even when the server does support resumption. Testing shows that this happens if a network error occurs before wget receives HTTP response from the server, a situation which is quite common on poor networks.

For testing, I created a web server which would behave as follows:

  • On the first request, it will send a response with `Content-Length: 1000` and 500 bytes of data, then wait.
  • On all other requests, it will just wait without sending any data.


Using wget to download from such server produces this:

$ wget --debug --timeout 1 --tries 4 'http://[::1]:8888/test'
Setting --timeout (timeout) to 1
Setting --tries (tries) to 4
DEBUG output created by Wget 1.18 on linux-gnu.

Reading HSTS entries from $HOME/.wget-hsts
URI encoding = 'ANSI_X3.4-1968'
converted 'http://[::1]:8888/test' (ANSI_X3.4-1968) -> 'http://[::1]:8888/test' (UTF-8)
Converted file name 'test' (UTF-8) -> 'test' (ANSI_X3.4-1968)
--2016-06-15 17:42:36--  http://[::1]:8888/test
Connecting to [::1]:8888... connected.
Created socket 4.
Releasing 0x000055593fd586f0 (new refcount 0).
Deleting unused 0x000055593fd586f0.

---request begin---
GET /test HTTP/1.1
User-Agent: Wget/1.18 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: [::1]:8888
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.0 200 OK
Content-Length: 1000

---response end---
200 OK
Registered socket 4 for persistent reuse.
Length: 1000
Saving to: 'test'

test                 50%[=========>          ]     500  --.-KB/s    in 1.0s

Disabling further reuse of socket 4.
Closed fd 4
2016-06-15 17:42:37 (500 B/s) - Read error at byte 500/1000 (Connection timed out). Retrying.

--2016-06-15 17:42:38--  (try: 2)  http://[::1]:8888/test
Connecting to [::1]:8888... connected.
Created socket 4.
Releasing 0x000055593fd586f0 (new refcount 0).
Deleting unused 0x000055593fd586f0.

---request begin---
GET /test HTTP/1.1
Range: bytes=500-
User-Agent: Wget/1.18 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: [::1]:8888
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Closed fd 4
Retrying.

--2016-06-15 17:42:41--  (try: 3)  http://[::1]:8888/test
Connecting to [::1]:8888... connected.
Created socket 4.
Releasing 0x000055593fd586f0 (new refcount 0).
Deleting unused 0x000055593fd586f0.

---request begin---
GET /test HTTP/1.1
User-Agent: Wget/1.18 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: [::1]:8888
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Closed fd 4
Retrying.

--2016-06-15 17:42:45--  (try: 4)  http://[::1]:8888/test
Connecting to [::1]:8888... connected.
Created socket 4.
Releasing 0x000055593fd586f0 (new refcount 0).
Deleting unused 0x000055593fd586f0.

---request begin---
GET /test HTTP/1.1
User-Agent: Wget/1.18 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: [::1]:8888
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Closed fd 4
Giving up.


As you may see, only the second request includes `Range` header. Starting from the third request, `Range` header is not included, so the download would not be resumed at this point. In practice, this means that a big download would suddenly restart because the network was down for some time, which is undesirable.

I attached a test program to reproduce the issue. It listens on [::1]:8888 and acts as a web server. You need to restart it before every test.

Related bugs:

  • #31653: a fix for that bug is probably what introduced this bug. Read the discussion there.
  • #48123: a bug similar to this one, but there it is not clear that the server supports resumed downloads.
Evgeny Kapun <abacabadabacaba>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #37488:  wget-test added by abacabadabacaba (597B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by ajuaristi (Updated the item)
  • -email is unavailable- added by abacabadabacaba (Submitted the item)
  •  

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-01-25 rockdaboot StatusNone Fixed
        Open/ClosedOpen Closed
        Release1.18 trunk
        Fixed ReleaseNone trunk
        Planned ReleaseNone 1.21
    2016-06-16 ajuaristi Attached File- Added 0001-Do-not-overwrite-restval-if-len-is-smaller.patch, #37503
    2016-06-15 abacabadabacaba Attached File- Added wget-test, #37488

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code