bugGNU Wget - Bugs: bug #33833, -c scans binary files as if they...


bug #33833: -c scans binary files as if they were html, after receiving 416 response

Submitted by:  Noël Köthe <nok>
Submitted on:  Sat 23 Jul 2011 10:16:28 AM UTC  
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.12 Operating System:  GNU/Linux
Reproducibility:  None Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Sat 23 Jul 2011 10:16:28 AM UTC, original submission:

While using "wget -c -r" on a directory of large binary files, I
noticed long delays after the "The file is already fully retrieved;
nothing to do." message.

It turns out this is because the server returned a 416 response with
Content-Type: text/html, and so Wget decides to scan the file for
links, as if it were HTML. But the file is not HTML -- just the 416
response body was.


$ cd /tmp
$ wget -c -d -r http://www.gnu.org/graphics/t-desktop-4-small.jpg

(The file is downloaded as expected, and not scanned for URLs)

$ wget -c -d -r http://www.gnu.org/graphics/t-desktop-4-small.jpg

(This time, notice in the debug output how the file was "Loaded" and
scanned for "no-follow" links. This is the source of the delay on
large binary files).

---response begin---
HTTP/1.1 416 Requested Range Not Satisfiable
Date: Mon, 16 May 2011 21:34:24 GMT
Server: Apache
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=iso-8859-1

---response end---
416 Requested Range Not Satisfiable

The file is already fully retrieved; nothing to do.

Closed fd 3
Loaded http://www.gnu.org/graphics/t-desktop-4-small.jpg (size 30195).
no-follow in http://www.gnu.org/graphics/t-desktop-4-small.jpg: 0



Noël Köthe <nok>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by nok (Submitted the item)
  • -email is unavailable- added by nok (Debian BTS)



    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-07-23 nok Carbon-Copy- => Added -email is unavailable-

    Back to the top

    Powered by Savane 3.3