bugGNU Wget - Bugs: bug #33833, -c scans binary files as if they...

 
 

bug #33833: -c scans binary files as if they were html, after receiving 416 response

Submitted by:  Noël Köthe <nok>
Submitted on:  Sat 23 Jul 2011 10:16:28 AM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: 1.12Operating System: GNU/Linux
Reproducibility: NoneFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 23 Jul 2011 10:16:28 AM UTC, original submission:

While using "wget -c -r" on a directory of large binary files, I
noticed long delays after the "The file is already fully retrieved;
nothing to do." message.

It turns out this is because the server returned a 416 response with
Content-Type: text/html, and so Wget decides to scan the file for
links, as if it were HTML. But the file is not HTML -- just the 416
response body was.

Example:

$ cd /tmp
$ wget -c -d -r http://www.gnu.org/graphics/t-desktop-4-small.jpg

(The file is downloaded as expected, and not scanned for URLs)

$ wget -c -d -r http://www.gnu.org/graphics/t-desktop-4-small.jpg

(This time, notice in the debug output how the file was "Loaded" and
scanned for "no-follow" links. This is the source of the delay on
large binary files).

---response begin---
HTTP/1.1 416 Requested Range Not Satisfiable
Date: Mon, 16 May 2011 21:34:24 GMT
Server: Apache
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=iso-8859-1

---response end---
416 Requested Range Not Satisfiable

The file is already fully retrieved; nothing to do.

Closed fd 3
Loaded http://www.gnu.org/graphics/t-desktop-4-small.jpg (size 30195).
no-follow in http://www.gnu.org/graphics/t-desktop-4-small.jpg: 0

-jim

http://bugs.debian.org/626992

Noël Köthe <nok>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by nok (Submitted the item)
  • -unavailable- added by nok (Debian BTS)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 23 Jul 2011 10:16:28 AM UTCnokCarbon-Copy-=>Added -unavailable-

    Back to the top


    Powered by Savane 3.1-cleanup