bugGNU Wget - Bugs: bug #48708, Wget downloads file but refuses to...

 
 

bug #48708: Wget downloads file but refuses to examine it for links to follow

Submitted by:  Dale Worley <worley>
Submitted on:  Fri 05 Aug 2016 03:15:13 PM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: worleyOpen/Closed: Open
Release: 1.16.1Operating System: GNU/Linux
Reproducibility: Every TimeFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 24 Oct 2016 07:34:36 PM UTC, comment #1:

See patch submitted to the mailing list 16 Oct 2016 for a solution to this and some related problems.

Dale Worley <worley>
Fri 05 Aug 2016 03:15:13 PM UTC, original submission:

To demonstrate (including useful debugging output):

$ wget -d -r --include-directories=/assignments,/protocols http://www.iana.org/protocols/index.html

Naively, I expect wget to download the index.html file and scan it for links to recurse on.

The complication seems to arise from the fact that /protocols/index.html is redirected to http://www.iana.org/protocols. That file is fetched, and stored as http://www.iana.org/protocols/index.html, that is, the file name is based on the original URL, not the redirected one.

However, wget does not examine the file for links to follow. Wget gives the following messages:

Deciding whether to enqueue "http://www.iana.org/protocols".
http://www.iana.org/protocols () is excluded/not-included.
Decided NOT to load it.
Redirection "http://www.iana.org/protocols" failed the test.

This is unexpected; I expect that the file is treated consistently in regard to (1) whether to download it, (2) what file name to store it in, and (3) whether to examine it for links, in that all three decisions would be made based on either the original URL or the ultimate redirected URL. (The decision to use the original URL seems to be the correct choice to me.) But wget's behavior is to make decision (3) based on the redirected name, not the original name.

In addition, (as I read the documentation) wget will read all URLs that are named on the command line, regardless of whether they meet the include/exclude criteria, and so I expect that with -r, all those URLs would be scanned for links. However it is clear that wget does not always scan provided URL for links.

Dale Worley <worley>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by worley (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    No Changes Have Been Made to This Item

    Back to the top


    Powered by Savane 3.1-cleanup1