bugGNU Wget - Bugs: bug #37986, Redirection to the same url...

 
 

bug #37986: Redirection to the same url prevents recursion

Submitted by:  Enne Walker <enne>
Submitted on:  Fri 28 Dec 2012 07:26:46 AM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: 1.13.4Operating System: GNU/Linux
Reproducibility: Every TimeFixed Release: None
Planned Release: NoneRegression: Yes
Work Required: NonePatch Included: Yes

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 28 Dec 2012 07:26:46 AM UTC, original submission:

If a redirect ends up ultimately redirecting to the same url, recursion aborts because that url is already in the blacklist. This causes --page-requisites to fail to download any resources from that page (and likely would cause general --recursion to fail as well).

For example, most Dreamwidth pages do this sort of redirection if you're logged in, where they bounce to one page, and then back to the original. I was using the following command (with a logged in cookie jar in cookie.txt):

wget --no-clobber -e robots=off --force-directories --no-check-certificate --ignore-tags=a --debug --wait 2 --adjust-extension --page-requisites --span-hosts --backup-converted --convert-links --load-cookies cookie.txt http://somehost.dreamwidth.org/profile/

...which yields a log that looks something like this:

--snip--
Enqueuing http://somehost.dreamwidth.org/profile/ at depth 0
Dequeuing http://somehost.dreamwidth.org/profile/ at depth 0

[...redirects, etc...]

Deciding whether to enqueue "http://somehost.dreamwidth.org/profile/".
Already on the black list.
Decided NOT to load it.
Redirection "http://somehost.dreamwidth.org/profile/" failed the test.
FINISHED --2012-12-27 21:26:46--
--snip--

This redirection to the same blacklisted url causes descend_redirect_p to return false and print the "failed the test" line above. It seems like the best way to handle this would be to not consider it a redirect.

Patch attached that fixes the problem locally for me in 1.13.4. This used to work in past versions of wget (most likely 1.12.1 shipped with ubuntu 10.04), but I haven't narrowed down which version had the regression.

Enne Walker <enne>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by enne (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 28 Dec 2012 07:26:46 AM UTCenneAttached File-=>Added wget-redirected_to_same_host.patch, #27156

    Back to the top


    Powered by Savane 3.1-cleanup