bugGNU Wget - Bugs: bug #37986, Redirection to the same url...


bug #37986: Redirection to the same url prevents recursion

Submitted by:  Enne Walker <enne>
Submitted on:  Fri 28 Dec 2012 07:26:46 AM UTC  
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: 1.13.4Operating System: GNU/Linux
Reproducibility: Every TimeFixed Release: None
Planned Release: NoneRegression: Yes
Work Required: NonePatch Included: Yes

Add a New Comment (Rich MarkupRich Markup):

You are not logged in

Please log in, so followups can be emailed to you.


Sat 05 Sep 2015 08:36:42 AM UTC, comment #2:

Thanks for the info.

http://somehost.dreamwidth.org/ doesn't work for testing.
Same with http://frobozz.seltzer.edu/

If you don't want to post a real URL in public, either
- set up a test case
- send the URL to me privately
to help us reproducing and fixing the issue.

Tim Ruehsen <rockdaboot>
Project Administrator
Sat 05 Sep 2015 12:42:23 AM UTC, comment #1:

There may be a related problem: if the URL has the form

http://frobozz.seltzer.edu/archives/frobozz.seltzer.edu/....more stuff...

It will not recurse. I've done basic tests in both Windows and Linux.

Fri 28 Dec 2012 07:26:46 AM UTC, original submission:

If a redirect ends up ultimately redirecting to the same url, recursion aborts because that url is already in the blacklist. This causes --page-requisites to fail to download any resources from that page (and likely would cause general --recursion to fail as well).

For example, most Dreamwidth pages do this sort of redirection if you're logged in, where they bounce to one page, and then back to the original. I was using the following command (with a logged in cookie jar in cookie.txt):

wget --no-clobber -e robots=off --force-directories --no-check-certificate --ignore-tags=a --debug --wait 2 --adjust-extension --page-requisites --span-hosts --backup-converted --convert-links --load-cookies cookie.txt http://somehost.dreamwidth.org/profile/

...which yields a log that looks something like this:

Enqueuing http://somehost.dreamwidth.org/profile/ at depth 0
Dequeuing http://somehost.dreamwidth.org/profile/ at depth 0

[...redirects, etc...]

Deciding whether to enqueue "http://somehost.dreamwidth.org/profile/".
Already on the black list.
Decided NOT to load it.
Redirection "http://somehost.dreamwidth.org/profile/" failed the test.
FINISHED --2012-12-27 21:26:46--

This redirection to the same blacklisted url causes descend_redirect_p to return false and print the "failed the test" line above. It seems like the best way to handle this would be to not consider it a redirect.

Patch attached that fixes the problem locally for me in 1.13.4. This used to work in past versions of wget (most likely 1.12.1 shipped with ubuntu 10.04), but I haven't narrowed down which version had the regression.

Enne Walker <enne>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

Attached Files


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by rockdaboot (Posted a comment)
  • -unavailable- added by enne (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 28 Dec 2012 07:26:46 AM UTCenneAttached File-=>Added wget-redirected_to_same_host.patch, #27156

    Back to the top

    Powered by Savane 3.1-cleanup