Fri 28 Dec 2012 07:26:46 AM UTC, original submission:
If a redirect ends up ultimately redirecting to the same url, recursion aborts because that url is already in the blacklist. This causes --page-requisites to fail to download any resources from that page (and likely would cause general --recursion to fail as well).
For example, most Dreamwidth pages do this sort of redirection if you're logged in, where they bounce to one page, and then back to the original. I was using the following command (with a logged in cookie jar in cookie.txt):
wget --no-clobber -e robots=off --force-directories --no-check-certificate --ignore-tags=a --debug --wait 2 --adjust-extension --page-requisites --span-hosts --backup-converted --convert-links --load-cookies cookie.txt http://somehost.dreamwidth.org/profile/
...which yields a log that looks something like this:
--snip--
Enqueuing http://somehost.dreamwidth.org/profile/ at depth 0
Dequeuing http://somehost.dreamwidth.org/profile/ at depth 0
[...redirects, etc...]
Deciding whether to enqueue "http://somehost.dreamwidth.org/profile/".
Already on the black list.
Decided NOT to load it.
Redirection "http://somehost.dreamwidth.org/profile/" failed the test.
FINISHED --2012-12-27 21:26:46--
--snip--
This redirection to the same blacklisted url causes descend_redirect_p to return false and print the "failed the test" line above. It seems like the best way to handle this would be to not consider it a redirect.
Patch attached that fixes the problem locally for me in 1.13.4. This used to work in past versions of wget (most likely 1.12.1 shipped with ubuntu 10.04), but I haven't narrowed down which version had the regression.
|