bugGNU Wget - Bugs: bug #20398, Save a list of the links that were...


bug #20398: Save a list of the links that were not followed

Submitted by:  Tony Lewis <tdlewis77>
Submitted on:  Sun 08 Jul 2007 04:46:06 AM UTC  
Category: Feature RequestSeverity: 3 - Normal
Priority: 5 - NormalStatus: Needs Discussion
Privacy: PublicAssigned to: Giuseppe Scrivano <gscrivano>
Originator Name: Tony LewisOpen/Closed: Closed
Release: trunkOperating System: None
Reproducibility: NoneFixed Release: None
Planned Release: NoneRegression: None
Work Required: 0 - HoursPatch Included: Yes

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


(Jump to the original submission Jump to the original submission)

Mon 10 Aug 2015 11:46:11 AM UTC, comment #7:

Fixed upstream with:

commit e4db00d74d7c8ade43e57f39344d8505d607308a
Author: Jookia <166291@gmail.com>
Date: Fri Jul 31 23:41:36 2015 +1000

Add option to write URL rejections to a tab-delimited CSV log.

* main.c: Add "--rejected-log" option.
* init.c: Add "rejectedlog" command.
* options.h: Add "rejected_log" parameter string.
* wget.texi: Add brief documentation on new --rejected-log option.
* recur.c: Optionally log details of URLs not traversed.
Add reject_reason enum.
(download_child_p -> download_child): Return a reject_reason.
(descend_redirect_p -> descend_redirect): Return a reject_reason.
(retrieve_tree): Support logging reasons for rejection.
Add write_reject_log_header that writes a CSV format header to a file.
Add write_reject_log_url that writes a url struct to a file in CSV format.
Add write_reject_log_reason that writes the URL and parent URL as well as the
rejection reason to a CSV file.
* Test--rejected-log.px: Add a basic test for the --rejected-log command.
* tests/Makefile.am: Run Test--rejected-log.px.

This allows you to figure out why URLs are being rejected and some context
around it. CSV is used as the output format since it can be used easily parsed,
it's delimited by tabs instead of commas to allow using all (quoted) URL
characters and includes column names which may be used for compatibility.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Mon 20 Jul 2015 05:29:03 PM UTC, comment #6:

It's been a few months so I've updated the patch to work with the latest HEAD.

(file #34470)

Jookia <jookia>
Thu 07 May 2015 03:58:52 PM UTC, comment #5:

I've found myself in need of this feature. I'm trying to download a website recursively without pulling in every single ad and its HTML. I'd like to be able to find out which URLs were rejected, why, and information about the domains (host, port, etc.)

I've patched my copy of Wget to dump all of this in to a CSV file which I can then tool through to get my desired results:

I've included a patch made in a few hours that does this.

(file #33955)

Jookia <jookia>
Sat 22 Mar 2008 09:14:21 PM UTC, comment #4:

This functionality would actually go better in the Session Info DB (MetaDataBase).

Micah Cowan <micahcowan>
Thu 26 Jul 2007 07:43:35 PM UTC, comment #3:

Link to patches for this against 1.10 codebase: http://www.exelana.com/unfollowed-links.tar.gz

Micah Cowan <micahcowan>
Fri 20 Jul 2007 05:19:41 AM UTC, comment #2:

Targeting to 1.12, contingent upon patch against recent code.

Micah Cowan <micahcowan>
Sun 08 Jul 2007 04:48:53 AM UTC, comment #1:

Add here at Micah's request. I have the code working against the 1.10 branch and will migrate and upload a patch against the trunk.

Tony Lewis <tdlewis77>
Sun 08 Jul 2007 04:46:06 AM UTC, original submission:

It would be useful if wget would dump the list of links that it did not follow to a file. Possible uses:

- mirror a website and get a list of all links from that website to other websites

- download with options that restrict the files that are downloaded (such as --level, --no-parent, and --reject) and use the list of unfollowed links to see if there were interesting files that were skipped

Tony Lewis <tdlewis77>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

Attached Files
file #33955:  0001-rejected-log-Add-option-to-dump-URL-rejections-to-a-.patch added by jookia (14KiB - text/x-patch - Patch to add --rejected-log option.)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by jookia (Updated the item)
  • -unavailable- added by micahcowan (Posted a comment)
  • -unavailable- added by tdlewis77 (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 17 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 10 Aug 2015 11:46:11 AM UTCgscrivanoAssigned totdlewis77=>gscrivano
      Planned Release1.12.x=>None
    Mon 20 Jul 2015 05:29:03 PM UTCjookiaAttached File-=>Added 0001-rejected-log-Add-option-to-dump-URL-rejections-to-a-.patch, #34470
    Thu 07 May 2015 03:58:52 PM UTCjookiaAttached File-=>Added 0001-rejected-log-Add-option-to-dump-URL-rejections-to-a-.patch, #33955
    Wed 23 Sep 2009 05:30:29 PM UTCmicahcowanSeverity1 - Wish=>3 - Normal
      StatusDuplicate=>Needs Discussion
      Planned Release1.12=>1.12.x
    Sat 22 Mar 2008 09:14:21 PM UTCmicahcowanStatusConfirmed=>Duplicate
    Thu 26 Jul 2007 07:43:35 PM UTCmicahcowanPatch IncludedNone=>Yes
    Fri 20 Jul 2007 05:19:41 AM UTCmicahcowanPlanned ReleaseNone=>1.12
    Sun 08 Jul 2007 04:48:53 AM UTCtdlewis77Severity3 - Normal=>1 - Wish
      Assigned toNone=>tdlewis77
      Work RequiredNone=>0 - Hours

    Back to the top

    Powered by Savane 3.1-cleanup1