bugGNU Wget - Bugs: bug #20398, Save a list of the links that were...


bug #20398: Save a list of the links that were not followed

Submitted by:  Tony Lewis <tdlewis77>
Submitted on:  Sun Jul 8 04:46:06 2007  
Category: Feature RequestSeverity: 3 - Normal
Priority: 5 - NormalStatus: Needs Discussion
Privacy: PublicAssigned to: Giuseppe Scrivano <gscrivano>
Originator Name: Tony LewisOpen/Closed: Closed
Release: trunkOperating System: None
Reproducibility: NoneFixed Release: None
Planned Release: NoneRegression: None
Work Required: 0 - HoursPatch Included: Yes

Add a New Comment (Rich MarkupRich Markup):

You are not logged in

Please log in, so followups can be emailed to you.


(Jump to the original submission Jump to the original submission)

Mon Aug 10 11:46:11 2015, comment #7:

Fixed upstream with:

commit e4db00d74d7c8ade43e57f39344d8505d607308a
Author: Jookia <166291@gmail.com>
Date: Fri Jul 31 23:41:36 2015 +1000

Add option to write URL rejections to a tab-delimited CSV log.

* main.c: Add "--rejected-log" option.
* init.c: Add "rejectedlog" command.
* options.h: Add "rejected_log" parameter string.
* wget.texi: Add brief documentation on new --rejected-log option.
* recur.c: Optionally log details of URLs not traversed.
Add reject_reason enum.
(download_child_p -> download_child): Return a reject_reason.
(descend_redirect_p -> descend_redirect): Return a reject_reason.
(retrieve_tree): Support logging reasons for rejection.
Add write_reject_log_header that writes a CSV format header to a file.
Add write_reject_log_url that writes a url struct to a file in CSV format.
Add write_reject_log_reason that writes the URL and parent URL as well as the
rejection reason to a CSV file.
* Test--rejected-log.px: Add a basic test for the --rejected-log command.
* tests/Makefile.am: Run Test--rejected-log.px.

This allows you to figure out why URLs are being rejected and some context
around it. CSV is used as the output format since it can be used easily parsed,
it's delimited by tabs instead of commas to allow using all (quoted) URL
characters and includes column names which may be used for compatibility.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Mon Jul 20 17:29:03 2015, comment #6:

It's been a few months so I've updated the patch to work with the latest HEAD.

(file #34470)

Jookia <jookia>
Thu May 7 15:58:52 2015, comment #5:

I've found myself in need of this feature. I'm trying to download a website recursively without pulling in every single ad and its HTML. I'd like to be able to find out which URLs were rejected, why, and information about the domains (host, port, etc.)

I've patched my copy of Wget to dump all of this in to a CSV file which I can then tool through to get my desired results:

I've included a patch made in a few hours that does this.

(file #33955)

Jookia <jookia>
Sat Mar 22 21:14:21 2008, comment #4:

This functionality would actually go better in the Session Info DB (MetaDataBase).

Micah Cowan <micahcowan>
Project Administrator
Thu Jul 26 19:43:35 2007, comment #3:

Link to patches for this against 1.10 codebase: http://www.exelana.com/unfollowed-links.tar.gz

Micah Cowan <micahcowan>
Project Administrator
Fri Jul 20 05:19:41 2007, comment #2:

Targeting to 1.12, contingent upon patch against recent code.

Micah Cowan <micahcowan>
Project Administrator
Sun Jul 8 04:48:53 2007, comment #1:

Add here at Micah's request. I have the code working against the 1.10 branch and will migrate and upload a patch against the trunk.

Tony Lewis <tdlewis77>
Sun Jul 8 04:46:06 2007, original submission:

It would be useful if wget would dump the list of links that it did not follow to a file. Possible uses:

- mirror a website and get a list of all links from that website to other websites

- download with options that restrict the files that are downloaded (such as --level, --no-parent, and --reject) and use the list of unfollowed links to see if there were interesting files that were skipped

Tony Lewis <tdlewis77>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

Attached Files
file #33955:  0001-rejected-log-Add-option-to-dump-URL-rejections-to-a-.patch added by jookia (14kB - text/x-patch - Patch to add --rejected-log option.)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by jookia (Updated the item)
  • -unavailable- added by micahcowan (Posted a comment)
  • -unavailable- added by tdlewis77 (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 17 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon Aug 10 11:46:11 2015gscrivanoAssigned totdlewis77=>gscrivano
      Planned Release1.12.x=>None
    Mon Jul 20 17:29:03 2015jookiaAttached File-=>Added 0001-rejected-log-Add-option-to-dump-URL-rejections-to-a-.patch, #34470
    Thu May 7 15:58:52 2015jookiaAttached File-=>Added 0001-rejected-log-Add-option-to-dump-URL-rejections-to-a-.patch, #33955
    Wed Sep 23 17:30:29 2009micahcowanSeverity1 - Wish=>3 - Normal
      StatusDuplicate=>Needs Discussion
      Planned Release1.12=>1.12.x
    Sat Mar 22 21:14:21 2008micahcowanStatusConfirmed=>Duplicate
    Thu Jul 26 19:43:35 2007micahcowanPatch IncludedNone=>Yes
    Fri Jul 20 05:19:41 2007micahcowanPlanned ReleaseNone=>1.12
    Sun Jul 8 04:48:53 2007tdlewis77Severity3 - Normal=>1 - Wish
      Assigned toNone=>tdlewis77
      Work RequiredNone=>0 - Hours

    Back to the top

    Powered by Savane 3.1-cleanup1