bugGNU Wget - Bugs: bug #20398, Save a list of the links that were...


bug #20398: Save a list of the links that were not followed

Submitted by:  Tony Lewis <tdlewis77>
Submitted on:  Sun 08 Jul 2007 04:46:06 AM UTC  
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  Needs Discussion
Privacy:  Public Assigned to:  Giuseppe Scrivano <gscrivano>
Originator Name:  Tony Lewis Open/Closed:  Closed
Release:  trunk Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  0 - Hours Patch Included:  Yes

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Mon 10 Aug 2015 11:46:11 AM UTC, comment #7:

Fixed upstream with:

commit e4db00d74d7c8ade43e57f39344d8505d607308a
Author: Jookia <166291@gmail.com>
Date:   Fri Jul 31 23:41:36 2015 +1000

    Add option to write URL rejections to a tab-delimited CSV log.


     * main.c: Add "--rejected-log" option.
     * init.c: Add "rejectedlog" command.
     * options.h: Add "rejected_log" parameter string.
     * wget.texi: Add brief documentation on new --rejected-log option.
     * recur.c: Optionally log details of URLs not traversed.
       Add reject_reason enum.
       (download_child_p -> download_child): Return a reject_reason.
       (descend_redirect_p -> descend_redirect): Return a reject_reason.
       (retrieve_tree): Support logging reasons for rejection.
       Add write_reject_log_header that writes a CSV format header to a file.
       Add write_reject_log_url that writes a url struct to a file in CSV format.
       Add write_reject_log_reason that writes the URL and parent URL as well as the
       rejection reason to a CSV file.
     * Test--rejected-log.px: Add a basic test for the --rejected-log command.
     * tests/Makefile.am: Run Test--rejected-log.px.


    This allows you to figure out why URLs are being rejected and some context
    around it. CSV is used as the output format since it can be used easily parsed,
    it's delimited by tabs instead of commas to allow using all (quoted) URL
    characters and includes column names which may be used for compatibility.

Giuseppe Scrivano <gscrivano>
Project AdministratorIn charge of this item.
Mon 20 Jul 2015 05:29:03 PM UTC, comment #6:

It's been a few months so I've updated the patch to work with the latest HEAD.

(file #34470)

Jookia <jookia>
Thu 07 May 2015 03:58:52 PM UTC, comment #5:

I've found myself in need of this feature. I'm trying to download a website recursively without pulling in every single ad and its HTML. I'd like to be able to find out which URLs were rejected, why, and information about the domains (host, port, etc.)

I've patched my copy of Wget to dump all of this in to a CSV file which I can then tool through to get my desired results:

% grep "DOMAIN" rejected.csv | head -1
% grep "DOMAIN" rejected.csv | cut -d"," -f4 | sort | uniq

I've included a patch made in a few hours that does this.

(file #33955)

Jookia <jookia>
Sat 22 Mar 2008 09:14:21 PM UTC, comment #4:

This functionality would actually go better in the Session Info DB (MetaDataBase).

Micah Cowan <micahcowan>
Thu 26 Jul 2007 07:43:35 PM UTC, comment #3:

Link to patches for this against 1.10 codebase: http://www.exelana.com/unfollowed-links.tar.gz

Micah Cowan <micahcowan>
Fri 20 Jul 2007 05:19:41 AM UTC, comment #2:

Targeting to 1.12, contingent upon patch against recent code.

Micah Cowan <micahcowan>
Sun 08 Jul 2007 04:48:53 AM UTC, comment #1:

Add here at Micah's request. I have the code working against the 1.10 branch and will migrate and upload a patch against the trunk.

Tony Lewis <tdlewis77>
Sun 08 Jul 2007 04:46:06 AM UTC, original submission:

It would be useful if wget would dump the list of links that it did not follow to a file. Possible uses:

- mirror a website and get a list of all links from that website to other websites

- download with options that restrict the files that are downloaded (such as --level, --no-parent, and --reject) and use the list of unfollowed links to see if there were interesting files that were skipped

Tony Lewis <tdlewis77>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #33955:  0001-rejected-log-Add-option-to-dump-URL-rejections-to-a-.patch added by jookia (14KiB - text/x-patch - Patch to add --rejected-log option.)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by jookia (Updated the item)
  • -email is unavailable- added by micahcowan (Posted a comment)
  • -email is unavailable- added by tdlewis77 (Submitted the item)



    Follow 17 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-08-10 gscrivano Assigned totdlewis77 => gscrivano
        Open/ClosedOpen => Closed
        Planned Release1.12.x => None
    2015-07-20 jookia Attached File- => Added 0001-rejected-log-Add-option-to-dump-URL-rejections-to-a-.patch, #34470
    2015-05-07 jookia Attached File- => Added 0001-rejected-log-Add-option-to-dump-URL-rejections-to-a-.patch, #33955
    2009-09-23 micahcowan Severity1 - Wish => 3 - Normal
        StatusDuplicate => Needs Discussion
        Open/ClosedClosed => Open
        Planned Release1.12 => 1.12.x
    2008-03-22 micahcowan StatusConfirmed => Duplicate
        Open/ClosedOpen => Closed
    2007-07-26 micahcowan Patch IncludedNone => Yes
    2007-07-20 micahcowan Planned ReleaseNone => 1.12
    2007-07-08 tdlewis77 Severity3 - Normal => 1 - Wish
        StatusNone => Confirmed
        Assigned toNone => tdlewis77
        Work RequiredNone => 0 - Hours

    Back to the top

    Powered by Savane 3.4