bugGNU Wget - Bugs: bug #59293, wget downloads nofollow links


bug #59293: wget downloads nofollow links

Submitter:  -- <nhagea>
Submitted:  Sun 18 Oct 2020 01:22:25 PM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.20 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Sun 18 Oct 2020 01:22:25 PM UTC, original submission:  

I want to crawl/scrape a wordpress website with wget.
Problem: wget will download documents/links despite them having a `rel=nofollow` attribute.
It is my understanding, that, when robots.txt is honored, also nofollow links are honored, but this is not the case.

Quote wget.info:

> Specify whether the norobots convention is respected by Wget, “on” by default. This switch controls both the ‘/robots.txt’ and the ‘nofollow’ aspect of the spec.

Overall it would be good to have those two separated - a user might want to ignore robots.txt but do not download nofollow
In the case of wordpress they are for example used to reply to and like comments - so every comment will lead to the same page being downloaded 3x.


> wget --mirror --page-requisites --adjust-extension --convert-links --restrict-file-names=windows --no-parent --span-hosts --domains=randomascii.wordpress.com,wp.com  https://randomascii.wordpress.com/about/

Now open the about folder and after some seconds you will see dozens of html files that stem from nofollow links: `index.html@share=reddit.html`, `index.html@share=twitter.html`, `index.html@replytocom=74214.html` ...

Is `nofollow` only checked when in a <meta> tag and not for <a>? That would be quite bad.

Tested version: `GNU Wget 1.20.1 built on linux-gnu.` (Debian).

-- <nhagea>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by nhagea (Submitted the item)

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-3cd0.
    Corresponding source code