bugGNU Wget - Bugs: bug #28713, Wget shouldn't obey robots.txt for...


bug #28713: Wget shouldn't obey robots.txt for -p

Submitted by:  Micah Cowan <micahcowan>
Submitted on:  Fri 22 Jan 2010 08:11:16 PM UTC  
Category: Program LogicSeverity: 2 - Minor
Priority: 5 - NormalStatus: Confirmed
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: NoneOperating System: None
Reproducibility: NoneFixed Release: None
Planned Release: 1.14Regression: No
Work Required: 0 - HoursPatch Included: None

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Mon 07 Jun 2010 07:17:32 PM UTC, comment #2:

Right; that was my intention by this issue report.

might be enough, though you might want to consider the combination of both -r and -p, such that robots get ignored whenever the link qualifies as a prerequisite, but are honored when the link was via an <a> tag or the like.

Micah Cowan <micahcowan>
Mon 07 Jun 2010 09:49:20 AM UTC, comment #1:

should not wget obey robots.txt only when -r is specified?

"wget obeys robots.txt" => "-r"

Giuseppe Scrivano <gscrivano>
Project Administrator
Fri 22 Jan 2010 08:11:16 PM UTC, original submission:

Wget currently downloads and obeys robots.txt when -p is given, even though that behavior mimics normal web-page browsing, more than robotting. It should probably not do that.

Note that you can't just do:

because that won't handle the case when both -p and -r (or -m) have been given.

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by micahcowan (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    No Changes Have Been Made to This Item

    Back to the top

    Powered by Savane 3.1-cleanup1