bugGNU Wget - Bugs: bug #28713, Wget shouldn't obey robots.txt for...

 
 

bug #28713: Wget shouldn't obey robots.txt for -p

Submitter:  Micah Cowan <micahcowan>
Submitted:  Fri 22 Jan 2010 08:11:16 PM UTC
   
 
Category:  Program Logic Severity:  2 - Minor
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  No
Work Required:  0 - Hours Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 07 Jun 2010 07:17:32 PM UTC, comment #2: 

Right; that was my intention by this issue report.


  if (opt.use_robots && opt.recursive)


might be enough, though you might want to consider the combination of both -r and -p, such that robots get ignored whenever the link qualifies as a prerequisite, but are honored when the link was via an <a> tag or the like.

Micah Cowan <micahcowan>
Mon 07 Jun 2010 09:49:20 AM UTC, comment #1: 

should not wget obey robots.txt only when -r is specified?

"wget obeys robots.txt" => "-r"

Giuseppe Scrivano <gscrivano>
Group administrator
Fri 22 Jan 2010 08:11:16 PM UTC, original submission:  

Wget currently downloads and obeys robots.txt when -p is given, even though that behavior mimics normal web-page browsing, more than robotting. It should probably not do that.

Note that you can't just do:


  if (opt.use_robots && ! opt.page_requisites)


because that won't handle the case when both -p and -r (or -m) have been given.

Micah Cowan <micahcowan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by micahcowan (Submitted the item)
  •  

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code