bugGNU Wget - Bugs: bug #33044, wget -r cannot avoid following...

 
 

bug #33044: wget -r cannot avoid following links in a particular file

Submitted by:  David Skalinder <dskalinder>
Submitted on:  Mon 11 Apr 2011 02:25:17 AM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: 1.12Operating System: GNU/Linux
Reproducibility: Every TimeFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: No

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 11 Apr 2011 10:43:35 PM UTC, comment #1:

Sorry, just realized that the robots line uses the filenames from the demo at http://davidskalinder.com/wgettest/. To use the example filenames from the first paragraph, the second robots line would read:

David Skalinder <dskalinder>
Mon 11 Apr 2011 02:25:17 AM UTC, original submission:

wget has no option to not follow the links in a particular file while doing a recursive download. For example, if the pages /recentlogs.html and /oldlogs.html contain links to many directories and to each other, a user cannot download all the links from /recentlogs without downloading all the links from /oldlogs.

This is a basic operation for a recursive downloader, and its omission is undocumented and counterintuitive.

It would be reasonable to expect that -R would do this, but instead -R downloads rejected files, queues their links, and then deletes the rejected files only.

-X does provide this functionality for entire directories, but not for individual files.

On the server side, the desired behavior can be achieved by adding the following to robots.txt:
But obviously many wget users will not have access to the server side.

A demo of this problem is up at http://davidskalinder.com/wgettest/.

David Skalinder <dskalinder>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by dskalinder (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    No Changes Have Been Made to This Item

    Back to the top


    Powered by Savane 3.1-cleanup