bugGNU Wget - Bugs: bug #56660, wget -r or mirror with robots-off...

 
 

bug #56660: wget -r or mirror with robots-off should still download robots.txt file

Submitter:  None
Submitted:  Tue 23 Jul 2019 03:45:32 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.20
Operating System:  None Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 23 Jul 2019 03:45:32 PM UTC, original submission:  

GNU Wget 1.20.3 built on darwin18.6.0.

with robots=off, wget does not download the robots.txt file

wget -r -e robots=off https://www.robotstxt.org/
robots.txt is not downloaded even though it is present

Expected:
downloading the root of a site with recursion or --mirror should still save the robots.txt file, even if it is being ignored.

The robots.txt file still contains useful information for site mirroring and archival purposes, even if it isn't being respected .

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Updated the item)
  •  

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-07-24 rockdaboot StatusNone Confirmed
        ReproducibilityNone Every Time

    Back to the top

    Powered by Savane 3.12.
    Corresponding source code