bugGNU Wget - Bugs: bug #20331, robots.txt, robots.txt.html -E -k


bug #20331: robots.txt, robots.txt.html -E -k

Submitter:  Micah Cowan <micahcowan>
Submitted:  Tue 03 Jul 2007 08:19:34 AM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  4 Status:  None
Privacy:  Public Assigned to:  micahcowan
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  1 - Days Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Mon 27 Aug 2007 05:16:25 PM UTC, comment #1: 

Related to bug 20749?

Micah Cowan <micahcowan>
Tue 03 Jul 2007 08:19:34 AM UTC, original submission:  

When I was doing a recursive retrieval with the -E option, it was downloading the robots.txt, saving it as robots.txt.html, and then trying to read robots.txt (unsuccessfully), and retrying to fetch robots.txt. I didn't keep the details when this happened, so some of this may be off. I'll have to spend some time reproducing this.

My guess is that the robots.txt file may have been served as text/html, in which case it's debatable what wget's reaction should be.

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by micahcowan (Submitted the item)

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-06-11 micahcowan Planned Release1.12 1.14
    2008-11-04 micahcowan Priority6 4
    2007-08-27 micahcowan Priority8 6
    2007-08-01 micahcowan Priority5 - Normal 8
    2007-07-20 micahcowan Work RequiredNone 1 - Days
    2007-07-20 micahcowan CategoryNone Program Logic
        Planned ReleaseNone 1.12

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code