bugGNU Wget - Bugs: bug #20331, robots.txt, robots.txt.html -E -k

 
 

bug #20331: robots.txt, robots.txt.html -E -k

Submitter:  Micah Cowan <micahcowan>
Submitted:  Tue 03 Jul 2007 08:19:34 AM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  4 Status:  None
Privacy:  Public Assigned to:  micahcowan
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  1 - Days Patch Included:  None
* Mandatory Fields

Post a Comment

Add a New Comment Rich Markup
   

Discussion

Mon 27 Aug 2007 05:16:25 PM UTC, comment #1: 

Related to bug 20749?

Micah Cowan <micahcowan>
Tue 03 Jul 2007 08:19:34 AM UTC, original submission:  

When I was doing a recursive retrieval with the -E option, it was downloading the robots.txt, saving it as robots.txt.html, and then trying to read robots.txt (unsuccessfully), and retrying to fetch robots.txt. I didn't keep the details when this happened, so some of this may be off. I'll have to spend some time reproducing this.

My guess is that the robots.txt file may have been served as text/html, in which case it's debatable what wget's reaction should be.

Micah Cowan <micahcowan>

 

Attached Files

This item currently has no attached files.

(Note: upload size limit is set to 4.0MiB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Dependencies

This item does not depend on any other items.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by micahcowan (Submitted the item)
  •  

    History

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-06-11 micahcowan Planned Release1.12 1.14
    2008-11-04 micahcowan Priority6 4
    2007-08-27 micahcowan Priority8 6
    2007-08-01 micahcowan Priority5 - Normal 8
    2007-07-20 micahcowan Work RequiredNone 1 - Days
    2007-07-20 micahcowan CategoryNone Program Logic
        Planned ReleaseNone 1.12

    Back to the top

    Powered by Savane 3.16-11ef.
    Corresponding source code