bugGNU Wget - Bugs: bug #20331, robots.txt, robots.txt.html -E -k

 
 

bug #20331: robots.txt, robots.txt.html -E -k

Submitted by:  Micah Cowan <micahcowan>
Submitted on:  Tue 03 Jul 2007 08:19:34 AM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 4Status: None
Privacy: PublicAssigned to: Micah Cowan <micahcowan>
Originator Name: Open/Closed: Open
Release: NoneOperating System: None
Reproducibility: NoneFixed Release: None
Planned Release: 1.14Regression: None
Work Required: 1 - DaysPatch Included: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 27 Aug 2007 05:16:25 PM UTC, comment #1:

Related to bug 20749?

Micah Cowan <micahcowan>In charge of this item.
Tue 03 Jul 2007 08:19:34 AM UTC, original submission:

When I was doing a recursive retrieval with the -E option, it was downloading the robots.txt, saving it as robots.txt.html, and then trying to read robots.txt (unsuccessfully), and retrying to fetch robots.txt. I didn't keep the details when this happened, so some of this may be off. I'll have to spend some time reproducing this.

My guess is that the robots.txt file may have been served as text/html, in which case it's debatable what wget's reaction should be.

Micah Cowan <micahcowan>In charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by micahcowan (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 11 Jun 2009 07:59:07 PM UTCmicahcowanPlanned Release1.12=>1.14
    Tue 04 Nov 2008 01:36:34 AM UTCmicahcowanPriority6=>4
    Mon 27 Aug 2007 05:16:25 PM UTCmicahcowanPriority8=>6
    Wed 01 Aug 2007 04:34:29 AM UTCmicahcowanPriority5 - Normal=>8
    Fri 20 Jul 2007 11:47:49 PM UTCmicahcowanWork RequiredNone=>1 - Days
    Fri 20 Jul 2007 02:03:53 AM UTCmicahcowanCategoryNone=>Program Logic
      Planned ReleaseNone=>1.12

    Back to the top


    Powered by Savane 3.1-cleanup1