bugGNU Wget - Bugs: bug #23306, -N doesn't work on robots.txt


bug #23306: -N doesn't work on robots.txt

Submitted by:  Micah Cowan <micahcowan>
Submitted on:  Tue 20 May 2008 10:28:19 PM UTC  
Category: Program LogicSeverity: 2 - Minor
Priority: 7 - HighStatus: Needs Discussion
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: NoneOperating System: None
Reproducibility: NoneFixed Release: None
Planned Release: 1.14Regression: None
Work Required: 0 - HoursPatch Included: Yes

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Fri 16 Jan 2009 12:57:13 AM UTC, comment #3:

Should investigate why this was done, since it appears to be deliberate. Thanks for the work, Xin!

Micah Cowan <micahcowan>
Tue 30 Dec 2008 07:38:44 AM UTC, comment #2:

I debugged step by step, and finally find the reason why robots.txt is downloaded everytime. Before download robots.txt, opt.timestamping is set to false, so no initial HEAD is sent.

Attached patch could fix this issue. But I still don't know why original author did this.

(file #17163)

Xin Zou <zouxin>
Tue 20 May 2008 10:30:04 PM UTC, comment #1:

Sorry, the specific behavior that's undesirable is that "robots.txt.2", etc, is saved, rather than timestamping on "robots.txt" and saving to that.

From the output, it doesn't appear that the initial HEAD is sent, either (should verify that with -d).

Micah Cowan <micahcowan>
Tue 20 May 2008 10:28:19 PM UTC, original submission:

At least, in combination with -nd, -nH, etc.:

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

Attached Files
file #17163:  patch.diff added by zouxin (381B - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by zouxin (Updated the item)
  • -unavailable- added by micahcowan (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 16 Jan 2009 12:57:13 AM UTCmicahcowanPriority5 - Normal=>7 - High
      StatusConfirmed=>Needs Discussion
      Patch IncludedNone=>Yes
    Tue 30 Dec 2008 07:38:44 AM UTCzouxinAttached File-=>Added patch.diff, #17163
    Thu 21 Aug 2008 11:44:17 PM UTCmicahcowanPlanned Release1.15=>1.14

    Back to the top

    Powered by Savane 3.1-cleanup1