bugGNU Wget - Bugs: bug #23306, -N doesn't work on robots.txt


bug #23306: -N doesn't work on robots.txt

Submitted by:  Micah Cowan <micahcowan>
Submitted on:  Tue 20 May 2008 10:28:19 PM UTC  
Category:  Program Logic Severity:  2 - Minor
Priority:  7 - High Status:  Needs Discussion
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  0 - Hours Patch Included:  Yes

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Fri 16 Jan 2009 12:57:13 AM UTC, comment #3:

Should investigate why this was done, since it appears to be deliberate. Thanks for the work, Xin!

Micah Cowan <micahcowan>
Tue 30 Dec 2008 07:38:44 AM UTC, comment #2:

I debugged step by step, and finally find the reason why robots.txt is downloaded everytime. Before download robots.txt, opt.timestamping is set to false, so no initial HEAD is sent.

Attached patch could fix this issue. But I still don't know why original author did this.

(file #17163)

Xin Zou <zouxin>
Tue 20 May 2008 10:30:04 PM UTC, comment #1:

Sorry, the specific behavior that's undesirable is that "robots.txt.2", etc, is saved, rather than timestamping on "robots.txt" and saving to that.

From the output, it doesn't appear that the initial HEAD is sent, either (should verify that with -d).

Micah Cowan <micahcowan>
Tue 20 May 2008 10:28:19 PM UTC, original submission:

At least, in combination with -nd, -nH, etc.:

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #17163:  patch.diff added by zouxin (381B - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by zouxin (Updated the item)
  • -email is unavailable- added by micahcowan (Submitted the item)



    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-01-16 micahcowan Priority5 - Normal => 7 - High
        StatusConfirmed => Needs Discussion
        Patch IncludedNone => Yes
    2008-12-30 zouxin Attached File- => Added patch.diff, #17163
    2008-08-21 micahcowan Planned Release1.15 => 1.14

    Back to the top

    Powered by Savane 3.3