bugGNU Wget - Bugs: bug #23306, -N doesn't work on robots.txt

 
 

bug #23306: -N doesn't work on robots.txt

Submitter:  Micah Cowan <micahcowan>
Submitted:  Tue 20 May 2008 10:28:19 PM UTC
   
 
Category:  Program Logic Severity:  2 - Minor
Priority:  7 - High Status:  Needs Discussion
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  0 - Hours Patch Included:  Yes
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 16 Jan 2009 12:57:13 AM UTC, comment #3: 

Should investigate why this was done, since it appears to be deliberate. Thanks for the work, Xin!

Micah Cowan <micahcowan>
Tue 30 Dec 2008 07:38:44 AM UTC, comment #2: 

I debugged step by step, and finally find the reason why robots.txt is downloaded everytime. Before download robots.txt, opt.timestamping is set to false, so no initial HEAD is sent.

Attached patch could fix this issue. But I still don't know why original author did this.


(file #17163)

Xin Zou <zouxin>
Tue 20 May 2008 10:30:04 PM UTC, comment #1: 

Sorry, the specific behavior that's undesirable is that "robots.txt.2", etc, is saved, rather than timestamping on "robots.txt" and saving to that.

From the output, it doesn't appear that the initial HEAD is sent, either (should verify that with -d).

Micah Cowan <micahcowan>
Tue 20 May 2008 10:28:19 PM UTC, original submission:  

At least, in combination with -nd, -nH, etc.:


wget -N -p -nd -nH micah.cowan.name
--2008-05-20 15:03:30--  http://micah.cowan.name/
Resolving micah.cowan.name... 66.150.225.51
Connecting to micah.cowan.name|66.150.225.51|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Last-modified header missing -- time-stamps turned off.
--2008-05-20 15:03:30--  http://micah.cowan.name/
Connecting to micah.cowan.name|66.150.225.51|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html'

    [  <=>                                  ] 69,284       171K/s   in 0.4s

2008-05-20 15:03:31 (171 KB/s) - `index.html' saved [69284]

Loading robots.txt; please ignore errors.
--2008-05-20 15:03:31--  http://micah.cowan.name/robots.txt
Connecting to micah.cowan.name|66.150.225.51|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 358 [text/plain]
Saving to: `robots.txt.2'

100%[======================================>] 358         --.-K/s   in 0s

2008-05-20 15:03:31 (2.30 MB/s) - `robots.txt.2' saved [358/358]

--2008-05-20 15:03:31--  http://micah.cowan.name/wp-content/themes/ocadia/style.css
Reusing existing connection to micah.cowan.name:80.
HTTP request sent, awaiting response... 200 OK
Length: 7287 (7.1K) [text/css]
Server file no newer than local file `style.css' -- not retrieving.

--2008-05-20 15:03:31--  http://micah.cowan.name/wp-includes/images/smilies/icon_wink.gif
Reusing existing connection to micah.cowan.name:80.
HTTP request sent, awaiting response... 200 OK
Length: 170 [image/gif]
Server file no newer than local file `icon_wink.gif' -- not retrieving.

--2008-05-20 15:03:32--  http://micah.cowan.name/wp-includes/images/smilies/icon_smile.gif
Reusing existing connection to micah.cowan.name:80.
HTTP request sent, awaiting response... 200 OK
Length: 174 [image/gif]
Server file no newer than local file `icon_smile.gif' -- not retrieving.

FINISHED --2008-05-20 15:03:32--
Downloaded: 2 files, 68K in 0.4s (172 KB/s)


Micah Cowan <micahcowan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #17163:  patch.diff added by zouxin (381B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by zouxin (Updated the item)
  • -email is unavailable- added by micahcowan (Submitted the item)
  •  

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-01-16 micahcowan Priority5 - Normal 7 - High
        StatusConfirmed Needs Discussion
        Patch IncludedNone Yes
    2008-12-30 zouxin Attached File- Added patch.diff, #17163
    2008-08-21 micahcowan Planned Release1.15 1.14

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code