bugGNU Wget - Bugs: bug #20329, Make HTTP timestamping use...


bug #20329: Make HTTP timestamping use If-Modified-Since

Submitter:  Micah Cowan <micahcowan>
Submitted:  Tue 03 Jul 2007 07:48:01 AM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  7 - High Status:  Awaiting Approval
Privacy:  Public Assigned to:  micahcowan
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  1 - Days Patch Included:  Yes
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Mon 06 Jul 2009 07:48:41 PM UTC, comment #7: 

Probably too risky for 1.12 at this point; should put it in 1.13.

Micah Cowan <micahcowan>
Fri 12 Jun 2009 11:24:53 PM UTC, comment #6: 

Patch was provided by Vinothkumar Raman.

Micah Cowan <micahcowan>
Tue 10 Feb 2009 07:38:52 AM UTC, comment #5: 

Proposed solution provided, need to look it over.

(file #17429)

Micah Cowan <micahcowan>
Fri 29 Aug 2008 05:36:37 PM UTC, comment #4: 

verbatim-mode's not all that readable.

The gist is, we should go ahead and use If-Modified-Since, perhaps even now before there's true HTTP/1.1 support (provided it works in a reasonable percentage of cases); and just ensure that any Last-Modified header is sane.

Micah Cowan <micahcowan>
Fri 29 Aug 2008 05:34:23 PM UTC, comment #3: 

From IRC chat with Daniel Stenberg of curl

<Bagder> micahcowan: btw, regarding time-conditional requests (I noticed a bug entry of yours), in curl we do time checks ourselves as well, in case the data comes anyway
<Bagder> for cases where the server doesn't support it, or ignore it
<micahcowan> Bagder, talking about the If-Modified-Since thing?
<Bagder> yes
<micahcowan> Bagder, have you encountered servers that give HTTP/1.1 in the response, and yet will still serve "old" content (complete with a Last-Modified date that precedes or is identical to the one in If-Modified-Since)?
<Bagder> I'm quite sure, but I worked on that ages ago so I don't recall the details
<micahcowan> I suppose it was usually specialized CGI scripts? Or were there actual servers serving static content that suffered from this?
<Bagder> I don't remember, but given that my additional check worked they still at least provided Last-Modified: headers
<micahcowan> Bagder, then, in your view, would we be best served by requiring an explicit --use-if-modified-since (or something with a better name ;) ) to support that header, regardless of whether we do HTTP/1.1 properly, and cache information about the server's HTTP version?
<micahcowan> Alternatively, I suppose Wget could assume that HTTP/1.1 servers DTRT, but also cache information about buggy servers, and stop trying If-Modified-Since after the first time it has to terminate a connection because it got a 200 response when it shouldn't have.
<micahcowan> (in further requests to the same server, that is)
<Bagder> not really, I don't think the header will hurt in the 1.0 case, it just won't be recognized by the server.
<Bagder> and if you get a 200 with a Last-Modified, you can still make your own secondary check
<Bagder> at least on http1.0 responses
<Bagder> ... assuming they still exist ;-)
<micahcowan> Bagder, so, use If-Modified-Since by default, even now before we've got HTTP/1.1 support, instead of using HEAD, and just cut the connection if we get old content anyway?
<Bagder> yes, I think I'd prefer that
<micahcowan> And that's what curl currently does? I think I had misinterpreted your first few sentences to mean that curl does the HEAD thing that Wget currently does.
<Bagder> curl never does HEAD
<Bagder> HEAD is unreliable
<Bagder> well, it does HEAD if you ask it explicitly but not otherwise
<micahcowan> Right, I figured that was what you meant. :)
<micahcowan> I think the primary reason Wget has historically used HEAD has been for timestamping. In which case it proably is reliable enough, if it bothers to include Last-Modified. But If-Modified-Since would certainly be preferable; at least if it works in most cases (and especially if it still works even when we claim to be speaking HTTP/1.0).
<micahcowan> The other, newer, use, is for determining what the local file's name will be, when we're doing timestamping (or --no-clobber) along with Content-Disposition stuff. I'm already planning to prefer doing GET and shutting the connection in the near future: I don't like the extra HEAD requests, and HEAD is even _less_ reliable than usual when it comes to Content-Disposition, AFAICT.
<Bagder> right, HEAD makes some sense when talking 1.0 since the time-conditional requests ain't there then
<micahcowan> Thanks for your experienced input; I appreciate it. :)

Micah Cowan <micahcowan>
Tue 26 Aug 2008 07:22:38 AM UTC, comment #2: 

vinocit points out that if the remote end will still send the full content even if it hasn't been modified since the timestamp, if the server doesn't grok the If-Modified-Since header (implements HTTP/1.0). The only reliable way to avoid this is to first determine the server's version (or, have it cached somewhere). That'd mean (a) waiting on this until we have proper HTTP/1.1 support, and (b) relying on the current HEAD-then-GET mechanism until we've gotten our first response back from the server, complete with version info.

Micah Cowan <micahcowan>
Thu 02 Aug 2007 07:59:13 AM UTC, comment #1: 

Thus avoiding the need for extra HEAD requests, potentially.

Micah Cowan <micahcowan>
Tue 03 Jul 2007 07:48:01 AM UTC, original submission:  

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #17429:  patch added by micahcowan (2KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by vinocit (Updated the item)
  • -email is unavailable- added by micahcowan (Submitted the item)

    Follow 16 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-07-06 micahcowan Planned Release1.12 1.14
    2009-02-10 micahcowan Attached File- Added patch, #17429
        Priority5 - Normal 7 - High
        StatusPostponed Awaiting Approval
        Assigned tovinocit micahcowan
        Planned Release1.15 1.12
        Patch IncludedNone Yes
    2008-08-26 vinocit StatusIn Progress Postponed
    2008-08-25 micahcowan StatusConfirmed In Progress
        Assigned toNone vinocit
    2008-01-27 micahcowan Planned Release1.12 1.15
    2007-08-09 micahcowan Work RequiredNone 1 - Days
    2007-08-02 micahcowan Planned Release1.15 1.12
    2007-07-20 micahcowan CategoryNone Program Logic
        StatusNone Confirmed
        Planned ReleaseNone 1.15

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code