bugGNU Wget - Bugs: bug #53818, Proposal: Check HTML suffix (for...


bug #53818: Proposal: Check HTML suffix (for TEXTHTML flag) also on unchanged files

Submitter:  Tsukasa OI <a4lg>
Submitted:  Thu 03 May 2018 10:00:52 AM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Inspected
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  1.20 Regression:  No
Work Required:  None Patch Included:  Yes
* Mandatory Fields

Add a New Comment Rich Markup

Thu 03 May 2018 10:04:13 AM UTC, comment #1: 

Sorry, when I fiddled the code I accidentally made a broken patch. I will upload v2.

(file #44070)

Tsukasa OI <a4lg>
Thu 03 May 2018 10:00:52 AM UTC, original submission:  

Version: 1.19.4

If both `-r' (recursive) and `-N' (check timestamp) options are given and the server returns 304 (Not Modified), the HTML file (already downloaded) is not considered as a HTML file and links in the HTML file are not followed.

If we want to (periodically) backup some website (all pages are linked from index.html directly or indirectly) to track some changes while avoiding unnecessary downloads, we naturally use `-N' option. However, if some "leaf" pages are changed but index.html is unchanged, we could miss some important changes.

I hate this behavior (`-nc' option mostly works because it guesses HTML file by its file name suffix but `-N' doesn't) so I decided to propose a small change.

The attached patch reuses `get_file_flags` (which guesses HTML file by file name suffix when -nc (no clobber) option is given) if the server returns 304 (Not Modified).

Note that:

  1. This patch slightly changes Wget's behavior.
  2. It makes a caveat similar to bug #50935. If solution to bug #50935 is invented, it can be (and should be) applied to this.
  3. I (as author) consider this patch is too small to be copyrighted.

I tested the patch but I'm not sure whether this patch is suitable for upstream merge. I consider this as improvement but you may consider I broke the behavior.

Please let me know if you have any feedback about this.

Tsukasa OI <a4lg>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #44070:  0001-Check-HTML-suffix-also-on-unchanged-files-v2.patch added by a4lg (2KiB - application/octet-stream - fixed compilation error)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by darnir (Updated the item)
  • -email is unavailable- added by a4lg (Submitted the item)

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-11-12 darnir StatusNone Inspected
        Planned ReleaseNone 1.20
    2018-05-03 a4lg Attached File- Added 0001-Check-HTML-suffix-also-on-unchanged-files-v2.patch, #44070
    2018-05-03 a4lg Attached File- Added 0001-Check-HTML-suffix-also-on-unchanged-files.patch, #44069

    Back to the top

    Powered by Savane 3.13-3cd0.
    Corresponding source code