bugGNU Wget - Bugs: bug #20371, Generalize --adjust-extensions to...


bug #20371: Generalize --adjust-extensions to cover wider range of types

Submitter:  Micah Cowan <micahcowan>
Submitted:  Fri 06 Jul 2007 10:35:35 PM UTC
Category:  None Severity:  3 - Normal
Priority:  3 - Low Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.15 Regression:  None
Work Required:  1 - Days Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Wed 29 Jul 2009 01:45:05 AM UTC, comment #4: 

Renamed --html-extensions to --adjust-extensions, to better reflect its actual current functionality. 4b5e21e1fdce

Leaving bug open to track the concept of generalizing the behavior to apply to a wider range of content/types (probably should use mime.types or whatever the system makes available).

Micah Cowan <micahcowan>
Fri 12 Sep 2008 07:00:38 PM UTC, comment #3: 

--mime-extensions may be a poor name choice, since HTTP != MIME. Perhaps --add-file-extensions or just --add-extensions.

Micah Cowan <micahcowan>
Thu 21 Aug 2008 11:50:24 PM UTC, comment #2: 

That last comment looks like some other feature (like, MIME-based accepts). What was originally meant for this bug is the idea that, in addition to tacking on a ".html" to files that don't have it, the --html-extension option (which should be renamed appropriately), should tack on appropriate suffixes to all files, based on their content-type.

We're already doing that in current sources for .css, now.

Micah Cowan <micahcowan>
Tue 01 Apr 2008 05:32:47 AM UTC, comment #1: 

(The following is mainly copy-pasted from my response to a GSoC proposal.)

As a directly applicable example, someone might wish to download all HTML content from a website. This is easy, as you can just specify "--accept .htm,.html" to download all files that end with those extensions.

...except that, quite a lot of HTML content is made available via URLs that don't end in those suffixes. For instance, an "index.php" file might be HTML. But then, "index.php?show=logo" might resolve to a JPEG image or somesuch.

Using "filename suffixes" is a poor way to try to determine what sort of thing a given URL refers to. HTTP has built into it ways of identifying resources, and we should be using that instead of filenames.

For instance, .htm and .html files and the HTML index.php file, will all identify their Content-Type as "text/html". On the other hand, the "index.php?show=logo" that resolves to a JPEG would be identified as "image/jpeg". It would be very useful to have Wget support accept/reject based on the information in the Content-Type header, rather than filenames. This would be determined by initial HEAD requests to verify that the type is acceptable, before issuing a full-blown GET request to fetch it.

The advantage is that Content-Type offers a much more precise way to accept or reject resources based on what kind of resource they are. The disadvantage is that, unless type information is embedded in the links themselves (which HTML supports, but few authors take advantage of), it requires extra requests to be sent to the server to determine what the Content-Type is. Also, not all servers support HEAD for all resources. Name-based matches have the advantage that the decision is made without involving the network.

Micah Cowan <micahcowan>
Fri 06 Jul 2007 10:35:35 PM UTC, original submission:  

(From TODO:)

Generalize --html-extension to something like --mime-extensions and have consult mime.types for the preferred extension.  Non-HTML files with filenames changed this way would be re-downloaded each time despite -N unless .orig files were saved for them.  (#### Why?  The HEAD request we use to implement -N would still be able to construct the correct file name based on the declared Content-Type.)
Since .orig would contain the same data as non-.orig, the latter could be just a link to the former.  Another possibility would be to implement a per-directory database called something like .wget_url_mapping containing URLs and their corresponding filenames.

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by micahcowan (Submitted the item)

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-07-29 micahcowan SummaryGeneralize --html-extension -&gt; --fix-extensions Generalize --adjust-extensions to cover wider range of types
    2009-07-29 micahcowan Priority7 - High 3 - Low
        Planned Release1.12 1.15
        Work Required0 - Hours 1 - Days
    2009-07-28 micahcowan Priority4 7 - High
        Work Required2 - Weeks 0 - Hours
        SummaryGeneralize --html-extension -&gt; --mime-extensions Generalize --html-extension -> --fix-extensions
    2008-11-04 micahcowan Priority5 - Normal 4
    2008-08-21 micahcowan Severity2 - Minor 3 - Normal
        Planned Release1.15 1.12
    2007-07-20 micahcowan Planned ReleaseNone 1.15

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code