bugGNU Wget - Bugs: bug #20371, Generalize --adjust-extensions to...

 
 

bug #20371: Generalize --adjust-extensions to cover wider range of types

Submitted by:  Micah Cowan <micahcowan>
Submitted on:  Fri 06 Jul 2007 10:35:35 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Priority: 3 - LowStatus: Confirmed
Privacy: PublicAssigned to: None
Originator Name: Open/Closed: Open
Release: NoneOperating System: None
Reproducibility: NoneFixed Release: None
Planned Release: 1.15Regression: None
Work Required: 1 - DaysPatch Included: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Wed 29 Jul 2009 01:45:05 AM UTC, comment #4:

Renamed --html-extensions to --adjust-extensions, to better reflect its actual current functionality. 4b5e21e1fdce

Leaving bug open to track the concept of generalizing the behavior to apply to a wider range of content/types (probably should use mime.types or whatever the system makes available).

Micah Cowan <micahcowan>
Fri 12 Sep 2008 07:00:38 PM UTC, comment #3:

--mime-extensions may be a poor name choice, since HTTP != MIME. Perhaps --add-file-extensions or just --add-extensions.

Micah Cowan <micahcowan>
Thu 21 Aug 2008 11:50:24 PM UTC, comment #2:

That last comment looks like some other feature (like, MIME-based accepts). What was originally meant for this bug is the idea that, in addition to tacking on a ".html" to files that don't have it, the --html-extension option (which should be renamed appropriately), should tack on appropriate suffixes to all files, based on their content-type.

We're already doing that in current sources for .css, now.

Micah Cowan <micahcowan>
Tue 01 Apr 2008 05:32:47 AM UTC, comment #1:

(The following is mainly copy-pasted from my response to a GSoC proposal.)

As a directly applicable example, someone might wish to download all HTML content from a website. This is easy, as you can just specify "--accept .htm,.html" to download all files that end with those extensions.

...except that, quite a lot of HTML content is made available via URLs that don't end in those suffixes. For instance, an "index.php" file might be HTML. But then, "index.php?show=logo" might resolve to a JPEG image or somesuch.

Using "filename suffixes" is a poor way to try to determine what sort of thing a given URL refers to. HTTP has built into it ways of identifying resources, and we should be using that instead of filenames.

For instance, .htm and .html files and the HTML index.php file, will all identify their Content-Type as "text/html". On the other hand, the "index.php?show=logo" that resolves to a JPEG would be identified as "image/jpeg". It would be very useful to have Wget support accept/reject based on the information in the Content-Type header, rather than filenames. This would be determined by initial HEAD requests to verify that the type is acceptable, before issuing a full-blown GET request to fetch it.

The advantage is that Content-Type offers a much more precise way to accept or reject resources based on what kind of resource they are. The disadvantage is that, unless type information is embedded in the links themselves (which HTML supports, but few authors take advantage of), it requires extra requests to be sent to the server to determine what the Content-Type is. Also, not all servers support HEAD for all resources. Name-based matches have the advantage that the decision is made without involving the network.

Micah Cowan <micahcowan>
Fri 06 Jul 2007 10:35:35 PM UTC, original submission:

(From TODO:)

Generalize --html-extension to something like --mime-extensions and have consult mime.types for the preferred extension. Non-HTML files with filenames changed this way would be re-downloaded each time despite -N unless .orig files were saved for them. (#### Why? The HEAD request we use to implement -N would still be able to construct the correct file name based on the declared Content-Type.)

Since .orig would contain the same data as non-.orig, the latter could be just a link to the former. Another possibility would be to implement a per-directory database called something like .wget_url_mapping containing URLs and their corresponding filenames.

Micah Cowan <micahcowan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by micahcowan (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 11 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 29 Jul 2009 01:48:11 AM UTCmicahcowanSummaryGeneralize --html-extension -> --fix-extensions=>Generalize --adjust-extensions to cover wider range of types
    Wed 29 Jul 2009 01:45:04 AM UTCmicahcowanPriority7 - High=>3 - Low
      Planned Release1.12=>1.15
      Work Required0 - Hours=>1 - Days
    Tue 28 Jul 2009 07:41:33 AM UTCmicahcowanPriority4=>7 - High
      Work Required2 - Weeks=>0 - Hours
      SummaryGeneralize --html-extension -> --mime-extensions=>Generalize --html-extension -> --fix-extensions
    Tue 04 Nov 2008 01:52:07 AM UTCmicahcowanPriority5 - Normal=>4
    Thu 21 Aug 2008 11:50:24 PM UTCmicahcowanSeverity2 - Minor=>3 - Normal
      Planned Release1.15=>1.12
    Fri 20 Jul 2007 04:17:46 AM UTCmicahcowanPlanned ReleaseNone=>1.15

    Back to the top


    Powered by Savane 3.1-cleanup1