bugGNU Wget - Bugs: bug #20378, An accept option based on MIME...


bug #20378: An accept option based on MIME types

Submitter:  Micah Cowan <micahcowan>
Submitted:  Fri 06 Jul 2007 10:56:51 PM UTC
Category:  Feature Request Severity:  3 - Normal
Priority:  6 Status:  Awaiting Approval
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  1.14 Regression:  None
Work Required:  None Patch Included:  Yes
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Tue 06 Sep 2016 09:03:39 AM UTC, comment #14: 

This patch looks great!
it's already four years since it was proposed.
Any idea of why it has not yet been approved?

David Portabella <dportabella>
Thu 02 Jul 2009 10:24:09 PM UTC, comment #13: 

(There's also the fact that the current behavior makes --accept useless for filtering out which specific HTML files one wants to download: --accept's not just for filename extensions, after all.)

Micah Cowan <micahcowan>
Thu 02 Jul 2009 09:56:49 PM UTC, comment #12: 

I don't see what the point could be for either --accept or --content-type if recursion is off, especially as both options should have no effect whatsoever if recursion is off (''nothing'' should/does prevent downloading explicitly-specified URLs). The existing behavior for --accept can and does cause problems for some users, and is broken anyway since it only looks at filename extensions, and so .htm and .html are traversed without any means of prevention, but .php and .asp are not. But this isn't really the place for this discussion: feel free to take it up on the mailing list (might check the archives for previous discussions)

Micah Cowan <micahcowan>
Thu 02 Jul 2009 09:07:46 PM UTC, comment #11: 

Surely thats what the recursive option is for? If recursive is off then I agree that the --content-type option should function as you suggest and only retrieve files that match the content-type, however there is no way that content-types other than HTML, etc.. can be downloaded recursively without recursion of these other content types.

Rich <pomp>
Thu 02 Jul 2009 08:56:12 PM UTC, comment #10: 

Actually, that behavior of the --accept option is annoying for many applications, and I'd just as soon not duplicate it. For instance, if what you want is to fetch all the images that are linked from one specific page, --accept still doesn't prevent the wasteful download of all kinds of pages we don't want.

If you really want HTML to be grabbed, then it should be requested IMO. We should have a separate option to fine-tune what gets deleted afterward (or, once we move to a streaming html parser, what never gets saved in the first place). There's already an issue filed for that option, IIRC.

Micah Cowan <micahcowan>
Thu 02 Jul 2009 08:45:05 PM UTC, comment #9: 

I've compiled patch file #16272 against the latest wget source but am having problems trying to use wget with the option "--content-type=image*", as you can see from the output:

HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
Not downloading, content type rejected.

From what I understand, wget isn't able to recursively crawl the website which is HTML to find images to download as the option rejects anything that doesn't match. Should the --content-type option not function in the same was as the --accept option which seems to download HTML files even if rejected but does not save them to disk?

Rich <pomp>
Mon 11 Aug 2008 06:33:42 AM UTC, comment #8: 

Attaching new patch which adds some tests. I'm not sure whether I'm using the test system correctly though, and as far as I can see there's no way to verify that it's sending a HEAD request before doing anything else.

(file #16272)

Lars Kotthoff <larsko>
Wed 06 Aug 2008 10:41:03 PM UTC, comment #7: 

Should get a suite of tests.

Micah Cowan <micahcowan>
Mon 28 Jul 2008 06:45:19 AM UTC, comment #6: 

Attaching new patch which sends a HEAD request first (which turned out be be a trivial change!) and fixes some logging and return code quirks.

(file #16195)

Lars Kotthoff <larsko>
Sat 26 Jul 2008 06:41:03 AM UTC, comment #5: 

Ah, understood. This looks a bit hairy though and would be a bigger change. I'll look into it.

Lars Kotthoff <larsko>
Sat 26 Jul 2008 06:29:48 AM UTC, comment #4: 

If it wasn't included in the response header to GET, it certainly wouldn't be included in the response to HEAD.

No, I mean it should issue an explicit HEAD to determine the content-type, to determine whether it should then issue a GET. The send_head_first bool var is what controls this, I believe. Currently it sends HEAD first when it needs to do time-stamping, and some situations involving Content-Disposition.

Micah Cowan <micahcowan>
Sat 26 Jul 2008 06:20:15 AM UTC, comment #3: 

> Probably needs to send HEAD first, though, to determine the content-type.

From reading the code, my understanding is that it does exactly that. In gethttp it reads the header of the response to figure out how to continue.

Do you mean do an explicit HEAD if the content-type isn't included in the response header?

Lars Kotthoff <larsko>
Sat 26 Jul 2008 05:57:37 AM UTC, comment #2: 

Patch looks good. Probably needs to send HEAD first, though, to determine the content-type.

Micah Cowan <micahcowan>
Sat 26 Jul 2008 05:42:46 AM UTC, comment #1: 

Attaching patch which adds --content-type=LIST and --content-type-exclude=LIST options which allow to specify MIME type patterns to filter retrieved stuff.

(file #16187)

Lars Kotthoff <larsko>
Fri 06 Jul 2007 10:56:51 PM UTC, original submission:  

From TODO:

"Handle MIME types correctly.  There should be an option to (not) retrieve files based on MIME types, e.g. `--accept-types=image/*'.  This would work for FTP by translating file extensions to MIME types using mime.types."

I'm not convinced we should bother mapping them to file extensions; we already have -A...

Micah Cowan <micahcowan>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #16272:  content-type.patch added by larsko (10KiB - text/x-patch - patch with tests)
file #16195:  content-type.patch added by larsko (6KiB - text/x-patch - Revised patch -- HEAD first, code cleanup)
file #16187:  content-type.patch added by larsko (6KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by dportabella (Posted a comment)
  • -email is unavailable- added by pomp (Posted a comment)
  • -email is unavailable- added by larsko (Updated the item)
  • -email is unavailable- added by micahcowan (Submitted the item)

    Follow 14 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-07-02 micahcowan StatusConfirmed Awaiting Approval
    2008-08-21 micahcowan Planned Release1.12 1.14
    2008-08-11 larsko Attached File- Added content-type.patch, #16272
    2008-08-06 micahcowan Severity2 - Minor 3 - Normal
        Priority5 - Normal 6
        Planned Release1.15 1.12
        Work Required0 - Hours None
    2008-07-28 larsko Attached File- Added content-type.patch, #16195
    2008-07-26 micahcowan Work Required1 - Days 0 - Hours
        Patch IncludedNone Yes
    2008-07-26 larsko Attached File- Added content-type.patch, #16187
    2008-01-27 micahcowan Planned Release2.0 1.15
    2007-08-12 mnml Carbon-Copy- Added mnml
    2007-07-20 micahcowan Planned ReleaseNone 2.0

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code