bugGNU Wget - Bugs: bug #44817, http/gzip compression


bug #44817: http/gzip compression

Submitted by:  None
Submitted on:  Sun 12 Apr 2015 11:33:43 AM UTC  
Category: Feature RequestSeverity: 3 - Normal
Priority: 5 - NormalStatus: Fixed
Privacy: PublicAssigned to: None
Originator Name: Originator Email: -unavailable-
Open/Closed: ClosedRelease: 1.19.1
Operating System: NoneReproducibility: Every Time
Fixed Release: NonePlanned Release: None
Regression: NoneWork Required: None
Patch Included: No

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Mon 13 Apr 2015 12:11:15 PM UTC, comment #1:

For me, the best place to decompress the response body would be at fd_read_body(), just after the call to fd_read(). Let fd_read() return, and uncompress the read bytes. Also, we should consider whether the data will be written to warc compressed or uncompressed. In the latter, we should discuss how to handle fd_read_line(), since it expects to encounter a line separator.

We could add an extra command line switch to let the user decide the supported compression algorithms.

Finally, the best way to implement this would be using zlib, since it supports the two most used compression algorithms on the web: gzip and deflate. Don't forget to update the dependency list for bootstrap/autoconf.

Ander Juaristi <ajuaristi>
Sun 12 Apr 2015 11:33:43 AM UTC, original submission:

Why does wget still not have any http/gzip compression?


It's an essential feature for a web crawler because it saves both the server and the downloader huge amounts of bandwidth. Server admins are banning wget now because it requests everything uncompressed and uses 4 times the bandwidth as other programs.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by darnir (Updated the item)
  • -unavailable- added by ajuaristi (Posted a comment)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 17 Jan 2018 01:28:40 PM UTCdarnirStatusNone=>Fixed

    Back to the top

    Powered by Savane 3.1-cleanup1