bugGNU Wget - Bugs: bug #44817, http/gzip compression


bug #44817: http/gzip compression

Submitted by:  None
Submitted on:  Sun 12 Apr 2015 11:33:43 AM UTC  
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  1.19.1
Operating System:  None Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  No

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Mon 13 Apr 2015 12:11:15 PM UTC, comment #1:

For me, the best place to decompress the response body would be at fd_read_body(), just after the call to fd_read(). Let fd_read() return, and uncompress the read bytes. Also, we should consider whether the data will be written to warc compressed or uncompressed. In the latter, we should discuss how to handle fd_read_line(), since it expects to encounter a line separator.

We could add an extra command line switch to let the user decide the supported compression algorithms.

Finally, the best way to implement this would be using zlib, since it supports the two most used compression algorithms on the web: gzip and deflate. Don't forget to update the dependency list for bootstrap/autoconf.

Ander Juaristi <ajuaristi>
Sun 12 Apr 2015 11:33:43 AM UTC, original submission:

Why does wget still not have any http/gzip compression?


It's an essential feature for a web crawler because it saves both the server and the downloader huge amounts of bandwidth. Server admins are banning wget now because it requests everything uncompressed and uses 4 times the bandwidth as other programs.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by darnir (Updated the item)
  • -email is unavailable- added by ajuaristi (Posted a comment)



    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-01-17 darnir StatusNone => Fixed
        Open/ClosedOpen => Closed
        Release1.16.3 => 1.19.1

    Back to the top

    Powered by Savane 3.3