bugGNU Wget - Bugs: bug #44817, http/gzip compression

 
 

bug #44817: http/gzip compression

Submitter:  None
Submitted:  Sun 12 Apr 2015 11:33:43 AM UTC
   
 
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  1.19.1
Operating System:  None Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 13 Apr 2015 12:11:15 PM UTC, comment #1: 

For me, the best place to decompress the response body would be at fd_read_body(), just after the call to fd_read(). Let fd_read() return, and uncompress the read bytes. Also, we should consider whether the data will be written to warc compressed or uncompressed. In the latter, we should discuss how to handle fd_read_line(), since it expects to encounter a line separator.

We could add an extra command line switch to let the user decide the supported compression algorithms.

Finally, the best way to implement this would be using zlib, since it supports the two most used compression algorithms on the web: gzip and deflate. Don't forget to update the dependency list for bootstrap/autoconf.

Ander Juaristi <ajuaristi>
Sun 12 Apr 2015 11:33:43 AM UTC, original submission:  

Why does wget still not have any http/gzip compression?

http://en.wikipedia.org/wiki/HTTP_compression

It's an essential feature for a web crawler because it saves both the server and the downloader huge amounts of bandwidth. Server admins are banning wget now because it requests everything uncompressed and uses 4 times the bandwidth as other programs.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by darnir (Updated the item)
  • -email is unavailable- added by ajuaristi (Posted a comment)
  •  

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-01-17 darnir StatusNone Fixed
        Open/ClosedOpen Closed
        Release1.16.3 1.19.1

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code