bugGNU Wget - Bugs: bug #44817, http/gzip compression

 
 

bug #44817: http/gzip compression

Submitted by:  None
Submitted on:  Sun 12 Apr 2015 11:33:43 AM UTC  
 
Category: Feature RequestSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: Originator Email: -unavailable-
Open/Closed: OpenRelease: 1.16.3
Operating System: NoneReproducibility: Every Time
Fixed Release: NonePlanned Release: None
Regression: NoneWork Required: None
Patch Included: No

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 13 Apr 2015 12:11:15 PM UTC, comment #1:

For me, the best place to decompress the response body would be at fd_read_body(), just after the call to fd_read(). Let fd_read() return, and uncompress the read bytes. Also, we should consider whether the data will be written to warc compressed or uncompressed. In the latter, we should discuss how to handle fd_read_line(), since it expects to encounter a line separator.

We could add an extra command line switch to let the user decide the supported compression algorithms.

Finally, the best way to implement this would be using zlib, since it supports the two most used compression algorithms on the web: gzip and deflate. Don't forget to update the dependency list for bootstrap/autoconf.

Ander Juaristi <ajuaristi>
Sun 12 Apr 2015 11:33:43 AM UTC, original submission:

Why does wget still not have any http/gzip compression?

http://en.wikipedia.org/wiki/HTTP_compression

It's an essential feature for a web crawler because it saves both the server and the downloader huge amounts of bandwidth. Server admins are banning wget now because it requests everything uncompressed and uses 4 times the bandwidth as other programs.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by ajuaristi (Posted a comment)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    No Changes Have Been Made to This Item

    Back to the top


    Powered by Savane 3.1-cleanup1