bugGNU Wget - Bugs: bug #61649, Wget not honouring...

 
 

bug #61649: Wget not honouring Content-Encoding: gzip

Submitter:  None
Submitted:  Thu 09 Dec 2021 01:43:42 PM UTC
   
 
Category:  Protocol Issue Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Jim Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.20
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 09 Jan 2022 11:47:28 AM UTC, comment #3: 

Wget2 handles the cloudfront example correctly.
And --no-compression means "do not send Accept-Encoding", while the Content-Encoding header is fully respected.
--compression=none is the same as --compression=identity (send Accept-Encding: identity).

When doing any changes, we have to keep the behavior of wget and wget2 in sync.

We can possibly stay with the wget2 behavior and amend wget to be in sync !?

 Tim Ruehsen <rockdaboot>
Group administrator
Thu 09 Dec 2021 09:34:56 PM UTC, comment #2: 

I like the idea and will gladly accept a patch for it :)

 Darshit Shah <darnir>
Group administrator
Thu 09 Dec 2021 02:05:43 PM UTC, comment #1: 

Perhaps we could use the options:
--compression=auto
Don't set the Accept-encoding header, but respond to the Content-Encoding: header uncompressing if required

--compression=none
Don't set the Accept-encoding header, and ignore the Content-Encoding: header

--compression=gzip
Set the Accept-encoding header, and respond to the Content-Encoding: header uncompressing if required

with --compression=auto being the default behaviour as that will bring it's behaviour in line with the vast majority of user agents, and deliver minimal change to the behaviour of wget whilst maintaining availability of current behaviour in the event it is needed.

 Anonymous
Thu 09 Dec 2021 01:43:42 PM UTC, original submission:  

Web servers can be seen to deliver http data with Content-Encoding: gzip whether it has been requested or not.

This is notably observable with Amazon's cloudfront.

Amazon's behaviour is likely enabled by the fact all popular web browsers will respond to the Content-Encoding: field on the response header at all times.

To test the behaviour:
wget --server-response http://d3n8a8pro7vhmx.cloudfront.net/assets/liquid/v3/main-5c3aced637d54a6ea9c73f20ce4bdc9bdc1991e1a33ab6d8d3f489a734ae06f5.js

On the response header note:
Content-Encoding: gzip

The saved file name will be
main-5c3aced637d54a6ea9c73f20ce4bdc9bdc1991e1a33ab6d8d3f489a734ae06f5.js
Note: no gz suffix.

cat main-5c3aced637d54a6ea9c73f20ce4bdc9bdc1991e1a33ab6d8d3f489a734ae06f5.js
will return binary gzipped data

zcat main-5c3aced637d54a6ea9c73f20ce4bdc9bdc1991e1a33ab6d8d3f489a734ae06f5.js
will return the plaintext javascript.

If you run:
wget --server-response --compression=auto http://d3n8a8pro7vhmx.cloudfront.net/assets/liquid/v3/main-5c3aced637d54a6ea9c73f20ce4bdc9bdc1991e1a33ab6d8d3f489a734ae06f5.js
The output will be as expected; a plaintext js file.

The above behaviour is the same, irrespective of the location of the cloudfront server, or whether http or https is used.

Suggested solution: wget --compression parameter should set the Accept-encoding of the request header but irrespective of the request header Accept-encoding field, a response-header Content-Encoding: gzip should always be honoured and the data transparently uncompressed.

 Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by darnir (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    •  

    No changes have been made to this item

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-f13d.
    Corresponding source code