bugGNU Wget - Bugs: bug #61649, Wget not honouring...


bug #61649: Wget not honouring Content-Encoding: gzip

Submitter:  None
Submitted:  Thu 09 Dec 2021 01:43:42 PM UTC
Category:  Protocol Issue Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Jim Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.20
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Sun 09 Jan 2022 11:47:28 AM UTC, comment #3: 

Wget2 handles the cloudfront example correctly.
And --no-compression means "do not send Accept-Encoding", while the Content-Encoding header is fully respected.
--compression=none is the same as --compression=identity (send Accept-Encding: identity).

When doing any changes, we have to keep the behavior of wget and wget2 in sync.

We can possibly stay with the wget2 behavior and amend wget to be in sync !?

Tim Ruehsen <rockdaboot>
Group administrator
Thu 09 Dec 2021 09:34:56 PM UTC, comment #2: 

I like the idea and will gladly accept a patch for it :)

Darshit Shah <darnir>
Group administrator
Thu 09 Dec 2021 02:05:43 PM UTC, comment #1: 

Perhaps we could use the options:
Don't set the Accept-encoding header, but respond to the Content-Encoding: header uncompressing if required

Don't set the Accept-encoding header, and ignore the Content-Encoding: header

Set the Accept-encoding header, and respond to the Content-Encoding: header uncompressing if required

with --compression=auto being the default behaviour as that will bring it's behaviour in line with the vast majority of user agents, and deliver minimal change to the behaviour of wget whilst maintaining availability of current behaviour in the event it is needed.

Thu 09 Dec 2021 01:43:42 PM UTC, original submission:  

Web servers can be seen to deliver http data with Content-Encoding: gzip whether it has been requested or not.

This is notably observable with Amazon's cloudfront.

Amazon's behaviour is likely enabled by the fact all popular web browsers will respond to the Content-Encoding: field on the response header at all times.

To test the behaviour:
wget --server-response http://d3n8a8pro7vhmx.cloudfront.net/assets/liquid/v3/main-5c3aced637d54a6ea9c73f20ce4bdc9bdc1991e1a33ab6d8d3f489a734ae06f5.js

On the response header note:
Content-Encoding: gzip

The saved file name will be
Note: no gz suffix.

cat main-5c3aced637d54a6ea9c73f20ce4bdc9bdc1991e1a33ab6d8d3f489a734ae06f5.js
will return binary gzipped data

zcat main-5c3aced637d54a6ea9c73f20ce4bdc9bdc1991e1a33ab6d8d3f489a734ae06f5.js
will return the plaintext javascript.

If you run:
wget --server-response --compression=auto http://d3n8a8pro7vhmx.cloudfront.net/assets/liquid/v3/main-5c3aced637d54a6ea9c73f20ce4bdc9bdc1991e1a33ab6d8d3f489a734ae06f5.js
The output will be as expected; a plaintext js file.

The above behaviour is the same, irrespective of the location of the cloudfront server, or whether http or https is used.

Suggested solution: wget --compression parameter should set the Accept-encoding of the request header but irrespective of the request header Accept-encoding field, a response-header Content-Encoding: gzip should always be honoured and the data transparently uncompressed.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by darnir (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-ee12.
    Corresponding source code