bugGNU Wget - Bugs: bug #63308, wget uses http_proxy variable when...


bug #63308: wget uses http_proxy variable when I download HSTS http:// link

Submitter:  Askar Safin <safinaskar>
Submitted:  Wed 02 Nov 2022 06:10:57 PM UTC
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  None Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  No
Work Required:  None Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup

Wed 02 Nov 2022 06:10:57 PM UTC, original submission:  

When I try to download http:// link using wget and wget tries to download https:// instead (because of HSTS policy), then wget tries to use proxy from http_proxy variable, which is (I think) wrong.

Steps to reproduce:

1. Run "wget http://d-i.debian.org". $http_proxy should not be set. This will store d-i.debian.org to HSTS database
2. Verify that d-i.debian.org got to HSTS database ( "$HOME/.wget-hsts" )
3. Install "apt-cacher-ng". Don't modify its default configuration
4. Set http_proxy to "http://localhost:3142/" (this is port used by apt-cacher-ng)
5. Run "wget http://d-i.debian.org". You will see this:

%%% begin %%%
root@f521fb232cf5:/# wget http://d-i.debian.org
URL transformed to HTTPS due to an HSTS policy
--2022-11-02 16:50:19--  https://d-i.debian.org/
Resolving localhost (localhost)..., ::1
Connecting to localhost (localhost)||:3142... connected.
Proxy tunneling failed: CONNECT denied (ask the admin to allow HTTPS tunnels)Unable to establish SSL connection.
%%% end %%%

Wget version is 1.21.3.

As you can see URL transforms to https://d-i.debian.org/ and wget tries to download this new URL using content of $http_proxy variable. But I think this is wrong, because this new URL is https:// URL, and thus $http_proxy should not be used.

If I run "wget https://d-i.debian.org", then $http_proxy is (I think correctly) ignored. But if I run "wget http://d-i.debian.org", then URL transforms to https:// one, and $http_proxy is used for that https:// URL, which is (I think) wrong.

You can see full steps of reproducing with output (including output of --debug) here: https://builds.sr.ht/~safinaskar/job/874219 . Click "view manifest ยป" to see script (with comments).

Now let me say why I need all this.

I have a script, which downloads (using wget) files from various Debian domains using various URLs, some http and some https. For speed I use apt-cacher-ng and set $http_proxy when I run this script. I use apt-cacher-ng in its default configuration. In the default configuration apt-cacher-ng doesn't have any kind of https support.

So I set $http_proxy var, but don't set $https_proxy var. And this works. When I download (using wget) files from http:// URLs, they are downloaded through apt-cacher-ng. And when I download files from https:// URLs, then are downloaded directly (and this is good, because apt-cacher-ng in its default configuration doesn't support https). And this is exactly what I want.

But when I use "wget http://d-i.debian.org/...", it fails (let's assume that we already run "wget http://d-i.debian.org/..." at least one time in the past, i. e. let's assume the domain is already in HSTS database). Wget tries to download https:// URL, not http:// one, and it tries to do this using proxy! But the proxy doesn't support https, so the downloading fails. I. e. wget tries to download https:// link using $http_proxy variable, which is (in my opinion) wrong.

Of course, I can use some workaround, i. e. to configure apt-cacher-ng differently or just to use "wget https://d-i.debian.org/..." instead of "wget http://d-i.debian.org/...", but I still think there is a bug in wget

Askar Safin <safinaskar>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by safinaskar (Submitted the item)

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code