bugGNU Wget - Bugs: bug #51666, Please hash the hostname in...


bug #51666: Please hash the hostname in ~/.wget-hsts files

Submitter:  Noël Köthe <nok>
Submitted:  Sat 05 Aug 2017 02:14:03 PM UTC
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  trunk Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  No
Work Required:  None Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Fri 24 Aug 2018 12:31:04 PM UTC, comment #6: 

Thank you for your feedback.

Regarding the salt: I used one because OpenSSH uses one and that was both what the original bug report references, and written by somebody likely more experienced than me Re: security. If you want me to remove the salting or something I can do so.

This version of the patch (only one this time, because there's little point in only separating adding the tests) has the following changes from the previous patchset:

  • Uses gnulib hashing functions instead of gnutls/openssl ones (Sorry, I didn't know gnulib had those!).
  • Fixes the interleaved versions duplicate entry thing.
  • Adds dependency on gnulib's crypto/gc and crypto/gc-random (not any crypto/gc-<hash algorithm>) to generate decent quality randomness for the salt. Seeing as the salt apparently isn't that important, I guess it could be switched to random_number if this is a problem.
  • In case someone does want to switch to crypto/gc-<hash algorithm> in the future (to easily add another hashing algorithm or something), the prefix was switched from 1 to 5 to match the value of GC_SHA256 in gc.h.

(file #44848)

Ethan Welker <nuew>
Fri 24 Aug 2018 08:46:49 AM UTC, comment #5: 

Thanks for addressing the issue.

Saving the salt together with the (salted) hash isn't of big help when we talk about a limited set of input strings. You can get complete lists of existing domains and brute force through them in a few seconds. Can even be optimized by starting with the top 1m domains. I just mention this to make clear that this way of obscuring is far from being safe. It is just slightly more effort to reverse the domain names in comparison to unsalted hashes.

Anyways, it helps from being fly-by looked at, e.g. on the console.

I would like to ask you to not use OpenSSL for hashing. We have/use the SHA256 digest functions from gnulib anyways. So it should be straight forward.

Tim Ruehsen <rockdaboot>
Group administrator
Fri 24 Aug 2018 05:18:45 AM UTC, comment #4: 


I've written a series of patches (attached) that adds this feature. Please let me know if there's some other way that I should be submitting these instead.

Some notes:

  • I have little security experience, so these should be checked over somewhat carefully.
  • The OpenSSL implementation of ssl_hash_salt uses functions only in OpenSSL 0.9.7 and later. I'm not familiar enough with OpenSSL to get it to support anything prior to that, but the rest of openssl.c seems to imply that some versions before that are supported. Sorry about this!
  • This patch seamlessly and automatically converts old hsts databases to hashed databases. Unpatched versions of Wget parse those databases without issue, though they functionally ignore any converted entry.
    • If a patched Wget creates a hashed entry for a HSTS-enabled website, then an unpatched Wget creates a unhashed entry for the same website, then a patched Wget converting that entry will create a duplicate entry in the store. I'm not sure how much of a concern this is.
  • This switches the HSTS store from a hash table to an array, and each search for a host creates a hash for each element in that array until either one matches or the end of the array is reached. This isn't a problem for reasonably sized stores; in my test on a 50-entry .wget-hsts, an unpatched wget was 100 ms slower (presumably due to network jitter). However I can imagine that if you have several thousand entries this could become a noticeable slowdown.

(file #44842, file #44843, file #44844)

Ethan Welker <nuew>
Fri 18 Aug 2017 06:53:12 PM UTC, comment #3: 

> We can do both, hash and still keep the readable to the user only

... hash and still keep the files readable ...

Ander Juaristi <ajuaristi>
Fri 18 Aug 2017 06:51:12 PM UTC, comment #2: 

I'm not generally against these kind of small tweaks that don't harm and slightly improve user's privacy.

If Firefox doesn't do it, we don't care: it's their business and they will end up doing it if users request that feature (maybe because they saw it in wget).

Private SSH keys can be protected with a password if you want to.

> While we could hash anything, it would be way safer for you to protect your complete home directory

We can do both, hash and still keep the readable to the user only. If the overhead is not much I would go for it. That is the basis of every security framework out there: if the benefits of having 2 security mechanisms instead of only 1 outweigh the drawbacks, then implement 2 instead of 1.

Ander Juaristi <ajuaristi>
Sat 05 Aug 2017 03:07:46 PM UTC, comment #1: 

The files are read/write only to the user (in case you didn't protect your home directory appropriately).

If you are still concerned, what about ~/.mozilla/firefox/*.default/SiteSecurityServiceState.txt and likely hundreds or thousands of other files containing potential private information ?

Not thinking of your private keys lying around in .ssh/.

And even file dates and sizes are potential private data leaks.

While we could hash anything, it would be way safer for you to protect your complete home directory.

We could think about an option, just in case you want to keep your HSTS cache in a public space, e.g. /var. But we already think of using SQLite for storing and maybe sharing with Firefox.

Tim Ruehsen <rockdaboot>
Group administrator
Sat 05 Aug 2017 02:14:03 PM UTC, original submission:  


a feature request from a Debian user:


I recently discovered the .wget-hsts file in my home directory which is used to persist state for HSTS:


It contains lines such as:

  # HSTS 1.0 Known Hosts database for GNU Wget.
  # Edit at your own risk.
  # <hostname>  <port>  <incl. subdomains>      <created>       <max-age>
  github.com    0       1       1450887745      31536000
  ftp-master.debian.org 0       0       1472482586      15552000
  diffoscope.org        0       0       1449765396      15768000
  reproducible-builds.org       0       0       1471542629      15552000
  www.dropbox.com  0       1       1458394011      15552000
  reproducible.debian.net       0       0       1448074844      15552000

I can't help but think of ~/.ssh/known_hosts which moved to hashing the hostname for various security/privacy concerns. Shouldn't wget make the parallel change?

Thank you.

Noël Köthe <nok>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by nuew (Updated the item)
  • -email is unavailable- added by ajuaristi (Posted a comment)
  • -email is unavailable- added by nok
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by nok (Submitted the item)
  • -email is unavailable- added by nok

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-08-24 nuew Attached File- Added 0001-Hash-domains-in-HSTS-database-to-improve-privacy.patch, #44848
    2018-08-24 nuew Attached File- Added 0001-Add-utility-functions-for-hashing-and-salting.patch, #44842
        Attached File- Added 0002-src-hsts.c-Hash-HSTS-enabled-domains-to-improve-priv.patch, #44843
        Attached File- Added 0003-src-hsts.c-Add-test-for-hashed-domains-stored-in-dat.patch, #44844
    2017-08-07 nok Carbon-Copy- Added -email is unavailable-
    2017-08-05 nok Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-3cd0.
    Corresponding source code