bugGNU Wget - Bugs: bug #56808, wget uses HEAD method when both...


bug #56808: wget uses HEAD method when both --spider and --post-data options are used

Submitter:  None
Submitted:  Mon 26 Aug 2019 10:27:59 AM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.19.4
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup

Sun 08 May 2022 02:38:55 PM UTC, comment #3: 

The spider has two components:
1. Send a HEAD (and repeat with GET)
2. Ignore response

The 1 can be replaced with a direct --method=HEAD
Repeat functionality is doubtful: all webservers supports the HEAD.
Only some APIs that generates a response may not support the HEAD.
But a GET response to them would be an overkill because the response generation may be heavy.

The 2 is actually a useful thing e.g. you just want to send a request but not interested in the response to keep resources.
This would be better than output to /dev/null because wget can close the connection instead of a full download.

I think that a proper solution would be to allow mutual usage of the spider with all other options:
1. If the --method specified use it but ignore response.
2. If specified --post-data or --post-file then send a POST as usual but again ignore the response.
3. If none of the above is specified then make HEAD and then GET but discard response as usual.

The --content-on-error option should disable the response discarding.

This will make the logic plain and backward compatible.

Currently I'm working on the BusyBox wget clone and to keep logic simple and plain and code small this is a best option.
It would be great if GNU wget will work like that.
Even simple error message about spider and post-data conflict takes a space precious for embedded devices.

Sergey Ponomarev <stokito>
Mon 26 Aug 2019 08:00:40 PM UTC, comment #2: 

`--post-data` works by setting `opt.method` and `opt.body_data` while `--spider` works by setting `opt.method`, albeit indirectly.

Now, I believe that it makes absolutely no sense to set both of those in contradicting ways. Especially with something like `POST` or `PUT` which requires a Request body. Since a request body is not allowed with a HEAD request.

My suggestion here would be to add a check that prevents any change to `opt.method` if `--spider` is also passed.

This prevents not only `--post-data`, but also `--method` from setting something funny that makes no sense.

Darshit Shah <darnir>
Group administrator
Mon 26 Aug 2019 11:10:48 AM UTC, comment #1: 

You are right. The combination of both options doesn't make much sense.

Using a POST request with --spider seems to be dangerous as it leads to unexpected data / state on the server side. So this i not an option.

Silently doing a GET request is also not what someone would expect.

But not allowing the combination removes some degree of user's freedom of choice. I am split.

Tim Ruehsen <rockdaboot>
Group administrator
Mon 26 Aug 2019 10:27:59 AM UTC, original submission:  

HEAD request should return the same result as of GET request, except that the response body is empty.


So if both --spider and --post-data options are used, I think wget should use POST method. If wget sends HEAD request, then the response header would be that of GET request, not POST request.

Or perhaps it should be forbidden to use the two options at the same time, as is the case of --method and --post-data.

I'm not sure if this is a bug or a feature. Please ignore this if you think this is not a bug. Who would use that two options at the same time for normal use?




(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by stokito (Posted a comment)
  • -email is unavailable- added by darnir (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.12.
    Corresponding source code