bugGNU Wget - Bugs: bug #46611, log errors with...

 
 

bug #46611: log errors with --trust-server-names

Submitter:  Shawn <shawnkhall>
Submitted:  Sat 05 Dec 2015 09:31:03 AM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  rockdaboot
Originator Name:  Open/Closed:  Open
Release:  trunk Operating System:  Microsoft Windows
Reproducibility:  Intermittent Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 23 Mar 2016 03:53:17 PM UTC, comment #10: 

The stat() is done with the original filename (.exe) in ensure_extension(). If it exists, '.html' is appended. That is also what your strace output shows.

ensure_extension() is only called when Wget assumes that the original is a HTML file (wrong default assumption in this case).

From that point on (http.c, ~L3578) the extension is part of the file name.

If the server answers with 200 OK instead of with 304, Wget would continue with the .html appended instead of stopping.

IMO, this is not only a display bug.
We'll (c|sh)ould write a few test cases with all those combinations of redirections, 200, 304, If-Modified-Since that plays a role here.

Tim Ruehsen <rockdaboot>
Group administrator
Thu 17 Mar 2016 10:41:47 PM UTC, comment #9: 

The problem is not that with -E a .html is appended, but that it checks .exe but reports about .exe.html

Running:
LANG=C strace -e stat,open wget -E -d --timestamping --trust-server-names http://downloads.malwarebytes.org/file/mbam

I get:
304 Not Modified
Disabling further reuse of socket 3.
Closed fd 3
Registered socket 4 for persistent reuse.
stat("mbam-setup-2.2.0.1024.exe", {st_mode=S_IFREG|0644, st_size=4509107, ...}) = 0
File 'mbam-setup-2.2.0.1024.exe.html' not modified on server. Omitting download.

which shows very clearly how wget worked on mbam-setup-2.2.0.1024.exe but is reporting about the missing mbam-setup-2.2.0.1024.exe.html

I agree in not changing the default Content-Type, but that's not the bug :)

Ángel <angel_>
Thu 17 Mar 2016 09:54:32 PM UTC, comment #8: 

I think it is too risky to change this behavior now and for something that is not clear.  Who knows how many scripts we are going to break with such a change.

Giuseppe Scrivano <gscrivano>
Group administrator
Wed 16 Mar 2016 11:50:31 AM UTC, comment #7: 

Sorry to answer my own question:

From the Wget source code:

  /* If content-type is not given, assume text/html.  This is because
     of the multitude of broken CGI's that "forget" to generate the
     content-type.  */

We should leave it as it is.
We could do
- content sniffing
- examine the file name extension

WDYT ?

Tim Ruehsen <rockdaboot>
Group administrator
Wed 16 Mar 2016 11:20:13 AM UTC, comment #6: 

The reason why the wrong '.html' extension is sometimes mentioned and sometimes now is that there are different servers that we become redirected to.

One of the servers answers with
HTTP/1.1 304 Not Modified
Date: Wed, 16 Mar 2016 10:55:31 GMT
Accept-Ranges: bytes
ETag: "1444344092"
Content-Type: application/octet-stream
X-HW: 1458125731.dop012.fr7.t,1458125731.cds062.fr7.c
Content-Disposition: attachment; filename="mbam-setup-2.2.0.1024.exe"

The second server answers (here we see the .html later in the logs):
HTTP/1.1 304 Not Modified
Accept-Ranges: bytes
Cache-Control: max-age=86400, public
Date: Wed, 16 Mar 2016 10:37:45 GMT
Etag: "482eb1-15d8fd8-5219f871809fa"
Expires: Thu, 17 Mar 2016 10:37:45 GMT
Last-Modified: Thu, 08 Oct 2015 22:38:53 GMT
Server: ECAcc (fcn/9FA9)
X-Cache: HIT

The Content-Type: header field is missing, which leaves Wget with the default for HTTP. And I guess Wget's default Content-Type is text/html.

RFC 2616 says:
Any HTTP/1.1 message containing an entity-body SHOULD include a Content-Type header field defining the media type of that body. If and only if the media type is not given by a Content-Type field, the recipient MAY attempt to guess the media type via inspection of its content and/or the name extension(s) of the URI used to identify the resource. If the media type remains unknown, the recipient SHOULD treat it as type "application/octet-stream".

The updated RFC 7231 3.1.1.5. says:
A sender that generates a message containing a payload body SHOULD
   generate a Content-Type header field in that message unless the
   intended media type of the enclosed representation is unknown to the
   sender.  If a Content-Type header field is not present, the recipient
   MAY either assume a media type of "application/octet-stream"
   ([RFC2046], Section 4.5.1) or examine the data to determine its type.

   In practice, resource owners do not always properly configure their
   origin server to provide the correct Content-Type for a given
   representation, with the result that some clients will examine a
   payload's content and override the specified type.  Clients that do
   so risk drawing incorrect conclusions, which might expose additional
   security risks (e.g., "privilege escalation").  Furthermore, it is
   impossible to determine the sender's intent by examining the data
   format: many data formats match multiple media types that differ only
   in processing semantics.  Implementers are encouraged to provide a
   means of disabling such "content sniffing" when it is used.


Before we are going to change Wget's default content-type to application/octet-stream, I would like to hear some voices. There might be a good reason for the current behavior of Wget.

Tim Ruehsen <rockdaboot>
Group administrator
Wed 16 Mar 2016 10:51:19 AM UTC, comment #5: 

Thanks for insisting.
Now I could reproduce it, setting this issue to confirmed.
I'll have a closer look into it.

Tim Ruehsen <rockdaboot>
Group administrator
Wed 16 Mar 2016 09:02:10 AM UTC, comment #4: 

I'm afraid I still don't understand how this is fine. I've tested now and see that you're right: the --adjust-extension feature is what is responsible for creating the ghost ".html" extension in the output log.

The thing is that the downloaded file isn't actually named ".html" -- it's clearly an EXE file and the content-type being returned is neither "text/html" nor "application/xhtml+xml" (it's "application/octet-stream"), so the appended ".html" in the log should not be there.

This "cosmetic" issue makes it difficult to reliably parse the log for downloaded file names.

Shawn <shawnkhall>
Tue 15 Dec 2015 01:48:52 PM UTC, comment #3: 

Ok, but your example has no ...exe.html' - what your initial complain was.

Again, I assume you used -E / --adjust-extension in your very first example. If you did everything is fine.

Tim Ruehsen <rockdaboot>
Group administrator
Tue 15 Dec 2015 11:07:53 AM UTC, comment #2: 

Sorry for the delay. Here's the first command with -d


c:\Downloads>wget -d --timestamping --trust-server-names http://downloads.malwarebytes.org/file/mbam

Setting --timestamping (timestamping) to 1
Setting --trust-server-names (trustservernames) to 1
DEBUG output created by Wget 1.17.1 on mingw32.

Reading HSTS entries from C:\Tools/.wget-hsts
URI encoding = 'ASCII'
converted 'http://downloads.malwarebytes.org/file/mbam' (ASCII) -> 'http://downloads.malwarebytes.org/file/mbam' (UTF-8)
--2015-12-15 03:03:28--  http://downloads.malwarebytes.org/file/mbam
Resolving downloads.malwarebytes.org (downloads.malwarebytes.org)... seconds 0.00, 93.184.216.111
Caching downloads.malwarebytes.org => 93.184.216.111
Connecting to downloads.malwarebytes.org (downloads.malwarebytes.org)|93.184.216.111|:80... seconds 0.00, connected.
Created socket 3.
Releasing 0x0000000000348670 (new refcount 1).

---request begin---
GET /file/mbam HTTP/1.1

User-Agent: Wget/1.17.1 (mingw32)

Accept: /

Accept-Encoding: identity

Host: downloads.malwarebytes.org

Connection: Keep-Alive



---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 302 Found

Accept-Ranges: bytes

Content-Type: text/html; charset=iso-8859-1

Date: Tue, 15 Dec 2015 11:03:29 GMT

Location: http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe

Server: ECS (rhv/817F)

X-Cache: 302-HIT

X-Frame-Options: SAMEORIGIN

Content-Length: 245



---response end---
302 Found
Registered socket 3 for persistent reuse.
URI content encoding = 'iso-8859-1'
Location: http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe [following]
Skipping 245 bytes of body: [<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe">here</a>.</p>
</body></html>
] done.
URI content encoding = None
converted 'http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe' (ASCII) -> 'http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe' (UTF-8)
--2015-12-15 03:03:28--  http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe
conaddr is: 93.184.216.111
Resolving data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)... seconds 0.00, 205.185.208.98
Caching data-cdn.mbamupdates.com => 205.185.208.98
Releasing 0x0000000000349b40 (new refcount 1).
Found data-cdn.mbamupdates.com in host_name_addresses_map (0000000000349b40)
Connecting to data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)|205.185.208.98|:80... seconds 0.00, connected.
Created socket 4.
Releasing 0x0000000000349b40 (new refcount 1).

---request begin---
GET /web/mbam-setup-2.2.0.1024.exe HTTP/1.1

If-Modified-Since: Thu, 08 Oct 2015 22:41:33 GMT

User-Agent: Wget/1.17.1 (mingw32)

Accept: /

Accept-Encoding: identity

Host: data-cdn.mbamupdates.com

Connection: Keep-Alive



---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 304 Not Modified

Date: Tue, 15 Dec 2015 11:03:28 GMT

Accept-Ranges: bytes

ETag: "1444344092"

Content-Type: application/octet-stream

X-HW: 1450177409.dop008.la2.t,1450177408.cds012.la2.c

Content-Disposition: attachment; filename="mbam-setup-2.2.0.1024.exe"



---response end---
304 Not Modified
Disabling further reuse of socket 3.
Closed fd 3
Registered socket 4 for persistent reuse.
File 'mbam-setup-2.2.0.1024.exe' not modified on server. Omitting download.

Saving HSTS entries to C:\Tools/.wget-hsts

Shawn <shawnkhall>
Sat 05 Dec 2015 03:05:29 PM UTC, comment #1: 

I could not reproduce the first example (...exe.html).
Are you sure you did not use -E / --adjust-extension ?
If I use that, the output looks like in your first example, else like in your second.

Please add -d to the wget options and send us that output for the first example.

Tim Ruehsen <rockdaboot>
Group administrator
Sat 05 Dec 2015 09:31:03 AM UTC, original submission:  

There appears to be a bug with the log when using he --trust-server-names feature in 1.17. The behavior is inconsistent but the following command demonstrates it (sometimes):

# wget --timestamping --trust-server-names http://downloads.malwarebytes.org/file/mbam
--2015-12-05 01:16:01--  http://downloads.malwarebytes.org/file/mbam
Resolving downloads.malwarebytes.org (downloads.malwarebytes.org)... 93.184.216.111
Connecting to downloads.malwarebytes.org (downloads.malwarebytes.org)|93.184.216.111|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe [following]
--2015-12-05 01:16:02--  http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe
Resolving data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)... 93.184.215.124
Connecting to data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)|93.184.215.124|:80... connected.
HTTP request sent, awaiting response... 304 Not Modified
File 'mbam-setup-2.2.0.1024.exe.html' not modified on server. Omitting download.


# wget --timestamping --trust-server-names http://downloads.malwarebytes.org/file/mbam
--2015-12-05 01:23:54--  http://downloads.malwarebytes.org/file/mbam
Resolving downloads.malwarebytes.org (downloads.malwarebytes.org)... 93.184.216.111
Connecting to downloads.malwarebytes.org (downloads.malwarebytes.org)|93.184.216.111|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe [following]
--2015-12-05 01:23:54--  http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe
Resolving data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)... 205.185.208.98
Connecting to data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)|205.185.208.98|:80... connected.
HTTP request sent, awaiting response... 304 Not Modified
File 'mbam-setup-2.2.0.1024.exe' not modified on server. Omitting download.


Specifically, the issue is in the last line indicating the name of the existing file. Sometimes there is an errant ".html" added to the filenames where it doesn't belong.

Shawn <shawnkhall>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by angel_ (Posted a comment)
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by shawnkhall (Submitted the item)
  •  

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-03-16 rockdaboot StatusNone Confirmed
        Assigned toNone rockdaboot
        Release1.17 trunk

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code