bugGNU Wget - Bugs: bug #46611, log errors with...

 
 

bug #46611: log errors with --trust-server-names

Submitted by:  Shawn <shawnkhall>
Submitted on:  Sat 05 Dec 2015 09:31:03 AM UTC  
 
Category: Program LogicSeverity: 3 - Normal
Priority: 5 - NormalStatus: Confirmed
Privacy: PublicAssigned to: Tim Ruehsen <rockdaboot>
Originator Name: Open/Closed: Open
Release: trunkOperating System: Microsoft Windows
Reproducibility: IntermittentFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: No

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 23 Mar 2016 03:53:17 PM UTC, comment #10:

The stat() is done with the original filename (.exe) in ensure_extension(). If it exists, '.html' is appended. That is also what your strace output shows.

ensure_extension() is only called when Wget assumes that the original is a HTML file (wrong default assumption in this case).

From that point on (http.c, ~L3578) the extension is part of the file name.

If the server answers with 200 OK instead of with 304, Wget would continue with the .html appended instead of stopping.

IMO, this is not only a display bug.
We'll (c|sh)ould write a few test cases with all those combinations of redirections, 200, 304, If-Modified-Since that plays a role here.

Tim Ruehsen <rockdaboot>
Project AdministratorIn charge of this item.
Thu 17 Mar 2016 10:41:47 PM UTC, comment #9:

The problem is not that with -E a .html is appended, but that it checks .exe but reports about .exe.html

Running:
LANG=C strace -e stat,open wget -E -d --timestamping --trust-server-names http://downloads.malwarebytes.org/file/mbam

I get:
304 Not Modified
Disabling further reuse of socket 3.
Closed fd 3
Registered socket 4 for persistent reuse.
stat("mbam-setup-2.2.0.1024.exe", {st_mode=S_IFREG|0644, st_size=4509107, ...}) = 0
File 'mbam-setup-2.2.0.1024.exe.html' not modified on server. Omitting download.

which shows very clearly how wget worked on mbam-setup-2.2.0.1024.exe but is reporting about the missing mbam-setup-2.2.0.1024.exe.html

I agree in not changing the default Content-Type, but that's not the bug :)

Ángel <angel_>
Thu 17 Mar 2016 09:54:32 PM UTC, comment #8:

I think it is too risky to change this behavior now and for something that is not clear. Who knows how many scripts we are going to break with such a change.

Giuseppe Scrivano <gscrivano>
Project Administrator
Wed 16 Mar 2016 11:50:31 AM UTC, comment #7:

Sorry to answer my own question:

From the Wget source code:

/* If content-type is not given, assume text/html. This is because
of the multitude of broken CGI's that "forget" to generate the
content-type. */

We should leave it as it is.
We could do
- content sniffing
- examine the file name extension

WDYT ?

Tim Ruehsen <rockdaboot>
Project AdministratorIn charge of this item.
Wed 16 Mar 2016 11:20:13 AM UTC, comment #6:

The reason why the wrong '.html' extension is sometimes mentioned and sometimes now is that there are different servers that we become redirected to.

One of the servers answers with
HTTP/1.1 304 Not Modified
Date: Wed, 16 Mar 2016 10:55:31 GMT
Accept-Ranges: bytes
ETag: "1444344092"
Content-Type: application/octet-stream
X-HW: 1458125731.dop012.fr7.t,1458125731.cds062.fr7.c
Content-Disposition: attachment; filename="mbam-setup-2.2.0.1024.exe"

The second server answers (here we see the .html later in the logs):
HTTP/1.1 304 Not Modified
Accept-Ranges: bytes
Cache-Control: max-age=86400, public
Date: Wed, 16 Mar 2016 10:37:45 GMT
Etag: "482eb1-15d8fd8-5219f871809fa"
Expires: Thu, 17 Mar 2016 10:37:45 GMT
Last-Modified: Thu, 08 Oct 2015 22:38:53 GMT
Server: ECAcc (fcn/9FA9)
X-Cache: HIT

The Content-Type: header field is missing, which leaves Wget with the default for HTTP. And I guess Wget's default Content-Type is text/html.

RFC 2616 says:
Any HTTP/1.1 message containing an entity-body SHOULD include a Content-Type header field defining the media type of that body. If and only if the media type is not given by a Content-Type field, the recipient MAY attempt to guess the media type via inspection of its content and/or the name extension(s) of the URI used to identify the resource. If the media type remains unknown, the recipient SHOULD treat it as type "application/octet-stream".

The updated RFC 7231 3.1.1.5. says:
A sender that generates a message containing a payload body SHOULD
generate a Content-Type header field in that message unless the
intended media type of the enclosed representation is unknown to the
sender. If a Content-Type header field is not present, the recipient
MAY either assume a media type of "application/octet-stream"
([RFC2046], Section 4.5.1) or examine the data to determine its type.

In practice, resource owners do not always properly configure their
origin server to provide the correct Content-Type for a given
representation, with the result that some clients will examine a
payload's content and override the specified type. Clients that do
so risk drawing incorrect conclusions, which might expose additional
security risks (e.g., "privilege escalation"). Furthermore, it is
impossible to determine the sender's intent by examining the data
format: many data formats match multiple media types that differ only
in processing semantics. Implementers are encouraged to provide a
means of disabling such "content sniffing" when it is used.

Before we are going to change Wget's default content-type to application/octet-stream, I would like to hear some voices. There might be a good reason for the current behavior of Wget.

Tim Ruehsen <rockdaboot>
Project AdministratorIn charge of this item.
Wed 16 Mar 2016 10:51:19 AM UTC, comment #5:

Thanks for insisting.
Now I could reproduce it, setting this issue to confirmed.
I'll have a closer look into it.

Tim Ruehsen <rockdaboot>
Project AdministratorIn charge of this item.
Wed 16 Mar 2016 09:02:10 AM UTC, comment #4:

I'm afraid I still don't understand how this is fine. I've tested now and see that you're right: the --adjust-extension feature is what is responsible for creating the ghost ".html" extension in the output log.

The thing is that the downloaded file isn't actually named ".html" -- it's clearly an EXE file and the content-type being returned is neither "text/html" nor "application/xhtml+xml" (it's "application/octet-stream"), so the appended ".html" in the log should not be there.

This "cosmetic" issue makes it difficult to reliably parse the log for downloaded file names.

Shawn <shawnkhall>
Tue 15 Dec 2015 01:48:52 PM UTC, comment #3:

Ok, but your example has no ...exe.html' - what your initial complain was.

Again, I assume you used -E / --adjust-extension in your very first example. If you did everything is fine.

Tim Ruehsen <rockdaboot>
Project AdministratorIn charge of this item.
Tue 15 Dec 2015 11:07:53 AM UTC, comment #2:

Sorry for the delay. Here's the first command with -d

c:\Downloads>wget -d --timestamping --trust-server-names http://downloads.malwarebytes.org/file/mbam

Setting --timestamping (timestamping) to 1
Setting --trust-server-names (trustservernames) to 1
DEBUG output created by Wget 1.17.1 on mingw32.

Reading HSTS entries from C:\Tools/.wget-hsts
URI encoding = 'ASCII'
converted 'http://downloads.malwarebytes.org/file/mbam' (ASCII) -> 'http://downloads.malwarebytes.org/file/mbam' (UTF-8)
--2015-12-15 03:03:28-- http://downloads.malwarebytes.org/file/mbam
Resolving downloads.malwarebytes.org (downloads.malwarebytes.org)... seconds 0.00, 93.184.216.111
Caching downloads.malwarebytes.org => 93.184.216.111
Connecting to downloads.malwarebytes.org (downloads.malwarebytes.org)|93.184.216.111|:80... seconds 0.00, connected.
Created socket 3.
Releasing 0x0000000000348670 (new refcount 1).

---request begin---
GET /file/mbam HTTP/1.1

User-Agent: Wget/1.17.1 (mingw32)

Accept: /

Accept-Encoding: identity

Host: downloads.malwarebytes.org

Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 302 Found

Accept-Ranges: bytes

Content-Type: text/html; charset=iso-8859-1

Date: Tue, 15 Dec 2015 11:03:29 GMT

Location: http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe

Server: ECS (rhv/817F)

X-Cache: 302-HIT

X-Frame-Options: SAMEORIGIN

Content-Length: 245

---response end---
302 Found
Registered socket 3 for persistent reuse.
URI content encoding = 'iso-8859-1'
Location: http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe [following]
Skipping 245 bytes of body: [<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe">here</a>.</p>
</body></html>
] done.
URI content encoding = None
converted 'http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe' (ASCII) -> 'http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe' (UTF-8)
--2015-12-15 03:03:28-- http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe
conaddr is: 93.184.216.111
Resolving data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)... seconds 0.00, 205.185.208.98
Caching data-cdn.mbamupdates.com => 205.185.208.98
Releasing 0x0000000000349b40 (new refcount 1).
Found data-cdn.mbamupdates.com in host_name_addresses_map (0000000000349b40)
Connecting to data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)|205.185.208.98|:80... seconds 0.00, connected.
Created socket 4.
Releasing 0x0000000000349b40 (new refcount 1).

---request begin---
GET /web/mbam-setup-2.2.0.1024.exe HTTP/1.1

If-Modified-Since: Thu, 08 Oct 2015 22:41:33 GMT

User-Agent: Wget/1.17.1 (mingw32)

Accept: /

Accept-Encoding: identity

Host: data-cdn.mbamupdates.com

Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 304 Not Modified

Date: Tue, 15 Dec 2015 11:03:28 GMT

Accept-Ranges: bytes

ETag: "1444344092"

Content-Type: application/octet-stream

X-HW: 1450177409.dop008.la2.t,1450177408.cds012.la2.c

Content-Disposition: attachment; filename="mbam-setup-2.2.0.1024.exe"

---response end---
304 Not Modified
Disabling further reuse of socket 3.
Closed fd 3
Registered socket 4 for persistent reuse.
File 'mbam-setup-2.2.0.1024.exe' not modified on server. Omitting download.

Saving HSTS entries to C:\Tools/.wget-hsts

Shawn <shawnkhall>
Sat 05 Dec 2015 03:05:29 PM UTC, comment #1:

I could not reproduce the first example (...exe.html).
Are you sure you did not use -E / --adjust-extension ?
If I use that, the output looks like in your first example, else like in your second.

Please add -d to the wget options and send us that output for the first example.

Tim Ruehsen <rockdaboot>
Project AdministratorIn charge of this item.
Sat 05 Dec 2015 09:31:03 AM UTC, original submission:

There appears to be a bug with the log when using he --trust-server-names feature in 1.17. The behavior is inconsistent but the following command demonstrates it (sometimes):

# wget --timestamping --trust-server-names http://downloads.malwarebytes.org/file/mbam
--2015-12-05 01:16:01-- http://downloads.malwarebytes.org/file/mbam
Resolving downloads.malwarebytes.org (downloads.malwarebytes.org)... 93.184.216.111
Connecting to downloads.malwarebytes.org (downloads.malwarebytes.org)|93.184.216.111|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe [following]
--2015-12-05 01:16:02-- http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe
Resolving data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)... 93.184.215.124
Connecting to data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)|93.184.215.124|:80... connected.
HTTP request sent, awaiting response... 304 Not Modified
File 'mbam-setup-2.2.0.1024.exe.html' not modified on server. Omitting download.

# wget --timestamping --trust-server-names http://downloads.malwarebytes.org/file/mbam
--2015-12-05 01:23:54-- http://downloads.malwarebytes.org/file/mbam
Resolving downloads.malwarebytes.org (downloads.malwarebytes.org)... 93.184.216.111
Connecting to downloads.malwarebytes.org (downloads.malwarebytes.org)|93.184.216.111|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe [following]
--2015-12-05 01:23:54-- http://data-cdn.mbamupdates.com/web/mbam-setup-2.2.0.1024.exe
Resolving data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)... 205.185.208.98
Connecting to data-cdn.mbamupdates.com (data-cdn.mbamupdates.com)|205.185.208.98|:80... connected.
HTTP request sent, awaiting response... 304 Not Modified
File 'mbam-setup-2.2.0.1024.exe' not modified on server. Omitting download.

Specifically, the issue is in the last line indicating the name of the existing file. Sometimes there is an errant ".html" added to the filenames where it doesn't belong.

Shawn <shawnkhall>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by angel_ (Posted a comment)
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by rockdaboot (Posted a comment)
  • -unavailable- added by shawnkhall (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 16 Mar 2016 10:51:19 AM UTCrockdabootStatusNone=>Confirmed
      Assigned toNone=>rockdaboot
      Release1.17=>trunk

    Back to the top


    Powered by Savane 3.1-cleanup1