bugGNU Wget - Bugs: bug #41003, wget tries to reuse connection...

 
 

bug #41003: wget tries to reuse connection despite http/1.0 and no connection: keep-alive

Submitter:  Noël Köthe <nok>
Submitted:  Thu 26 Dec 2013 11:41:40 AM UTC
   
 
Category:  Protocol Issue Severity:  3 - Normal
Priority:  5 - Normal Status:  In Progress
Privacy:  Public Assigned to:  darnir
Originator Name:  Open/Closed:  Open
Release:  trunk Operating System:  GNU/Linux
Reproducibility:  None Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 17 Jan 2014 05:43:27 AM UTC, comment #3: 

Giuseppe, I have posted the --debug output in my previous comment.

As can be seen, Wget does indeed register the socket for persistent reuse, however, just 2 lines later it also closes the socket and creates a new one.

This is correct according to the HTTP protocol, a new socket was used for a new request, however in my opinion this is also a waste of machine cycles. First registering a socket and then deleting it.

Darshit Shah <darnir>
Group administrator
Fri 10 Jan 2014 12:20:00 PM UTC, comment #2: 

by inspecting the code, I see that wget doesn't check for the version of the HTTP protocol and it should assume no keep-alive connection by default when HTTP/1.0 is used.

Darshit, if you specify --debug, do you get the "Registered socket %d for persistent reuse.\n" message?

I was quickly testing with netcat as:

$ printf "HTTP/1.0 200 OK\r\nContent-Length: 0\r\n\r\n" | nc -l -p 8080

$ LANG=C ./wget -d -O /dev/null http://localhost:8080/ 2>&1  | grep Registered
Registered socket 4 for persistent reuse.


Giuseppe Scrivano <gscrivano>
Group administrator
Fri 27 Dec 2013 04:58:24 AM UTC, comment #1: 

This bug cannot be reproduced. I have set up the exact same environment as described in the bug report. A Python BaseHTTPServer with protocol_version = HTTP/1.0.

Wget does indeed close the connection after each request and creates a new connection. This is evidently seen in the debug output:

```
Running Test Test Debian #701032
/home/sauron/Programming/wget/src/wget -d --user=Sauron --password=TheEye 127.0.0.1:35404/File1
['/home/sauron/Programming/wget/src/wget', '-d', '--user=Sauron', '--password=TheEye', '127.0.0.1:35404/File1']
Setting --user (user) to Sauron
Setting --password (password) to TheEye
DEBUG output created by Wget 1.14.99-fb23 on linux-gnu.

URI encoding = ‘UTF-8’
--2013-12-27 10:15:23--  http://127.0.0.1:35404/File1
Host ‘127.0.0.1’ has not issued a general basic challenge.
Connecting to 127.0.0.1:35404... connected.
Created socket 3.
Releasing 0x0000000001859b80 (new refcount 0).
Deleting unused 0x0000000001859b80.

---request begin---
GET /File1 HTTP/1.1
User-Agent: Wget/1.14.99-fb23 (linux-gnu)
Accept: /
Host: 127.0.0.1:35404
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 127.0.0.1 - - [27/Dec/2013 10:15:23] "GET /File1 HTTP/1.1" 401 -
Unable to Authenticate

---response begin---
HTTP/1.0 401 Authorization Required
Server: BaseHTTP/0.6 Python/3.3.3
Date: Fri, 27 Dec 2013 04:45:23 GMT
WWW-Authenticate: Basic realm="Wget-Test"

---response end---
401 Authorization Required
Registered socket 3 for persistent reuse.
] done.
Inserted ‘127.0.0.1’ into basic_authed_hosts
Disabling further reuse of socket 3.
Closed fd 3
Connecting to 127.0.0.1:35404... connected.
Created socket 3.
Releasing 0x00000000018599e0 (new refcount 0).
Deleting unused 0x00000000018599e0.

---request begin---
GET /File1 HTTP/1.1
User-Agent: Wget/1.14.99-fb23 (linux-gnu)
Accept: /
Host: 127.0.0.1:35404
Connection: Keep-Alive
Authorization: Basic U2F1cm9uOlRoZUV5ZQ==

---request end---
HTTP request sent, awaiting response... 127.0.0.1 - - [27/Dec/2013 10:15:23] "GET /File1 HTTP/1.1" 200 -

---response begin---
HTTP/1.0 200 OK
Server: BaseHTTP/0.6 Python/3.3.3
Date: Fri, 27 Dec 2013 04:45:23 GMT
Content-type: text/plain
Content-Length: 24

---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 24 [text/plain]
Saving to: ‘File1’

100%[====================================================================================================================>] 24          --.-K/s   in 0s     

2013-12-27 10:15:23 (2.34 MB/s) - ‘File1’ saved [24/24]
```


Also, addressing a question about no auth string sent in the first request, that is the default behaviour of Wget for identification of the auth protocol. This can be overridden by using the switch --auth-no-challenge so it won't wait for a challenge from the server and will instead send the credentials in the first request itself.

Darshit Shah <darnir>
Group administrator
Thu 26 Dec 2013 11:41:40 AM UTC, original submission:  

Hello,

a forwarded Debian bug report from http://bugs.debian.org/701032

--8<--
   * What led up to the situation?
  
   i built a simple webserver with pythons BaseHTTPServer, which
   requires a basic http auth. the server answers in http 1.0 and
   sends no Connection: keep-alive header.
  
   Now i tried to test it with wget, which sent at first (even though
   i gave it user and password) no auth-string, probably to test which
   auth-scheme is used.
  
   the server sent a 401 with the WWW-Authenticate: Basic header, and NOW
  
   wget tried to reuse the connection, which is already closed, and
   should not have been reused, because there was no Content-length
   header, there was no Keep-alive header AND it was http 1.0, but it
   did, and the request failed. even though the server didn't do
   anything wrong.
  
   * What outcome did you expect instead?
  
   i expected one connection per request which would have worked. i
   didn't expect to read 'reusing existing connection' if there was
   known to be no keep-alive
--8<--

I requested a --debug output but didn't get an answer.

Thank you.

Noël Köthe <nok>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by darnir (Posted a comment)
  • -email is unavailable- added by nok
  • -email is unavailable- added by nok
  • -email is unavailable- added by nok (Submitted the item)
  •  

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-06-08 darnir StatusNone In Progress
        Assigned toNone darnir
    2013-12-26 nok Carbon-Copy- Added -email is unavailable-
        Carbon-Copy- Added moritz molle <mmolle@gmx.net>

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code