bugGNU Wget - Bugs: bug #41003, wget tries to reuse connection...

 
 

bug #41003: wget tries to reuse connection despite http/1.0 and no connection: keep-alive

Submitted by:  Noël Köthe <nok>
Submitted on:  Thu 26 Dec 2013 11:41:40 AM UTC  
 
Category: Protocol IssueSeverity: 3 - Normal
Priority: 5 - NormalStatus: In Progress
Privacy: PublicAssigned to: Darshit Shah <darnir>
Originator Name: Open/Closed: Open
Release: trunkOperating System: GNU/Linux
Reproducibility: NoneFixed Release: None
Planned Release: NoneRegression: None
Work Required: NonePatch Included: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Fri 17 Jan 2014 05:43:27 AM UTC, comment #3:

Giuseppe, I have posted the --debug output in my previous comment.

As can be seen, Wget does indeed register the socket for persistent reuse, however, just 2 lines later it also closes the socket and creates a new one.

This is correct according to the HTTP protocol, a new socket was used for a new request, however in my opinion this is also a waste of machine cycles. First registering a socket and then deleting it.

Darshit Shah <darnir>
Project AdministratorIn charge of this item.
Fri 10 Jan 2014 12:20:00 PM UTC, comment #2:

by inspecting the code, I see that wget doesn't check for the version of the HTTP protocol and it should assume no keep-alive connection by default when HTTP/1.0 is used.

Darshit, if you specify --debug, do you get the "Registered socket %d for persistent reuse.\n" message?

I was quickly testing with netcat as:

$ printf "HTTP/1.0 200 OK\r\nContent-Length: 0\r\n\r\n" | nc -l -p 8080

$ LANG=C ./wget -d -O /dev/null http://localhost:8080/ 2>&1 | grep Registered
Registered socket 4 for persistent reuse.

Giuseppe Scrivano <gscrivano>
Project Administrator
Fri 27 Dec 2013 04:58:24 AM UTC, comment #1:

This bug cannot be reproduced. I have set up the exact same environment as described in the bug report. A Python BaseHTTPServer with protocol_version = HTTP/1.0.

Wget does indeed close the connection after each request and creates a new connection. This is evidently seen in the debug output:

```
Running Test Test Debian #701032
/home/sauron/Programming/wget/src/wget -d --user=Sauron --password=TheEye 127.0.0.1:35404/File1
['/home/sauron/Programming/wget/src/wget', '-d', '--user=Sauron', '--password=TheEye', '127.0.0.1:35404/File1']
Setting --user (user) to Sauron
Setting --password (password) to TheEye
DEBUG output created by Wget 1.14.99-fb23 on linux-gnu.

URI encoding = ‘UTF-8’
--2013-12-27 10:15:23-- http://127.0.0.1:35404/File1
Host ‘127.0.0.1’ has not issued a general basic challenge.
Connecting to 127.0.0.1:35404... connected.
Created socket 3.
Releasing 0x0000000001859b80 (new refcount 0).
Deleting unused 0x0000000001859b80.

---request begin---
GET /File1 HTTP/1.1
User-Agent: Wget/1.14.99-fb23 (linux-gnu)
Accept: /
Host: 127.0.0.1:35404
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 127.0.0.1 - - [27/Dec/2013 10:15:23] "GET /File1 HTTP/1.1" 401 -
Unable to Authenticate

---response begin---
HTTP/1.0 401 Authorization Required
Server: BaseHTTP/0.6 Python/3.3.3
Date: Fri, 27 Dec 2013 04:45:23 GMT
WWW-Authenticate: Basic realm="Wget-Test"

---response end---
401 Authorization Required
Registered socket 3 for persistent reuse.
] done.
Inserted ‘127.0.0.1’ into basic_authed_hosts
Disabling further reuse of socket 3.
Closed fd 3
Connecting to 127.0.0.1:35404... connected.
Created socket 3.
Releasing 0x00000000018599e0 (new refcount 0).
Deleting unused 0x00000000018599e0.

---request begin---
GET /File1 HTTP/1.1
User-Agent: Wget/1.14.99-fb23 (linux-gnu)
Accept: /
Host: 127.0.0.1:35404
Connection: Keep-Alive
Authorization: Basic U2F1cm9uOlRoZUV5ZQ==

---request end---
HTTP request sent, awaiting response... 127.0.0.1 - - [27/Dec/2013 10:15:23] "GET /File1 HTTP/1.1" 200 -

---response begin---
HTTP/1.0 200 OK
Server: BaseHTTP/0.6 Python/3.3.3
Date: Fri, 27 Dec 2013 04:45:23 GMT
Content-type: text/plain
Content-Length: 24

---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 24 [text/plain]
Saving to: ‘File1’

100%[====================================================================================================================>] 24 --.-K/s in 0s

2013-12-27 10:15:23 (2.34 MB/s) - ‘File1’ saved [24/24]
```

Also, addressing a question about no auth string sent in the first request, that is the default behaviour of Wget for identification of the auth protocol. This can be overridden by using the switch --auth-no-challenge so it won't wait for a challenge from the server and will instead send the credentials in the first request itself.

Darshit Shah <darnir>
Project AdministratorIn charge of this item.
Thu 26 Dec 2013 11:41:40 AM UTC, original submission:

Hello,

a forwarded Debian bug report from http://bugs.debian.org/701032

--8<--
* What led up to the situation?

i built a simple webserver with pythons BaseHTTPServer, which
requires a basic http auth. the server answers in http 1.0 and
sends no Connection: keep-alive header.

Now i tried to test it with wget, which sent at first (even though
i gave it user and password) no auth-string, probably to test which
auth-scheme is used.

the server sent a 401 with the WWW-Authenticate: Basic header, and NOW

wget tried to reuse the connection, which is already closed, and
should not have been reused, because there was no Content-length
header, there was no Keep-alive header AND it was http 1.0, but it
did, and the request failed. even though the server didn't do
anything wrong.

* What outcome did you expect instead?

i expected one connection per request which would have worked. i
didn't expect to read 'reusing existing connection' if there was
known to be no keep-alive
--8<--

I requested a --debug output but didn't get an answer.

Thank you.

Noël Köthe <nok>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by darnir (Posted a comment)
  • -unavailable- added by nok
  • -unavailable- added by nok
  • -unavailable- added by nok (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 08 Jun 2014 02:49:30 PM UTCdarnirStatusNone=>In Progress
      Assigned toNone=>darnir
    Thu 26 Dec 2013 11:41:40 AM UTCnokCarbon-Copy-=>Added -unavailable-
      Carbon-Copy-=>Added moritz molle

    Back to the top


    Powered by Savane 3.1-cleanup