bugGNU Wget - Bugs: bug #50579, wget --continue: Download fails...

 
 

bug #50579: wget --continue: Download fails when server does not support HEAD request

Submitter:  Noël Köthe <nok>
Submitted:  Sat 18 Mar 2017 06:07:36 PM UTC
   
 
Category:  Feature Request Severity:  3 - Normal
Priority:  5 - Normal Status:  Wont Fix
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  None Operating System:  None
Reproducibility:  None Fixed Release:  None
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 24 Mar 2017 08:19:17 AM UTC, comment #3: 

As Dale mentions, what Wget does is exactly according to the RFC. In fact it is the server that breaks the spec. And a workaround for this server may cause breakages for other servers that actually follow the specifications. As a result, I vote that we close this as a non-issue

Darshit Shah <darnir>
Group administrator
Fri 24 Mar 2017 01:34:05 AM UTC, comment #2: 

RFC 2616 section 9.4 says that an HTTP server should return the same headers for a HEAD request as it does for a GET of the same URL.  So I see nothing incorrect with wget relying on that behavior.

Dale Worley <worley>
Sat 18 Mar 2017 06:40:57 PM UTC, comment #1: 

As you already stated, the server should have answered the HEAD request properly. If a server refuses to do so, --continue currently doesn't work.

Especially with --content-disposition, Wget doesn't know about the filename to look/save at, so it simply gives up here.

If now Wget tries to be 'clever' and (silently) falls back to the filename derived from the URL, it could be that the wrong file becomes saved, clobbered, destroyed (depending on additional options).

So IMO, we better leave it up to the user to exactly say what (s)he wants.

The title is misleading, the problem is the same wether with or without trailing slash.

I leave this bug open for discussion and mark it feature request.

Tim Ruehsen <rockdaboot>
Group administrator
Sat 18 Mar 2017 06:07:36 PM UTC, original submission:  

Hello,

a forwarded bug report from a Debian user https://bugs.debian.org/843620

--8<--
It would be nice if the behavior of --continue would also handle
corner cases, such as the following:

$ wget --content-disposition --continue
http://www.mckesson.com/documents/providers/mckesson-radiology-12-dicom-conformance-statement/
--2016-11-08 12:48:07--
http://www.mckesson.com/documents/providers/mckesson-radiology-12-dicom-conformance-statement/
Resolving www.mckesson.com (www.mckesson.com)... 2.20.250.32, 2.20.250.41
Connecting to www.mckesson.com (www.mckesson.com)|2.20.250.32|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2016-11-08 12:48:08 ERROR 503: Service Unavailable.

On the other hand this one works OK:

$ wget --content-disposition --continue
http://www3.gehealthcare.com/~/media/documents/us-global/products/interoperability/dicom/nuclear-medicine/gehc-dicom-conformance_ventri_5179057dhf_rev1.pdf
--2016-11-08 12:49:25--
http://www3.gehealthcare.com/~/media/documents/us-global/products/interoperability/dicom/nuclear-medicine/gehc-dicom-conformance_ventri_5179057dhf_rev1.pdf
Resolving www3.gehealthcare.com (www3.gehealthcare.com)...
2.20.250.19, 2.20.250.10
Connecting to www3.gehealthcare.com
(www3.gehealthcare.com)|2.20.250.19|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 709762 (693K) [application/pdf]
--2016-11-08 12:49:25--
http://www3.gehealthcare.com/~/media/documents/us-global/products/interoperability/dicom/nuclear-medicine/gehc-dicom-conformance_ventri_5179057dhf_rev1.pdf
Reusing existing connection to www3.gehealthcare.com:80.
HTTP request sent, awaiting response... 206 Partial Content
Length: 709762 (693K), 0 remaining [application/pdf]
Saving to: ‘GEHC-DICOM-Conformance_Ventri_5179057DHF_Rev1.pdf’

GEHC-DICOM-Conformance_Ventri_5179057DHF_Re
100%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++]
693.13K  --.-KB/s   in 0s

2016-11-08 12:49:26 (0.00 B/s) -
‘GEHC-DICOM-Conformance_Ventri_5179057DHF_Rev1.pdf’ saved
[709762/709762]


For some reason HEAD would not get the Content-Disposition tag, but
looking at curl output I can see it:

  • Hostname was NOT found in DNS cache

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:--
--:--:--     0*   Trying 2.20.250.32...

> GET /documents/providers/mckesson-radiology-12-dicom-conformance-statement/ HTTP/1.1
> User-Agent: curl/7.38.0
> Host: www.mckesson.com
> Accept: /
>

  0     0    0     0    0     0      0      0 --:--:-- --:--:--
--:--:--     0< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Length: 3292374
< Content-Type: application/pdf

  • Server Microsoft-IIS/7.5 is not blacklisted

< Server: Microsoft-IIS/7.5
< Content-Disposition: inline; filename="McKesson Radiology 12.1.1
DICOM Conformance Statement rev1.0.pdf"
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
< X-UA-Compatible: IE=edge,chrome=1
< Date: Tue, 08 Nov 2016 11:45:47 GMT
< Connection: keep-alive
< Set-Cookie: EktGUID=3f3d7804-163d-4a59-bbf4-676abf1a1dab;
expires=Wed, 08-Nov-2017 11:45:46 GMT; path=/
< Set-Cookie: EkAnalytics=0; expires=Wed, 08-Nov-2017 11:45:46 GMT; path=/
< Set-Cookie: ASP.NET_SessionId=nprzkvb3mnsu41ahuslx1auy; path=/; HttpOnly
< Set-Cookie: ecm=user_id=0&isMembershipUser=0&site_id=&username=&new_site=/&unique_id=0&site_preview=0&langvalue=0&DefaultLanguage=1033&NavLanguage=1033&LastValidLanguageID=1033&DefaultCurrency=840&SiteCurrency=840&ContType=&UserCulture=1033&dm=www.mckesson.com&SiteLanguage=1033;
path=/
< Set-Cookie: BIGipServerMCKESSON.COM_80=801417408.20480.0000;
expires=Tue, 08-Nov-2016 12:05:46 GMT; path=/
<
{ [data not shown]

Noël Köthe <nok>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by darnir (Posted a comment)
  • -email is unavailable- added by worley (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by nok (Submitted the item)
  • -email is unavailable- added by nok
  •  

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-03-24 rockdaboot StatusNeeds Discussion Wont Fix
        Open/ClosedOpen Closed
    2017-03-18 rockdaboot CategoryProgram Logic Feature Request
        StatusNone Needs Discussion
        Summarywget --continue: URL with trailing slash '/' (but Content-Disposition) wget --continue: Download fails when server does not support HEAD request
    2017-03-18 nok Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code