bugGNU Wget - Bugs: bug #60106, wget mistakenly interprets...


bug #60106: wget mistakenly interprets Content-Range as bytes and truncates content

Submitter:  Andreas Franke <afranke>
Submitted:  Tue 23 Feb 2021 07:10:59 PM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Confirmed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  1.19.4 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  None
Planned Release:  None Regression:  No
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup

Jump to the original submission

Sat 13 Mar 2021 05:58:15 PM UTC, comment #6: 

I love to see that wget2 downloas the 4027 bytes and saves it as 'submission' ;-)

Tim Ruehsen <rockdaboot>
Group administrator
Sat 13 Mar 2021 05:53:21 PM UTC, comment #5: 

Thanks for reporting this.

Just skimmed the specs ... well, the server is special in several ways and seems to interpret standards "in his own way".
I'd even say it's simply buggy and should be fixed.
(E.g. the "Content-Range:" format is simply wrong regarding https://tools.ietf.org/html/rfc7233#section-4.2).

Anyways, wget should simply ignore the broken and the undefined HTTP headers and save the response body. The Content-Length header gives all the information needed (4027 bytes in the example below). I am sure that curl does exactly that.

The unregistered (private) Range-Unit header is already ignored by wget, so it has nothing to do with this issue. The title of this issue was possibly a bit misleading, so I'll amend it.

I'll reopen the bug, as IMO we should fix the behavior of wget here.

HTTP/1.1 206 Partial Content
Accept-Ranges: items
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Range, Accept-Ranges, Range-Unit
Cache-Control: no-cache,max-age=0
Content-Range: 0-9/25
Content-Type: application/json; charset=utf-8
Date: Sat, 13 Mar 2021 17:35:04 GMT
ETag: W/"fbb-DQKITUXXr7c2bHQ0GXXQJaUxUwg"
Link: </form/5b8c1017edc1a6c05601af8e/submission>; rel="next"; items="10-34", </form/5b8c1017edc1a6c05601af8e/submission>; rel="last"; items="20-44"
Range-Unit: items
Server: nginx/1.16.1
Vary: Origin
X-Powered-By: Express
Content-Length: 4027
Connection: keep-alive

Tim Ruehsen <rockdaboot>
Group administrator
Mon 08 Mar 2021 02:00:48 AM UTC, comment #4: 

(Just to clarify: Our current workaround for scripts accessing real-world form.io based sites is to use curl instead of wget. In effect, you are suggesting to make this workaround permanent.)

Andreas Franke <afranke>
Mon 08 Mar 2021 01:44:13 AM UTC, comment #3: 

So you'd say 'curl' is wrong by not doing the same as 'wget' here?

And what about this aspect:

> wget then retries with a range request for
> the remaining content using a bytes range,
> even though the server just stated it does
> NOT support bytes ranges, but only items ranges.

Andreas Franke <afranke>
Sun 07 Mar 2021 10:48:07 PM UTC, comment #2: 


As far as I can tell, the "Range-Unit" header is not mentioned in any of the HTTP/1.1 Specifications (RFC 7230-7237). The correct way to specify the range unit would be to add the unit in the "content-range" header itself. This is very clearly outlined in both RFC2616, the original HTTP/1.1 specification and in RFC7233 the latest version of HTTP/1.1. Please refer to the ABNF provided there to create valid HTTP responses that comform to the specification.

By the specification, if the unit is not specified, the client is allowed to interpret it as Bytes, which is exactly what Wget does.

Darshit Shah <darnir>
Group administrator
Tue 23 Feb 2021 07:27:08 PM UTC, comment #1: 

The problem occurs even when a valid 'Content-Length' header is present in the response. For example, GET on this URL returns something like 10 'company' items:

Andreas Franke <afranke>
Tue 23 Feb 2021 07:10:59 PM UTC, original submission:  

When a server sends the following response headers:

  Accept-Ranges: items
  Range-Unit: items
  Content-Range: 0-110/111

but no 'Content-Length' response header, then wget says

Length: 111 [...]

and truncates the response content after 111 bytes (or, when using https, after a few KB), even though the full response content is much larger (e.g. 720 KB).

What's worse, --ignore-length doesn't help with this at all.

While a range unit of "bytes" is the most common case, IMO wget should not assume a range unit of "bytes" when the 'Range-Unit' header says something else.

To aggravate the problem even further, wget then retries with a range request for the remaining content using a bytes range, even though the server just stated it does NOT support bytes ranges, but only items ranges.  (This latest problem can be avoided by disallowing retries with the -t1 option.)

This issue can occur e.g. with a REST api providing lists of items as application/json content.

Andreas Franke <afranke>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by darnir (Posted a comment)
  • -email is unavailable- added by afranke (Submitted the item)

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-03-13 rockdaboot Summarywget mistakenly interprets Content-Range as bytes, even with Range-Unit: items, and truncates content wget mistakenly interprets Content-Range as bytes and truncates content
    2021-03-13 rockdaboot StatusInvalid Confirmed
        Open/ClosedClosed Open
    2021-03-07 darnir StatusNone Invalid
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.12