bugGNU Wget - Bugs: bug #55771, content-disposition not respected...


bug #55771: content-disposition not respected with long URLs (such as AWS)

Submitter:  None
Submitted:  Sat 23 Feb 2019 09:57:08 PM UTC
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  Inspected
Privacy:  Public Assigned to:  None
Originator Name:  shawnkhall Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.20
Operating System:  Microsoft Windows Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup

Mon 09 Mar 2020 11:15:58 PM UTC, comment #3: 

The issue persists in 1.20.3.

Here's the version response from wget 1.20.3 from eternallybored:

wget --version
GNU Wget 1.20.3 built on mingw32.

-cares +digest +gpgme +https +ipv6 +iri +large-file +metalink -nls
+ntlm +opie +psl +ssl/openssl

    C:\Tools/.wgetrc (user)
    /win32dev/misc/wget/out64/etc/wgetrc (system)
    x86_64-w64-mingw32-gcc -DHAVE_CONFIG_H
    -DLOCALEDIR="/win32dev/misc/wget/out64/share/locale" -I. -I../lib
    -I../lib -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DPCRE2_STATIC
    -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
    -I/win32dev/misc/wget/out64/include -DNDEBUG -g -O2
    x86_64-w64-mingw32-gcc -I/win32dev/misc/wget/out64/include
    -I/win32dev/misc/wget/out64/include -DPCRE2_STATIC
    -I/win32dev/misc/wget/out64/include -DHAVE_LIBSSL
    -I/win32dev/misc/wget/out64/include -DNDEBUG -g -O2
    -L/win32dev/misc/wget/out64/lib -L/win32dev/misc/wget/out64/lib
    -lmetalink -lunistring -liconv -L/win32dev/misc/wget/out64/lib
    -lpcre2-8 -lidn2 -L/win32dev/misc/wget/out64/lib -lssl -lcrypto
    -L/win32dev/misc/wget/out64/lib -lz -L/win32dev/misc/wget/out64/lib
    -lpsl -lws2_32 -lunistring -lws2_32 -lole32 -lcrypt32 -lexpat
    -L/win32dev/misc/wget/out64/lib -lgpgme -lassuan -lws2_32
    -lgpg-error ftp-opie.o mswindows.o openssl.o http-ntlm.o
    ../lib/libgnu.a -lws2_32 -lws2_32 -lws2_32 -lws2_32 -lws2_32
    /win32dev/misc/wget/out64/lib/libunistring.a -lws2_32

Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.

Shawn <shawnkhall>
Mon 09 Mar 2020 07:51:15 PM UTC, comment #2: 

The current version 1.20.3 saves 'rufus-3.4p.exe' in case 3. You are using an older Mingw32 version. Could you try the 1.20.3 version from https://eternallybored.org/misc/wget/ to make sure if it's a Mingw or a wget 1.20 issue !?

The cases 1+2 show an issue with long Location: headers that would create file names with more than 255 characters in length. As 255 seems to be your max file name length, wget tries to shorten these names too early. Wget also does not send Cookie: headers in the second HEAD request that are set in the 302 response before. This might be the reason for the 403 Forbidden.
This needs investigation.

GNU Wget2 (current master) does not have any of these issues. But i can't point you to an WIN32 executable.

Tim Ruehsen <rockdaboot>
Group administrator
Wed 04 Mar 2020 06:35:50 PM UTC, comment #1: 

This is a problem on linux too and it disables scripting wget to download github releases without specifying an output filename manually or using workaround "-e content_disposition=off". Tiny bit sad..

Marcel Partap <empee584>
Sat 23 Feb 2019 09:57:08 PM UTC, original submission:  

Using Wget 1.20.

The --content-disposition option is not respected with long URLs.

Expected: download the file to the name from the Content-Disposition header.

Result: downloaded the filename with 240 characters of garbage that does not include the traits from Content-Disposition or the end of the URL which may have been able to be used to correct the filename afterwards.

Issue: GitHub is more and more using AWS for backend storage and AWS uses very long URLs with ugly parameterized output. Sadly, Wget trims the output URL to compose the filename even when a Content-Disposition header is included and the --content-disposition option is enabled.

This commonly results in 500+ character url-encoded filenames that are stripped down to 400-something, but since this includes slashes and semicolons, the resulting filename isn't valid, or is 240 characters and the relevant filename portion to be able to correct after download has stripped from the saved filename.

Ideally it should respect the filename exposed within the content-disposition header in order to prevent this problem.

If that's not an option, it should perform the character removals to "correct" the filename from the beginning of the filename and not from the end, to preserve the extension (or at least parsable data that could be used to identify the extension). Offering a trim option to select which side of the URL should be stripped would be an acceptable workaround temporarily.

Example requests:

wget -N --debug -e content_disposition=on https://github.com/pbatard/rufus/releases/download/v3.4/rufus-3.4p.exe

Results in "response1.txt"

wget -N --debug --content-disposition https://github.com/pbatard/rufus/releases/download/v3.4/rufus-3.4p.exe

Results in "response2.txt"

When using --content-disposition, these also send a pre-test HEAD request which can trigger duplicate downloads for servers that do not properly respond to HEAD requests (though that's not the case here).

Exclusion of the --content-disposition parameter results in the file downloading but not with the correct name.

wget -N --debug https://github.com/pbatard/rufus/releases/download/v3.4/rufus-3.4p.exe

Results in "response3.txt"



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #46344:  response1.txt added by None (11KiB - text/plain - wget output for the three requests in the summary section)
file #46345:  response2.txt added by None (11KiB - text/plain - wget output for the three requests in the summary section)
file #46346:  response3.txt added by None (18KiB - text/plain - wget output for the three requests in the summary section)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by shawnkhall (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by empee584 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-03-09 rockdaboot StatusNone Inspected
    2019-02-23 None Attached File- Added response1.txt, #46344
        Attached File- Added response2.txt, #46345
        Attached File- Added response3.txt, #46346

    Back to the top

    Powered by Savane 3.12.
    Corresponding source code