bugGNU Wget - Bugs: bug #61038, Windows post fails to post files...

 
 

bug #61038: Windows post fails to post files over 65536 bytes,

Submitter:  None
Submitted:  Fri 13 Aug 2021 08:54:25 PM UTC
   
 
Category:  Program Logic Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Paul Breed Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.20
Operating System:  Microsoft Windows Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 25 Jan 2025 11:45:35 AM UTC, comment #14: 

Ie I can download files over 64 KiO, upload files below 64 KiO, but upload of files over 64 KiO fails (in the recent test with WGET2 140 KiO failed, whereas 40 KiO succeeded, and after switch back to CURL I succeeded to upload that 140 KiO file to very same server from very same local system, after minimal necessary adjustments to the commandline syntax).

Anonymous
Sat 25 Jan 2025 11:38:43 AM UTC, comment #13: 

Thanks and sorry for the late answer.

Tested binary compiled by Martin Guy (maintainer of SOX):

"http://martinwguy.net/test" ca 23 MiO EXE size

"wget2.exe" 23'439'910

Practical result: very same as with 1.25.0 compiled by Martin Guy and same as with previous 1.19.4 compiled by "eternallybored".

Details:

  • error code is 8 now (instead of 4)
  • verbosiy is much higher and very different (and full of private details)
  • it hangs two times during ca one minute


Report (private details removed):

(much activity here)

    24.214526.802 HTTP2 stream id 1
    24.214526.803 [0] action=1 pending=1 host=0x263fc00 goaway=false
    24.214526.806 [0] action=2 pending=1 host=0x263fc00 goaway=false
    24.214526.808   ##  pending_requests = 1
    24.214526.810 [FRAME 0] > SETTINGS
    24.214526.811 [FRAME 0] > WINDOW_UPDATE
    24.214526.812 [FRAME 1] > HEADERS
    24.214526.814 [FRAME 1] > :method: POST
    24.214526.815 [FRAME 1] > :path: /xxx.php
    24.214526.816 [FRAME 1] > :scheme: https
    24.214526.817 [FRAME 1] > :authority: test.mydomain.net
    24.214526.819 [FRAME 1] > accept-encoding: gzip, deflate
    24.214526.822 [FRAME 1] > accept: /
    24.214526.823 [FRAME 1] > user-agent: TEST
    24.214526.826 [FRAME 1] > referer: https://test.mydomain.net
    24.214526.828 [FRAME 1] > cookie: MuchPrivateData=15-Jan-2099
    24.214526.841 [FRAME 1] > content-type: multipart/form-data; boundary=XOOpWpspPUVnNVvYuYqVnwPRStKPKPmWOLXZ
    24.214526.845 [FRAME 1] > content-length: 140738

(hangs here for ca one minute, length above is OK)

    24.214626.853 [FRAME 0] < SETTINGS
    24.214626.859 [FRAME 0] < SETTINGS
    24.214626.862 [FRAME 0] > SETTINGS
    24.214626.866 [FRAME 1] < WINDOW_UPDATE
    24.214626.874 [FRAME 0] < WINDOW_UPDATE
    24.214626.879 [FRAME 1] < WINDOW_UPDATE
    24.214626.892 [FRAME 0] < WINDOW_UPDATE
    24.214626.896 [FRAME 1] < WINDOW_UPDATE
    24.214626.904 [FRAME 0] < WINDOW_UPDATE
    24.214626.912 [FRAME 1] < WINDOW_UPDATE
    24.214626.919 [FRAME 0] < WINDOW_UPDATE
    24.214626.921 [FRAME 1] < WINDOW_UPDATE
    24.214626.923 [FRAME 0] < WINDOW_UPDATE
    24.214626.926 [FRAME 1] < WINDOW_UPDATE
    24.214626.927 [FRAME 0] < WINDOW_UPDATE
    24.214626.929 :status: 503

(oh dear, did we get an answer from the server??)

    24.214626.933 date: Sat, 15 Jan 2099 11:47:33 GMT
    24.214626.938 server: Varnish
    24.214626.939 content-type: text/html; charset=utf-8
    24.214626.940 age: 0
    24.214626.940 x-cache: cp3067 int
    24.214626.941 x-cache-status: int-front
    24.214626.942 server-timing: cache;desc="int-front", host;desc="cp3067"
    24.214626.944 strict-transport-security: max-age=106384710; includeSubDomains; preload
    24.214626.946 report-to: { MUCH PRIVATE DATA IN JSON FORMAT
    24.214626.956 x-client-ip: 99.111.99.252
    24.214626.957 content-length: 2073

(this is length of what ??)

    24.214626.959 [FRAME 1] < HEADERS
    24.214626.960 [FRAME 1] < WINDOW_UPDATE
    24.214626.962 [FRAME 0] < WINDOW_UPDATE
    24.214626.964 [FRAME 1] < WINDOW_UPDATE
    24.214626.965 [FRAME 0] < WINDOW_UPDATE

(hangs here for ca one minute)

    24.214726.968 [FRAME 1] < RST_STREAM
    24.214726.975 closing stream 1
    24.214726.977   ##  response status 503
    HTTP ERROR response 503  https://test.mydomain.net/xxx.php
    24.214726.983 set_exit_status(8)

(now it's ultimately botched)

    24.214726.993 keep_alive=1
    24.214726.996 update HSTS test.mydomain.net:443 (maxage=106384710, includeSubDomains=1)
    24.214727.001 _host_remove_job: 0x263bfa8
    24.214727.003 host_remove_job: qsize=0 host->qsize=0
    24.214727.014 [0] action=1 pending=0 host=0x263fc00 goaway=false
    24.214727.019 closing connection
    24.214727.022 [0] action=1 pending=0 host=0x0 goaway=n/a
    24.214727.033 main: wake up
    24.214727.035 main: done

(much cookie activity online and locally)

    24.214727.127 Successfully updated 'X:/wget/.wget-hsts'.
    24.214727.130 Saved 1 HSTS entry into 'X:/wget/.wget-hsts'
    24.214727.133 blacklist https://test.mydomain.net/xxx.php

(why blacklist ??)

(result: error code 8)

The sever has obligatory HTTPS and I have been using "--no-check-certificate" all the time in all my tests in this thread.

Anonymous
Tue 07 Jan 2025 11:18:55 AM UTC, comment #12: 


> Bloated

That may be because it includes everything

OK, I've managed to build wget2 - might you try that?

If no luck I can try to build 1.11.4 but it failed at the first attempt

Under http://martinwguy.net/test again

    M

Martin Guy <martinwguy>
Thu 02 Jan 2025 07:26:20 PM UTC, comment #11: 

WGET binary compiled by Martin Guy as part of SOX:

"http://martinwguy.net/test" ca 18 MiO EXE size

"wget.exe" 18'711'399

Test results:

  • the binary is aburdly bloated and the size is

  even odd (usually it's a multiple of 256 O)

  • it works somewhat
  • the debug verbosity is very high (much higher than I am

  used to from CURL)

  • I can GET files
  • I can POST files below 8 KiO
  • I can POST files over 8 KiO but below 64 KiO (this

  failed with version 1.19.4)

  • POST:ing of files over 64 KiO reliably fails (this works with

  CURL, and this used to work with good old 1.11.4 before the
  servers started hating its crypto support)

This is what the debug verbosity reveals:

    ---request begin---
    POST /xxx.php HTTP/1.1
    Host: www.xxx.net
    Referer: https://www.xxx.net
    User-Agent: myagent
    Accept: /
    Accept-Encoding: identity
    Connection: Keep-Alive
    Content-Type: multipart/form-data; boundary=xzsyuzMLSrrmYVXvnqNZNKrmKWnRmLrTpMNx
    Content-Length: 81517
    Cookie: xxx

    ---request end---
    [writing BODY file MYHUGEBODY.TXT ... Retrying.

    --2025-01-03 09:52:56--  (try: 3)  https://www.xxx.net/xxx.php
    Found www.xxx.net in host_name_addresses_map (03d641d0)
    Connecting to www.xxx.net|188.13.52.214|:443... seconds 60.00, Winsock error: 0
    connected.
    Created socket 4.
    Releasing 0x03d641d0 (new refcount 1).

    ---request begin---
    POST /xxx.php HTTP/1.1
    Host: www.xxx.net
    Referer: https://www.xxx.net
    User-Agent: myagent
    Accept: /
    Accept-Encoding: identity
    Connection: Keep-Alive
    Content-Type: multipart/form-data; boundary=xzsyuzMLSrrmYVXvnqNZNKrmKWnRmLrTpMNx
    Content-Length: 81517
    Cookie: xxx

    ---request end---
    [writing BODY file MYHUGEBODY.TXT ... Giving up.

    Saving cookies to XXX.TXT.
    Done saving cookies.

The error code is 4. The interesting line is:

    [writing BODY file MYHUGEBODY.TXT ... Retrying.

and

    [writing BODY file MYHUGEBODY.TXT ... Giving up.

While the debug verbosity is very high most of the time, it seems
to temporarily sink to ZERO exactly at the moment when the writing
of the body fails. :-( Absolutely no clue what the problem is.

Anonymous
Fri 20 Dec 2024 09:51:00 PM UTC, comment #10: 

Hi! Could someone try the wget.exe I included last month in the big sox_ng ("win32") tarball at https://codeberg.org/sox_ng/sox_ng/releases/tag/sox_ng-14.5.0.1-rc1

It was cross-compiled from Debian using MXE - not that SoX needs to POST, only GET, but trying it may indicate whether the problem is compiler-dependent or not.

Version: GNU Wget 1.25.0 built on mingw32.static

    M

Martin Guy <martinwguy>
Sun 12 Nov 2023 09:03:51 AM UTC, comment #9: 

I got the broken binaries from the only place available:

https://eternallybored.org/misc/wget/1.21.4/32/wget.exe

There is a new version:

https://github.com/rockdaboot/wget2/releases/download/v2.1.0/wget2.exe

2'386'432, "stripped", UPX:ed

unfortunately only 64-bit. Not YET tested.

Anonymous
Mon 31 Jan 2022 09:04:47 PM UTC, comment #8: 

Hi,

I successfully reproduce the problem with wget 1.21.2 on Windows which is build with gcc/mingw 9.3

I wrote a simple server app that accepts post data and return HTTP 200 if expected file size equals to actual one and return 400 if somethings goes wrong.

I tested the app on two different Window 10 desktop PC and works without error even for 1GB file uploads.

Then I published the app to a GitHub and test the app on GitHub Actions and tests failed after 16KB. wget uploads 16KB successfully but failed after 20th try for 256KB file.

Please see the logs from this url: https://github.com/webfolderio/wget-file-upload-test/runs/5012162406?check_suite_focus=true

I the tested server app on a public VPS and executed the tests and faced with the same results similar to GitHub actions test.

Similar to GitHub actions, wget fails to upload 256KB file after lots of try.

I attached the Wireshark capture data for the VPS test.
Hope that it helps to find the bug.

reference wget executable for tests: https://github.com/webfolderio/wget-windows

test scenarios:
localhost (wget) - localhost (server) - DESKTOP - OK
localhost (wget) - localhost (server) - GitHub Actions - 256KB FAILED
localhost (wget) - 0.0.0.0 (server)   - VPS Server - 256KB FAILED


(file #52783)

Ozhan Duz <ozhanduz>
Fri 21 Jan 2022 12:29:28 PM UTC, comment #7: 

On the first glimpse, https://github.com/rockdaboot/wget2/issues/256 is unrelated.

But then it seems that the poll/select function on Windows sometimes reports "ready to read" but the next read returns 0 bytes read. (Wget2 used non-blocking I/O).

Wget uses blocking I/O - but that doesn't matter. I assume that a read also sometimes reports 0 bytes.

Now this issue is about POST, so mainly write() functionality. I assume we might see the same or similar thing here as well, that write()/send() returns 0 and wget stops.

I never saw this behavior on native Linux nor with Wine emulation.
Also, testing on my wife's Win11 laptop never showed this behavior.

The aforementioned GH issue 256 contains two links to a statically wget2 binary. Please feel free to test with these binaries in order to reproduce the issue and report on that bug if possible.
More information may lead to a fix in gnulib which also affects wget1.x (just needs a recompilation then).

Tim Ruehsen <rockdaboot>
Group administrator
Mon 06 Dec 2021 03:33:57 PM UTC, comment #6: 

https://eternallybored.org/misc/wget

https://eternallybored.org/misc/wget/releases/wget-1.19.4-win32.zip

I tested many binaries from there and all are broken, most notably 1.19.4 and 1.21.1 , and IIRC also some older 1.17.xx and 1.15.xx .

Anonymous
Wed 01 Dec 2021 10:01:38 PM UTC, comment #5: 

Unfortunately, I am unable to reproduce this on Linux.

This seems to be a Windows only issue. Could you please mention where you got your Windows binaries from? Maybe raise an issue with them about it as well.

Darshit Shah <darnir>
Group administrator
Tue 05 Oct 2021 04:05:08 PM UTC, comment #4: 

1.21.2 is out ... and 2.0.0 out ... and reported to be broken. :-(

To   : -email is unavailable-
Subj : BUG in recent versions of WGET with POSTing more than 8 KiO data

> You are welcome to submit bug reports via the GNU Wget bug
> tracker (see <https://savannah.gnu.org/bugs/?func=additem&group=wget>)
> or to our mailing list <bug-wget@gnu.org>.
> Visit <https://lists.gnu.org/mailman/listinfo/bug-wget> to get
> more info (how to subscribe, list archives, ...).


There is (most likely) a severe bug in recent versions of WGET,
including 1.19.4 and 1.21.1, but not in good old 1.11.4 from
year 2008.

Expected behaviour: POSTing will work irrespective size of the postdata

Observed behaviour: POSTing fails with exit code 4 if size of the
                    postdata exceeds ca 8 KiO

With WGET 1.11.4 I could POST data irrespective size, only
percent-encoding tested at that time.

With CURL I can POST data irrespective size, both percent-encoded
and multipart-encoded.

With WGET 1.19.4 and 1.21.1 I can post data of limited size, both
percent-encoded and multipart-encoded.

With WGET 1.19.4 and 1.21.1 posting reliably fails if size of the
postdata exceeds ca 8 KiO.

I do the encoding myself and run WGET and CURL in a pretty raw way,
not caring about the fact that CURL can multipart-encode whereas
WGET can't.

Using the Win32 versions of WGET, Linux not installed.

Result from "--debug" success (exitcode is 0 and data arrives
at the server):

> ---request end---
> [writing BODY file YPOST.TMP ... done]
> HTTP request sent, awaiting response... seconds 60.00, Winsock error: 0
> seconds 60.00, Winsock error: 0
> seconds 60.00, Winsock error: 0


> ---response begin---
> HTTP/1.1 200 OK


Result from "--debug" failure (exitcode is 4 after 3 attempts
and data does not arrive at the server):

> ---request end---
> [writing BODY file YPOST.TMP ... Closed 4/SSL 0x009fb810
> Giving up.


> Saving cookies to YCOOK.TMP.
> Done saving cookies.


The report is not very verbose, "Closed" is the only hint about the
problem. The file YPOST.TMP contains the postdata and is good in
both cases, but bigger in the lower one.

Of course I can use CURL instead of WGET, but it's good to have
options, and bad to have bugs, given that this probably is a bug
of WGET. I am aware that CURL is more intended for posting than WGET,
but this used to work with old versions (1.11.4) before they got
unusable due to 3rd party hard cryptographic deprecation.

Anonymous
Tue 17 Aug 2021 12:56:45 PM UTC, comment #3: 
Anonymous
Sun 15 Aug 2021 01:07:05 PM UTC, comment #2: 

Since this is a Windows issue, could you please report this issue to where you got the binary from ?

Please feel free to update this issue with any information you get from there.

Tim Ruehsen <rockdaboot>
Group administrator
Fri 13 Aug 2021 09:05:54 PM UTC, comment #1: 

Confirmed this bug does not exist on a mac with version 1.20 It is a windows only issue.

Anonymous
Fri 13 Aug 2021 08:54:25 PM UTC, original submission:  


If you post a file > 65K the system sends (Size % 65K) of the file then closes the socket.

Version 1.11 works correctly. I've tried 1.19,1.20 and 1.21 and all three fail.

I've attached  --debug logs of working 1.11 and failing 1.20

Exact command line:

wget --post-file=obj\Release\SimpleHtml.bin 10.1.1.68:20034/appupdate.htm


I created a simple post target that doe snothing on the target.
And tried sending files of various sizes and any file < 16bits works and any file > 16 bits fails.

Have not yet tested oin Linux and Mac, suspect the issue is probably in the windows networking adapter layer.

Can also send pcaps of these failures if you want them.





Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #52783:  wget-file-upload-test-capture.pcapng.gz added by ozhanduz (17KiB - application/x-gzip - wget windows file upload wireshark capture)
file #51773:  Works_Log_1P11.txt added by None (1001B - text/plain)
file #51774:  Fails_Log_1P20.txt added by None (3KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ozhanduz (Updated the item)
  • -email is unavailable- added by darnir (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-01-31 ozhanduz Attached File- Added wget-file-upload-test-capture.pcapng.gz, #52783
    2021-08-13 None Attached File- Added Works_Log_1P11.txt, #51773
        Attached File- Added Fails_Log_1P20.txt, #51774

    Back to the top

    Powered by Savane 3.14-9aa3.
    Corresponding source code