bugGNU Wget - Bugs: bug #55229, memory exhausted even though...

 
 

bug #55229: memory exhausted even though stating: Omitting download

Submitter:  Fenevad Kan <fenevadkan>
Submitted:  Sat 15 Dec 2018 09:28:03 AM UTC
   
 
Category:  Memory Leak Severity:  3 - Normal
Priority:  5 - Normal Status:  Fixed
Privacy:  Public Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  1.18 Operating System:  GNU/Linux
Reproducibility:  Every Time Fixed Release:  1.20
Planned Release:  None Regression:  None
Work Required:  None Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 18 Dec 2018 08:49:25 AM UTC, comment #13: 

Also thanks for your investigation and time.

Tim Ruehsen <rockdaboot>
Group administrator
Mon 17 Dec 2018 10:09:40 PM UTC, comment #12: 

sry, I used wrong optons with 1.10, I left -np there accidentely, with wich it worked okey anyway.
However the bug still seems to be fixed, with the original command it started scan further the whole site like crazy looking for new isos, after severa minutes it started to download /var/www/ISO/blackarch-linux-live-2018.12.01-x86_64.iso
And no more memory exhausted crash.

So thanks again, I think we can close it, I'm not sure I can close my own bug report (and if yes, how?) or you can only close it

Fenevad Kan <fenevadkan>
Mon 17 Dec 2018 09:56:57 PM UTC, comment #11: 

Ok, indeed, it worked with that.

Alright, testing 2 ways now, it seems it has been already fixed in 1.20:

Thanks for the attention and support.

1.18:
=======
$ wget -nd -m --continue http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/ -P /var/www/ISO/  -A "*.iso"

--2018-12-17 22:52:29--  http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/
Resolving quantum-mirror.hu (quantum-mirror.hu)... 78.131.56.189
Connecting to quantum-mirror.hu (quantum-mirror.hu)|78.131.56.189|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/var/www/ISO/index.html.tmp’

index.html.tmp          [ <=>             ]   2.21K  --.-KB/s    in 0s     

Last-modified header missing -- time-stamps turned off.
2018-12-17 22:52:29 (39.3 MB/s) - ‘/var/www/ISO/index.html.tmp’ saved [2264]

Loading robots.txt; please ignore errors.
--2018-12-17 22:52:29--  http://quantum-mirror.hu/robots.txt
Reusing existing connection to quantum-mirror.hu:80.
HTTP request sent, awaiting response... 404 Not Found
2018-12-17 22:52:29 ERROR 404: Not Found.

Removing /var/www/ISO/index.html.tmp since it should be rejected.

--2018-12-17 22:52:29--  http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/
Reusing existing connection to quantum-mirror.hu:80.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/var/www/ISO/index.html.tmp’

index.html.tmp          [ <=>             ]     329  --.-KB/s    in 0s     

Last-modified header missing -- time-stamps turned off.
2018-12-17 22:52:29 (10.9 MB/s) - ‘/var/www/ISO/index.html.tmp’ saved [329]

Removing /var/www/ISO/index.html.tmp since it should be rejected.

--2018-12-17 22:52:29--  http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-DVD-1810.iso
Reusing existing connection to quantum-mirror.hu:80.
HTTP request sent, awaiting response... 304 Not Modified
File ‘/var/www/ISO/CentOS-7-x86_64-DVD-1810.iso’ not modified on server. Omitting download.

wget: memory exhausted
#################################################################

1.20:
======
$ /root/wget/bin/wget -np -nd -m --continue http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/ -P /var/www/ISO/  -A "*.iso"

--2018-12-17 22:52:59--  http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/
Resolving quantum-mirror.hu... 78.131.56.189
Connecting to quantum-mirror.hu|78.131.56.189|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/var/www/ISO/index.html.tmp’

index.html.tmp          [ <=>             ]   2.21K  --.-KB/s    in 0s     

Last-modified header missing -- time-stamps turned off.
2018-12-17 22:53:00 (56.6 MB/s) - ‘/var/www/ISO/index.html.tmp’ saved [2264]

Loading robots.txt; please ignore errors.
--2018-12-17 22:53:00--  http://quantum-mirror.hu/robots.txt
Reusing existing connection to quantum-mirror.hu:80.
HTTP request sent, awaiting response... 404 Not Found
2018-12-17 22:53:00 ERROR 404: Not Found.

Removing /var/www/ISO/index.html.tmp since it should be rejected.

--2018-12-17 22:53:00--  http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-DVD-1810.iso
Reusing existing connection to quantum-mirror.hu:80.
HTTP request sent, awaiting response... 304 Not Modified
File ‘/var/www/ISO/CentOS-7-x86_64-DVD-1810.iso’ not modified on server. Omitting download.

--2018-12-17 22:53:00--  http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-Everything-1810.iso
Reusing existing connection to quantum-mirror.hu:80.
HTTP request sent, awaiting response... 304 Not Modified
File ‘/var/www/ISO/CentOS-7-x86_64-Everything-1810.iso’ not modified on server. Omitting download.

--2018-12-17 22:53:00--  http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-LiveGNOME-1810.iso
Reusing existing connection to quantum-mirror.hu:80.
HTTP request sent, awaiting response... 304 Not Modified
File ‘/var/www/ISO/CentOS-7-x86_64-LiveGNOME-1810.iso’ not modified on server. Omitting download.

--2018-12-17 22:53:00--  http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-LiveKDE-1810.iso
Reusing existing connection to quantum-mirror.hu:80.
HTTP request sent, awaiting response... 304 Not Modified
File ‘/var/www/ISO/CentOS-7-x86_64-LiveKDE-1810.iso’ not modified on server. Omitting download.

--2018-12-17 22:53:00--  http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-Minimal-1810.iso
Reusing existing connection to quantum-mirror.hu:80.
HTTP request sent, awaiting response... 304 Not Modified
File ‘/var/www/ISO/CentOS-7-x86_64-Minimal-1810.iso’ not modified on server. Omitting download.

--2018-12-17 22:53:00--  http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-NetInstall-1810.iso
Reusing existing connection to quantum-mirror.hu:80.
HTTP request sent, awaiting response... 304 Not Modified
File ‘/var/www/ISO/CentOS-7-x86_64-NetInstall-1810.iso’ not modified on server. Omitting download.

FINISHED --2018-12-17 22:53:00--
Total wall clock time: 0.7s
Downloaded: 1 files, 2.2K in 0s (56.6 MB/s)

Fenevad Kan <fenevadkan>
Mon 17 Dec 2018 08:05:23 PM UTC, comment #10: 

you need libgnutls28-dev to build wget with gnutls.

Noël Köthe <nok>
Mon 17 Dec 2018 04:27:06 PM UTC, comment #9: 

There is no package like that in the repo, only these:

apt search libgnutls:

Sorting... Done
Full Text Search... Done
libgnutls-dane0/stable,now 3.5.8-5+deb9u4 amd64 [installed,automatic]
  GNU TLS library - DANE security support

libgnutls-deb0-28/now 3.3.8-6+deb8u7 amd64 [installed,local]
  GNU TLS library - main runtime library

libgnutls-openssl27/stable,now 3.5.8-5+deb9u4 amd64 [installed]
  GNU TLS library - OpenSSL wrapper

libgnutls26/now 2.12.20-8+deb7u5 amd64 [installed,local]
  GNU TLS library - runtime library

libgnutls28-dev/stable 3.5.8-5+deb9u4 amd64
  GNU TLS library - development files

libgnutls30/stable,now 3.5.8-5+deb9u4 amd64 [installed,automatic]
  GNU TLS library - main runtime library

libgnutlsxx28/stable 3.5.8-5+deb9u4 amd64
  GNU TLS library - C++ runtime library

python-gnutls/stable 3.0.0-1 amd64
  Python wrapper for the GNUTLS library

Fenevad Kan <fenevadkan>
Mon 17 Dec 2018 05:54:54 AM UTC, comment #8: 

Copy-pasting the reply posted on ML:

> checking for GNUTLS... no
> configure: error: Package requirements (gnutls) were not met:
>
> No package 'gnutls' found 


I believe you need libgnutls-dev.
Also, please consider going through https://git.savannah.gnu.org/cgit/wget.git/tree/README.checkout

Avinash Sonawane <rootkea>
Sun 16 Dec 2018 09:44:57 PM UTC, comment #7: 

Installed, now a next issue with the config:

checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for LIBPSL... yes
checking for psl_latest... yes
checking for ZLIB... no
checking for compress in -lz... no
checking for GNUTLS... no
configure: error: Package requirements (gnutls) were not met:

No package 'gnutls' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables GNUTLS_CFLAGS
and GNUTLS_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.


And I cannot install that or dont know the name that would contain that, found a gnutls-bin in debian, installed it, but that didnt help, it sill miss gnutls

Fenevad Kan <fenevadkan>
Sun 16 Dec 2018 08:51:29 PM UTC, comment #6: 

Execute `apt-get install pkg-config` and try again.

If it doesn't, I'll test building on stretch tomorrow and let you know the details. Without dependency hell ;-)

Tim Ruehsen <rockdaboot>
Group administrator
Sun 16 Dec 2018 08:33:03 AM UTC, comment #5: 

I do not know how can I test it, debian strech does not have it in the repo.

I tried to use the latest one with these commands:

wget http://ftp.gnu.org/gnu/wget/wget-latest.tar.gz
tar -xzf wget-latest.tar.gz
cd wget-latest
./configure --prefix=$HOME
make && make install


but configure commands already dropped an error:

configure: error: The pkg-config script could not be found or is too old.  Make sure it
is in your PATH or set the PKG_CONFIG environment variable to the full
path to pkg-config.


But maybe I would have run into a dependency hell with it anyway?
Is there a latest debian repo that I could use?

Fenevad Kan <fenevadkan>
Sat 15 Dec 2018 07:51:46 PM UTC, comment #4: 

We fixed several memleaks since wget 1.18. Please try to reproduce with the latest version (1.20).

Maybe there is (or was ?) an issue that results in loading the last file (see your log) completely into memory. This is just a wild guess and not reproducible here.

So the first step would be to check that behavior with the latest version of wget

Tim Ruehsen <rockdaboot>
Group administrator
Sat 15 Dec 2018 03:20:58 PM UTC, comment #3: 

Hi Tim,
I tried -np option, it helps indeed in my case, but I guess it is still an issue, if I also want other ISOs from other than the current path.
I also checked --spider option you used, in that case the problem wont occur, so it must behave differently, since it does not really download anything, it probably wont buffer and use up all the memory


Fenevad Kan <fenevadkan>
Sat 15 Dec 2018 12:26:39 PM UTC, comment #2: 


$wget --spider -nd -m https://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/ -P ISO/ -np  -A "*.iso" 2>&1|grep iso$

--2018-12-15 13:22:27--  https://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-DVD-1810.iso
--2018-12-15 13:22:27--  https://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-Everything-1810.iso
--2018-12-15 13:22:27--  https://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-LiveGNOME-1810.iso
--2018-12-15 13:22:27--  https://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-LiveKDE-1810.iso
--2018-12-15 13:22:27--  https://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-Minimal-1810.iso
--2018-12-15 13:22:27--  https://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-NetInstall-1810.iso


Tim Ruehsen <rockdaboot>
Group administrator
Sat 15 Dec 2018 12:16:14 PM UTC, comment #1: 

Well, you download possibly everything from http://quantum-mirror.hu, at least all *.iso files.

As I understood, you are only interested in isos from .../7.6.1810/isos/x86_64/, then you should try -np/--no-parent. This makes wget to stay in that directory.

Tim Ruehsen <rockdaboot>
Group administrator
Sat 15 Dec 2018 09:28:03 AM UTC, original submission:  


Not a classic memory leak, but in practical result it eats up memory because of wrong behaviour. It can be reproduced with the same memory and command usage.
 
Host memory: free -tm:
              total        used        free      shared  buff/cache   available
Mem:           3955         655        2135          10        1164        3068
Swap:           713         405         308
Total:         4669        1060        2444

#lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.6 (stretch)
Release: 9.6
Codename: stretch



Real Use Case: trying to mirror/sync Centos ISO images with this command:

wget -nd -m --continue http://quantum-mirror.hu/mirrors/pub/centos/7.6.1810/isos/x86_64/ -P /var/www/ISO/  -A "*.iso"


That should download these files:
CentOS-7-x86_64-DVD-1810.iso        
CentOS-7-x86_64-Everything-1810.iso 
CentOS-7-x86_64-LiveGNOME-1810.iso  
CentOS-7-x86_64-LiveKDE-1810.iso    
CentOS-7-x86_64-Minimal-1810.iso    
CentOS-7-x86_64-NetInstall-1810.iso 

In the end it downloaded everything except the last one with Netinstall iso. From that point no matter how often I try to repeat the command to continue downloading it fails because of not enough memory even at checking phase when there is nothing to be downloaded just yet. As it walks through the older files and stating omitting the seems to stuck at the CentOS-7-x86_64-DVD-1810.iso file and I watch free memory in another console and see how it goes to 0 then the wget:memory exhausted message follows.

Here is the exact output with the additional debug (-d) option:
...attached as a file


wget --version:
GNU Wget 1.18 built on linux-gnu.

-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
+ntlm +opie +psl +ssl/gnutls

Wgetrc:
    /etc/wgetrc (system)
Locale:
    /usr/share/locale
Compile:
    gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
    -DLOCALEDIR="/usr/share/locale" -I. -I../../src -I../lib
    -I../../lib -Wdate-time -D_FORTIFY_SOURCE=2 -I/usr/include
    -I/usr/include/p11-kit-1 -DHAVE_LIBGNUTLS -DNDEBUG -g -O2
    -fdebug-prefix-map=/build/wget-N5RMXj/wget-1.18=.
    -fstack-protector-strong -Wformat -Werror=format-security
    -DNO_SSLv2 -D_FILE_OFFSET_BITS=64 -g -Wall
Link:
    gcc -I/usr/include/p11-kit-1 -DHAVE_LIBGNUTLS -DNDEBUG -g -O2
    -fdebug-prefix-map=/build/wget-N5RMXj/wget-1.18=.
    -fstack-protector-strong -Wformat -Werror=format-security
    -DNO_SSLv2 -D_FILE_OFFSET_BITS=64 -g -Wall -Wl,-z,relro -Wl,-z,now
    -L/usr/lib -lpcre -luuid -lnettle -lgnutls -lz -lpsl -lidn
    ftp-opie.o gnutls.o http-ntlm.o ../lib/libgnu.a




Fenevad Kan <fenevadkan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by nok (Posted a comment)
  • -email is unavailable- added by rootkea (Posted a comment)
  • -email is unavailable- added by rockdaboot (Posted a comment)
  • -email is unavailable- added by fenevadkan (Submitted the item)
  •  

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-12-18 rockdaboot StatusNone Fixed
        Open/ClosedOpen Closed
        Fixed ReleaseNone 1.20
    2018-12-15 fenevadkan Attached File- Added wget-memory-exhausted.txt, #45670

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code