bugGNU Wget - Bugs: bug #27957, wget -i can use up lots of ram if...


bug #27957: wget -i can use up lots of ram if the list is long enough

Submitted by:  None
Submitted on:  Fri 06 Nov 2009 12:37:53 PM UTC  
Category:  Feature Request Severity:  3 - Normal
Priority:  3 - Low Status:  Needs Discussion
Privacy:  Public Assigned to:  None
Originator Name:  Jacques Mattheij Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.12
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Tue 25 May 2010 03:23:31 PM UTC, comment #1:

if the file is too big, you can consider using split(1) in N files and execute wget N times.

It can be done with a rather simple script.

I don't think it worths to move this logic in the wget program.

Giuseppe Scrivano <gscrivano>
Project Administrator
Fri 06 Nov 2009 12:37:53 PM UTC, original submission:

This version of wget seems to load the full url list into memory when invoked with the -i option, which if the list is large enough causes serious problems. We found out about this during our rescue attempt of geocities, the seeds lists were very long.

c01-03:~# wget --version
GNU Wget 1.12 built on linux-gnu.

+digest +ipv6 +nls -ntlm +opie +md5/builtin -https -gnutls -openssl

/usr/local/etc/wgetrc (system)
Locale: /usr/local/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
-DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I ../md5 -O2
Link: gcc -O2 -Wall -ldl -lrt ftp-opie.o gen-md5.o ../lib/libgnu.a

Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Currently maintained by Micah Cowan <micah@cowan.name>.
Please send bug reports and questions to <bug-wget@gnu.org>.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)



    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-05-29 gscrivano CategoryNone => Feature Request
        Priority5 - Normal => 3 - Low
    2010-05-25 gscrivano StatusNone => Needs Discussion

    Back to the top

    Powered by Savane 3.3