bugGNU Wget - Bugs: bug #27957, wget -i can use up lots of ram if...


bug #27957: wget -i can use up lots of ram if the list is long enough

Submitted by:  None
Submitted on:  Fri 06 Nov 2009 12:37:53 PM UTC  
Category: Feature RequestSeverity: 3 - Normal
Priority: 3 - LowStatus: Needs Discussion
Privacy: PublicAssigned to: None
Originator Name: Jacques MattheijOriginator Email: -unavailable-
Open/Closed: OpenRelease: 1.12
Operating System: GNU/LinuxReproducibility: Every Time
Fixed Release: NonePlanned Release: None
Regression: NoneWork Required: None
Patch Included: None

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Tue 25 May 2010 03:23:31 PM UTC, comment #1:

if the file is too big, you can consider using split(1) in N files and execute wget N times.

It can be done with a rather simple script.

I don't think it worths to move this logic in the wget program.

Giuseppe Scrivano <gscrivano>
Project Administrator
Fri 06 Nov 2009 12:37:53 PM UTC, original submission:

This version of wget seems to load the full url list into memory when invoked with the -i option, which if the list is large enough causes serious problems. We found out about this during our rescue attempt of geocities, the seeds lists were very long.

c01-03:~# wget --version
GNU Wget 1.12 built on linux-gnu.

+digest +ipv6 +nls -ntlm +opie +md5/builtin -https -gnutls -openssl

/usr/local/etc/wgetrc (system)
Locale: /usr/local/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
-DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I ../md5 -O2
Link: gcc -O2 -Wall -ldl -lrt ftp-opie.o gen-md5.o ../lib/libgnu.a

Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Currently maintained by Micah Cowan <micah@cowan.name>.
Please send bug reports and questions to <bug-wget@gnu.org>.



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -unavailable- added by gscrivano (Posted a comment)
  • -unavailable- added by None (Submitted the item)

    Please enter the title of George Orwell's famous dystopian book (it's a date):



    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 29 May 2010 08:43:22 PM UTCgscrivanoCategoryNone=>Feature Request
      Priority5 - Normal=>3 - Low
    Tue 25 May 2010 03:23:31 PM UTCgscrivanoStatusNone=>Needs Discussion

    Back to the top

    Powered by Savane 3.1-cleanup1