bugGNU Wget - Bugs: bug #27957, wget -i can use up lots of ram if...

 
 

bug #27957: wget -i can use up lots of ram if the list is long enough

Submitter:  None
Submitted:  Fri 06 Nov 2009 12:37:53 PM UTC
   
 
Category:  Feature Request Severity:  3 - Normal
Priority:  3 - Low Status:  Needs Discussion
Privacy:  Public Assigned to:  None
Originator Name:  Jacques Mattheij Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.12
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 25 May 2010 03:23:31 PM UTC, comment #1: 

if the file is too big, you can consider using split(1) in N files and execute wget N times.

It can be done with a rather simple script.

I don't think it worths to move this logic in the wget program.

Giuseppe Scrivano <gscrivano>
Group administrator
Fri 06 Nov 2009 12:37:53 PM UTC, original submission:  

This version of wget seems to load the full url list into memory when invoked with the -i option, which if the list is large enough causes serious problems. We found out about this during our rescue attempt of geocities, the seeds lists were very long.

c01-03:~# wget --version
GNU Wget 1.12 built on linux-gnu.

+digest +ipv6 +nls -ntlm +opie +md5/builtin -https -gnutls -openssl
-iri

Wgetrc:
    /usr/local/etc/wgetrc (system)
Locale: /usr/local/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
    -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I ../md5 -O2
    -Wall
Link: gcc -O2 -Wall -ldl -lrt ftp-opie.o gen-md5.o ../lib/libgnu.a
    ../md5/libmd5.a

Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Currently maintained by Micah Cowan <micah@cowan.name>.
Please send bug reports and questions to <bug-wget@gnu.org>.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gscrivano (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    •  

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-05-29 gscrivano CategoryNone Feature Request
        Priority5 - Normal 3 - Low
    2010-05-25 gscrivano StatusNone Needs Discussion

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.13-397b.
    Corresponding source code