bugGNU Wget - Bugs: bug #37030, When I run wget with wget -qci...

 
 

bug #37030: When I run wget with wget -qci <inputfile> it eats memory fast 3GB in 10 minutes

Submitter:  None
Submitted:  Sun 05 Aug 2012 02:09:29 PM UTC
   
 
Category:  Memory Leak Severity:  3 - Normal
Priority:  5 - Normal Status:  None
Privacy:  Public Assigned to:  None
Originator Name:  Marko Pavlovic Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  1.13.4
Operating System:  GNU/Linux Reproducibility:  Every Time
Fixed Release:  None Planned Release:  None
Regression:  None Work Required:  None
Patch Included:  No
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 04 Sep 2012 06:45:31 PM UTC, comment #1: 

I'm getting the same thing.  I'm using version 1.13.4 on Windows, from cygwin.  Here's my command line:

wget -E -H -k -p -np --mirror -R txt,jpg,gif -i urls.txt

The file urls.txt contains URLs to sec.gov.  I'm attaching the file so someone can reproduce the condition.

Wget will run for several hours (somewhere around 6 hours) before exiting with "out of memory" on my machine with 8 GB of RAM.  It usually has processed 14,000 to 16,000 URLs at that point.  However, in one case it had only processed 70 URLs (about 35 minutes).

There's clearly a memory leak.  In the attached screenshot of the process from the Microsoft SysInternals "Process Explorer" tool, you can see the memory usage grow with each file.  Some memory is returned, but some is not.

Possible clue: sometimes it exits saying "out of dynamic memory in yy_scan_bytes()".

(file #26471,

Greg Holmberg <gholmberg>
Sun 05 Aug 2012 02:09:29 PM UTC, original submission:  

I run Debian wheezy with linux image 3.2 on amd64 debian and when I try to run wget -qci <input file> as part of script for downloading pictures I get a giant memory leak.

Wget is on my system  1.13.4-3. Every time I start download of approx 200 pictures I get for 60 mb of download to disk,all together size of pictures, around 3GB size in ram of wget process.
I have searched but I dont see that someone else have had this sort of problem. Maybe it is not a bug I'm not sure, maybe its misconfiguration on my machine but I get that big leak every time I start wget in that script.

When I'm using wget for download from web and its started by other process like from firefox flashgot I have no problem with memory leaks.

Also to note I dont have a .wgetrc file at all. should I create it ? What to write in it ?

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #26471:  urls.txt.gz added by gholmberg (927KiB - application/x-gzip)
file #26472:  wget-mem.png added by gholmberg (96KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gholmberg (Updated the item)
  •  

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2012-09-04 gholmberg Attached File- Added urls.txt.gz, #26471
        Attached File- Added wget-mem.png, #26472

    Back to the top

    Powered by Savane 3.13-cf05.
    Corresponding source code