bugGNU Wget - Bugs: bug #37030, When I run wget with wget -qci...

 
 

bug #37030: When I run wget with wget -qci <inputfile> it eats memory fast 3GB in 10 minutes

Submitted by:  None
Submitted on:  Sun 05 Aug 2012 02:09:29 PM UTC  
 
Category: Memory LeakSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: Marko PavlovicOriginator Email: -unavailable-
Open/Closed: OpenRelease: 1.13.4
Operating System: GNU/LinuxReproducibility: Every Time
Fixed Release: NonePlanned Release: None
Regression: NoneWork Required: None
Patch Included: No

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 04 Sep 2012 06:45:31 PM UTC, comment #1:

I'm getting the same thing. I'm using version 1.13.4 on Windows, from cygwin. Here's my command line:

wget -E -H -k -p -np --mirror -R txt,jpg,gif -i urls.txt

The file urls.txt contains URLs to sec.gov. I'm attaching the file so someone can reproduce the condition.

Wget will run for several hours (somewhere around 6 hours) before exiting with "out of memory" on my machine with 8 GB of RAM. It usually has processed 14,000 to 16,000 URLs at that point. However, in one case it had only processed 70 URLs (about 35 minutes).

There's clearly a memory leak. In the attached screenshot of the process from the Microsoft SysInternals "Process Explorer" tool, you can see the memory usage grow with each file. Some memory is returned, but some is not.

Possible clue: sometimes it exits saying "out of dynamic memory in yy_scan_bytes()".

(file #26471, file #26472)

Greg Holmberg <gholmberg>
Sun 05 Aug 2012 02:09:29 PM UTC, original submission:

I run Debian wheezy with linux image 3.2 on amd64 debian and when I try to run wget -qci <input file> as part of script for downloading pictures I get a giant memory leak.

Wget is on my system 1.13.4-3. Every time I start download of approx 200 pictures I get for 60 mb of download to disk,all together size of pictures, around 3GB size in ram of wget process.
I have searched but I dont see that someone else have had this sort of problem. Maybe it is not a bug I'm not sure, maybe its misconfiguration on my machine but I get that big leak every time I start wget in that script.

When I'm using wget for download from web and its started by other process like from firefox flashgot I have no problem with memory leaks.

Also to note I dont have a .wgetrc file at all. should I create it ? What to write in it ?

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #26471:  urls.txt.gz added by gholmberg (927kB - application/x-gzip)
file #26472:  wget-mem.png added by gholmberg (96kB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by gholmberg (Updated the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 04 Sep 2012 06:45:31 PM UTCgholmbergAttached File-=>Added urls.txt.gz, #26471
      Attached File-=>Added wget-mem.png, #26472

    Back to the top


    Powered by Savane 3.1-cleanup