bugGNU Wget - Bugs: bug #33183, Wget dies with memory exhausted...

 
 

bug #33183: Wget dies with memory exhausted error

Submitted by:  None
Submitted on:  Fri 29 Apr 2011 06:46:30 PM UTC  
 
Category: Crash/Freeze/InfloopSeverity: 3 - Normal
Priority: 5 - NormalStatus: None
Privacy: PublicAssigned to: None
Originator Name: C HOriginator Email: -unavailable-
Open/Closed: OpenRelease: 1.12
Operating System: Mac OSReproducibility: Every Time
Fixed Release: NonePlanned Release: None
Regression: NoneWork Required: None
Patch Included: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 26 Jul 2011 01:34:53 AM UTC, comment #1:

I too am having a very similar issue to yours, also I am dealing with huge files some are as large as 30 GB and there are many that size in the mirror job I run. It seems to only have issues when dealing with very large files. Also the site I mirror has zero css in it so I know that is not the issue.

The Doctor <yessirdoctor>
Fri 29 Apr 2011 06:46:30 PM UTC, original submission:

Hi..

I'm mirroring a large directory tree from an https url, with a
username and password. The server has a self-signed certificate.
-------------------8<---cut here----8<-------
$ wget -c --no-check-certificate --user=xxx --password=xxx -nH -np --mirror https://example.com/downloads/xxx/

....

--2011-04-26 17:07:23-- https://example.com/downloads/4088379.GOUA.2010Jul22.tar
Connecting to example.com|11.22.33.44|:443... connected.
HTTP request sent, awaiting response... 200 OK

The file is already fully retrieved; nothing to do.

--2011-04-26 17:07:24-- https://example.com/downloads/4088380.GOUA.2010Jul22.tar
Connecting to example.com|11.22.33.44|:443... connected.
HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable

The file is already fully retrieved; nothing to do.

wget(13795,0xa043e540) malloc: *** mmap(size=2147483648) failed (error code=12)

      • error: can't allocate region
      • set a breakpoint in malloc_error_break to debug

wget: memory exhausted
51.071u 57.144s 20:28.73 8.8% 0+0k 2+2io 0pf+0w
$
-------------------8<---cut here----8<-------

This took 20 minutes to die. The files were mostly downloaded
already, so rerunning the wget dies every time. The destination directory has 91G of data in 2514 files already; I'm not sure how big the source tree is.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by yessirdoctor (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    No Changes Have Been Made to This Item

    Back to the top


    Powered by Savane 3.1-cleanup1