Just a pointer about how to trigger GC run: set_new_handler can be called (standard C++ interface) to install a function that would be called on out-of-memory condition. The function would run the GC and give 'new' another chance of succeeding:

http://stackoverflow.com/questions/5548639/what-can-new-handler-be-used-for-in-c-besides-garbage-collection

Such behavior won't fix memory fragmentation, but it is something to look at.

Hi there, any news on this issue?

I attach the patch which applied to r11694 runs the GC at every branch-back opcode.

It's not meant to be committed as there are surely better times
to run the GC than on branch-back, but is a proof of concept
about how memory use is reduced by releasing memory earlier.

Still not sure why the late-released-memory still sticks around
even with GLIBCXX_FORCE_NEW defined, anyway on low-memory devices
its still good to reduce memory usage (even out of loops:)

For the record, innocent6.as runs with 324m with early GC
vs. 723m with late GC. Around 400mb of memory saved....

(file #19267)

Alright, even just doing o = new Object inside that loop
spins memory up.

I'm afraid the only way to reduce this would be running the
GC more often. Maybe doing it at every branch or so would help.
Just we'll need to mark also the stack values and CallStack
if we go there.

The threshold shouldn't make this too much expensive

Another simplification, less variables, less opcodes

(file #19249)

Clearing (deeply, with release of any allocations) SafeStack after each advanceMovie doesn't fix so the culprit is not SafeStack.

I attach a further simplified test which drops any function.

(file #19247)

If the for loop is substituted by a frame-loop (onEnterFrame)
the memory increas doesn't occur.

Finally succeeded in producing a testcase for this.
Attached first version.

Note that a BIG difference is due to the length of the
string passed to the innocent() function.
I've made it around 55 chars which is what potlatch uses (urls)

(file #19242)

I made two changes while also thinking about this, both of which saved a lot of memory under certain circumstances, but neither of those case apply here.

One was to stop constructing as_super automatically, because in function2 calls there is a flag to show when it should be constructed. In built-in class constructors (which Gnash implements in C++ but the pp as function2s), there is also no need to create as_supers before they point they're used (which is not at all in built-in constructors). The as_supers were of course cleaned up at the end of a frame, but during a complex loop they could build up many tens of megabytes.

The other change was to stop making objects for XMLNodes unless they are accessed. Parsing the whole potlatch photos XML file is now fine (swfdec took over 30 minutes before I gave up and killed it) until the nodes are accessed.

The problem with the above two cases seems to be that as_super was never used in the potlatch loop and that all XMLNodes are accessed, so neither thing saves any memory at all for potlatch.

All DisplayObjects that are AS-referenceable have exactly one associated object.

Freed memory may not be returned to the operating system immediately, so that the continuing high usage may not necessarily be a problem. Just the fact it gets so high to start with.

By the way, valgrind's massif tool was really helpful in finding the as_super and XMLNode excesses. I bet a profile of the potlatch case would be interesting, but it is extremely slow.

Another interestings statistic is that if we drop
the whole 'attachMovie' block in photo.as, memory still
grows. Not as much as when attachMovie is called, bust still
way too much (1052m VIRT vs. 1628m).

This is only commenting the block but NOT the condition:

 if (p_icon && !_root.map.photos[innocent(p_icon)]) {
 /*
  var n=innocent(p_icon);
  _root.map.photos.attachMovie("photo",n,++photodepth);
  _root.map.photos[n].init(p_lat,p_lon,p_icon,p_desc,p_name);
 */
 }

If ALSO the condition is commented out, we only get to 510m VIRT.
This means about 1/2 GB allocated and staying resident due
to this single line:

 p_icon && !_root.map.photos[innocent(p_icon)]

!!!

I did some tests with local variables in a loop.

If I'm correct, any new objects created within a loop will not be freed until the GC runs at the end of the advance call.

(loop) {
    var i = new String("hi");
}

One memory hog there is as_super, which is currently created on all constructInstance calls. I don't see any good reason for this at the moment (though I haven't looked that carefully).

Constructing it when it's required in swf_function instead cuts usage from (with 65535 iterations in the loop above) from 128 to 92 MB.