bugGNU Octave - Bugs: bug #55287, Memory leak in graphics subsystem

 
 

bug #55287: Memory leak in graphics subsystem

Submitted by:  Lars Kindermann <larskindermann>
Submitted on:  Wed 26 Dec 2018 05:30:59 PM UTC  
 
Category:  Plotting Severity:  5 - Blocker
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  Closed
Release:  5.0.1 Operating System:  GNU/Linux

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Fri 18 Jan 2019 09:56:34 PM UTC, comment #63: 

The latest patch from Pantxo works for me.  I tested using the original two scripts, the solarsystem script, and then also by doing printing within a loop.  It all seems good so I pushed the cset here https://hg.savannah.gnu.org/hgweb/octave/rev/ad70a85f702b.  I'm going to close this bug report, but bug #55052 with excessive memory use per figure created is a separate issue that is still open.

Rik <rik5>
Project Administrator
Fri 18 Jan 2019 02:16:42 PM UTC, comment #62: 

I attached a patch: strlist (renamed m_strlist) is only modified during calls to text_to_strlist (when printing) and its value is reset when the function returns.

(file #46006)

Pantxo Diribarne <pantxo>
Project Member
Fri 18 Jan 2019 01:22:24 PM UTC, comment #61: 

If you change the ft_text_renderer destructor to also display the size of the strlist member variable, you'll see that it is zero each time the destructor is called for the text objects contained in the axes object.  But if you run a number of plot commands without doing clf, then when you quit Octave, you'll see a large value for the size of strlist.  I guess this comes from the axes object.  It must be the action of getting the extents of the tick labels.  This makes sense to me, since the render code calls accept/visit for the text objects and render clears strlist.  But the get_extent code, which also calls accept/visit and appends to strlist does not clear strlist like render does.  The following change seems to avoid the leak for me, but I'm not sure which of the two get_extent functions should really be calling strlist.clear().

diff --git a/libinterp/corefcn/ft-text-renderer.cc b/libinterp/corefcn/ft-text-renderer.cc
--- a/libinterp/corefcn/ft-text-renderer.cc
+++ b/libinterp/corefcn/ft-text-renderer.cc
@@ -1242,6 +1250,7 @@ namespace octave
   ft_text_renderer::get_extent (const std::string& txt, double rotation,
                                 const caseless_str& interpreter)
   {
+    strlist.clear ();
     text_element *elt = text_parser::parse (txt, interpreter);
     Matrix extent = get_extent (elt, rotation);
     delete elt;

Maybe there is a better place to clear strlist?  It seems to me that it should be called after executing accept/visit (when we are done with the strlist object) instead of calling it before accept/visit (to clean up after a previous call that appends to strlist).  But I don't really understand how this is supposed to work, so I'll leave that up to you.

John W. Eaton <jwe>
Project Administrator
Fri 18 Jan 2019 10:20:54 AM UTC, comment #60: 

The previous patch also prevents text from being printed :-). I attached a new version.

(file #46003)

Pantxo Diribarne <pantxo>
Project Member
Fri 18 Jan 2019 09:09:32 AM UTC, comment #59: 

@Rik: The strlist is meant for printing tex formated stings, not for caching. Clearly, I don't understand why strlist.clear(), which is called before each string rendering, does not free the consumed memory. Also, I don't understand why this memory increase only happens for the very first axes (clf, will clear the figure and avoid any further increase of the consumed memory).

I attached a preliminary patch that prevents the strlist from being filled unless it is actually needed (when text_to_strlist is called). This fixes the issue for me until I issue a "print" command. At that point I see an increase of the consumed memory of 20MB, and then the consumed memory becomes more or less stable whether or not I issue other "print" commands.

(file #46002)

Pantxo Diribarne <pantxo>
Project Member
Fri 18 Jan 2019 02:01:23 AM UTC, comment #58: 

I can confirm that the fix in comment #57 fixes the severe memory consumption of plot() and set() in the test scripts as well as my solarsystem.m script in bug #55064. Nice find!

Robert F <robcylon7>
Thu 17 Jan 2019 11:55:05 PM UTC, comment #57: 

I don't pretend to think that this is a good fix, but I do have one.  I looked at the problem cset identified by bisection and what seemed suspect was the addition of strlist.push_back calls.  I commented out the three calls and now when I run the leak tests things are fine.  I also ran the problem code from bug #55064 and that was also resolved.

From what I can tell, this feature was designed to cache data for increased performance.  So, at worst, we have to revert and suffer a little bit of a performance hit, but we get correct behavior.

(file #46001)

Rik <rik5>
Project Administrator
Thu 17 Jan 2019 11:04:02 PM UTC, comment #56: 

I've cooked up a very primitive "memory"-like function for Windows, see patch #9746.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 15 Jan 2019 06:10:07 AM UTC, comment #55: 

I added a similar print statement to the destructor ~ft_font.  When I call plot, the destructor is called 396 times.  When I call clf, the destructor is called 743 times.  Certainly more things seem to be getting removed by clf.

As an aside, why is this the destructor called so many times?  Does the ft_font class represent a single character in a single font/weight/size?

Rik <rik5>
Project Administrator
Tue 15 Jan 2019 06:02:27 AM UTC, comment #54: 

I went back to the problem changeset and expanded the destructor for ft_render to report when it was being executed.

ft_render::~ft_render (void)
{
  std::cerr << "~ft_render executing" << std::endl;
}

I find that when I just plot something, plot (1:10), the destructor is called 4 times.  I believe that this corresponds with the title, xlabel, ylabel, and zlabel objects.

However, when I execute clf then the destructor is called 5 times.  Using a debugger, the backtrace for the fifth call is

#0  ft_render::~ft_render (this=0x55f77985d348, __in_chrg=<optimized out>)
    at libinterp/corefcn/txt-eng-ft.cc:319
#1  0x00007f728589e5dc in axes::properties::~properties (this=0x55f77985c3b0,
    __in_chrg=<optimized out>) at libinterp/corefcn/graphics.h:5201
#2  0x00007f72858a8000 in axes::~axes (this=0x55f77985c3a0, __in_chrg=<optimized out>)
    at libinterp/corefcn/graphics.h:7524
#3  0x00007f72858a803c in axes::~axes (this=0x55f77985c3a0, __in_chrg=<optimized out>)
    at libinterp/corefcn/graphics.h:7524
#4  0x00007f7285766315 in graphics_object::~graphics_object (this=0x7ffd6165c9a0,
    __in_chrg=<optimized out>) at libinterp/corefcn/graphics.h:3247
#5  0x00007f72857839c6 in delete_graphics_object (h=...) at libinterp/corefcn/graphics.cc:2620
#6  0x00007f7285783a62 in delete_graphics_object (val=-18.717296929230308)
    at libinterp/corefcn/graphics.cc:2647
#7  0x00007f7285783ae5 in delete_graphics_objects (vals=...)
    at libinterp/corefcn/graphics.cc:2654
#8  0x00007f7285886d46 in F__go_delete__ (args=...) at libinterp/corefcn/graphics.cc:10578

In this case, besides the children of the axes, the axes destructor is called.  The axes destructor is empty, but it also calls the ~properties destructor.  As far as I can see, this destructor is also empty.  The only thing in the ft_render destructor is the print statement to cerr that I just added.  So, I'm sure that more work is done by clf.

Rik <rik5>
Project Administrator
Mon 14 Jan 2019 05:17:10 PM UTC, comment #53: 

Okay, I'll take a look at memory.m in a minute.

For the time being, I created tst_leak1.m and tst_leak2.m which are attached.  I find that for either test, the leaks begin with cset caa5dabc7913.  The leaks are much faster (roughly 6X) when using plot() vs. set(), but always there.  Also, I see no decline in memory used with increasing loop iteration.  The leak is always growing monotonically.  I think the key to understanding the problem is the referenced changeset.

tst_leak1.m

more off;
i=0;

do
i += 1;
if (mod (i,100) == 0)
   disp (i);
endif

data = rand (1e5, 1);
#clf
#cla
plot (data);
drawnow;

until (i > 5e3)

and tst_leak2.m

more off;
i=0;
data = rand (1e5, 1);
h = plot (data);

do
i += 1;
if (mod (i,100) == 0)
   disp (i);
endif

#clf
#cla
set (h, "ydata", rand (1e5, 1));
drawnow;

until (i > 5e3)

(file #45972, file #45973)

Rik <rik5>
Project Administrator
Mon 14 Jan 2019 04:57:33 PM UTC, comment #52: 

@Rik: "memory.m" is m-code only (file #45907) and it reports results that are the same as top (see Lars's comment #40).

Pantxo Diribarne <pantxo>
Project Member
Mon 14 Jan 2019 04:48:27 PM UTC, comment #51: 

@Pantxo: Seems like your text in comment #49 got clipped.

I do agree that ASAN isn't very helpful here since the memory is freed on exit from Octave.  We need intermediate memory reporting, for which I have been using top, but any tool, including the patch attached to this report, should work for that.  'top' is easier for me because it is built-in to the OS, and as I go backwards as far as 3 years ago in the code base it becomes hard to compile add-ins like the memory tool.

I think there are at least two bugs here which is why we are getting different behavior when the action is plot() versus set().  Rather than decide on the definitive test, I would just choose which test you want to debug first and mention it when you present results.

Rik <rik5>
Project Administrator
Mon 14 Jan 2019 02:24:47 PM UTC, comment #50: 

The figure without ASAN is now attached

(file #45971)

Pantxo Diribarne <pantxo>
Project Member
Mon 14 Jan 2019 02:21:23 PM UTC, comment #49: 

@Rik: I have spent a bit of time trying to track this issue but I'm afraid I didn't go far, especially since the actual bug is not well defined.
Let me explain: My main test is the one I posted in comment #41. This test gives me numbers that are quite reproducible on my dev machine, with ASAN enabled (See the figure attached to comment #41). The results I get on another machine, with the same script but without ASAN, are very different (See attached figure):

  • The test runs much more rapidly without ASAN, no surprise here,
  • The used memory is 7-8 times smaller without ASAN: this tends to show that the ASAN itself consumes memory that would otherwise not be an issue,
  • Without ASAN, if I use "plot" instead of "set" the timings are very different but the Used memory stops growing indefinitely after the first "clf", which tends to confirm the statement in comment #37.

Can we agree on a reproducible test, using the very handy "memory" function attached to this report? Can we agree on the fact the ASAN is of no use here since there is no unfreed memory to track? Then only

Pantxo Diribarne <pantxo>
Project Member
Fri 11 Jan 2019 04:26:37 PM UTC, comment #48: 

@Pantxo: No worries, sometimes "shit happens" as the saying goes.  In any case, as I went further back the leaks became smaller.  This was the first definitive cset where leaks began, but it probably means that once this issue is fixed we will be at the next layer of the onion.  For comparison, at caa5dabc7913 the leak is ~3MB every update period of top, while it is nearer to ~50MB if I use a recent tip.

Rik <rik5>
Project Administrator
Fri 11 Jan 2019 04:12:58 PM UTC, comment #47: 

@Rik: Many thanks for doing this (I suppose)painful bisection. I'm just not so happy of being the one who has introduced all those memory leaks ... I'll take a look.

Pantxo Diribarne <pantxo>
Project Member
Fri 11 Jan 2019 04:26:34 AM UTC, comment #46: 

After a very difficult bisection, I tracked a leak to this cset from 3 years ago

changeset:   20622:caa5dabc7913
user:        Pantxo Diribarne <pantxo.diribarne@gmail.com>
date:        Mon Oct 12 22:14:25 2015 +0200
summary:     Let the freetype renderer return a list of parsed substrings (bug #31468)

@Pantxo: Could you take a look?  Maybe we need to code a destructor that calls strlist.clear ()?

Rik <rik5>
Project Administrator
Thu 10 Jan 2019 02:34:55 AM UTC, comment #45: 

some more info:
if we plot rand(100,1) we see increasing numbers by  672 & 756
if we plot rand(1000,1) we see increasing numbers by  672 & 756
if we plot rand(100,2) we see increasing numbers by  768  844
if we plot rand(100,3) we see increasing numbers by  860  936
if we plot rand(100,4) we see increasing numbers by  948 1028
So it uses more memory with more dimensions and not with more data per dimension.

Doug Stewart <dastew>
Thu 10 Jan 2019 01:49:33 AM UTC, comment #44: 

Since this may defeat analysis, I tried to simply go backwards and use bisection to find which changeset introduced the issue.  But, there have been changes in gnulib which make it very difficult to compile with GCC > 7 as you go back in time.  So, defeated.

Rik <rik5>
Project Administrator
Thu 10 Jan 2019 12:27:21 AM UTC, comment #43: 

Comment #42 suggests that there might be additional children that are not cleared by delete (get (gca, 'children')), but are cleared by delete (gca).  THese would potentially be the 4 text objects for axis labels (title, xlabel, ylabel, zlabel).  However, I tried finding and deleting ALL children (visible or not) of the axes and it still leaks.

i=0;
do
i += 1;
data = rand (100_000, 1);
ht = findall (gca);
ht = setdiff (ht, gca);
delete (ht);
plot (data);
drawnow;
until (i > 1e3)

Rik <rik5>
Project Administrator
Wed 09 Jan 2019 11:31:28 PM UTC, comment #42: 

Just to add a possible clue, it may help narrow in on the culprit to point out that $delete(gca); (like clf) avoids any massive accumulation of memory, however, deleting only the axis' children with $delete(get(gca,'children'); does not.

---just a user

Robert F <robcylon7>
Wed 09 Jan 2019 10:07:29 PM UTC, comment #41: 

I attached a figure that I obtained using Lars's memory function and the attached script:

  • after each call to the "plot" function we see that the used memory increases steeply over the next ~50 calls to the "set" function. The size of the resulting "jump" increases with the size of the data set.
  • After 50-100 calls to the "set" function, the rate of increase of the memory used stabilizes to a value V that decreases with the number of points in the data set. 14 < V < 45kB/call.

For me, most of this memory is never released before exiting Octave.

(file #45922, file #45923)

Pantxo Diribarne <pantxo>
Project Member
Wed 09 Jan 2019 06:37:56 PM UTC, comment #40: 

Just for information, in addition to comment #28:

memory().VirtMemUsedOctave, units kB is the same as top:VIRT
memory().MemUsedOctave, units kB is the same as top:RES

Values are derived from the linux /proc/pid/status pseudofile

Lars Kindermann <larskindermann>
Wed 09 Jan 2019 06:07:10 PM UTC, comment #39: 

WRT comment 37.

I think you need to wait longer until the problem re-appear.
I wonder if this is some kind of lazy memory allocation
effect (clf frees some buffers which octave do not release back to OS but re-use for subsequent allocations).

E.g. with memory.m test (see comment 30) if I wait longer than 44 cycles, the 0 disappear and it returns to original pattern.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 09 Jan 2019 05:49:45 PM UTC, comment #38: 

RE: comment #37, that is not the behavior I see.  If I start Octave, issue a clf command and then execute the code from comment #34 (with or without the initial "close all") I still see a slow increase in the size of the octave process.

John W. Eaton <jwe>
Project Administrator
Wed 09 Jan 2019 04:34:40 PM UTC, comment #37: 

If you do a clf once after starting octave then this problem disappears.

Anonymous
Wed 09 Jan 2019 04:33:53 PM UTC, comment #36: 

OK, with the example in comment #34, I do see a slow increase in size.

Any idea how to figure out what type of object those are in the LSAN output?

John W. Eaton <jwe>
Project Administrator
Wed 09 Jan 2019 04:22:53 PM UTC, comment #35: 

There is an LSAN option
<<<<
report_objects 0 If 1, LSan will report the addresses of individual leaked objects.

>>>>


which adds some lines like:

<<<

Objects leaked above:
0x611000027b40 (256 bytes)

>>>


The total number of bytes and number of objects (2602) does not seem to grow with number of replots in the test.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 09 Jan 2019 03:20:40 PM UTC, comment #34: 

@jwe: With your modified example, I still do see the available amount of free memory decreasing in "top". Also, the following example, which should only update data, consumes more and more memory while no object has to be deleted/created:

close all
h = plot (rand (1,100000));
for ii = 1:10000
set (h, "ydata", rand (1,100000))
pause (.2)
endfor

Pantxo Diribarne <pantxo>
Project Member
Wed 09 Jan 2019 02:31:32 PM UTC, comment #33: 

I think it is a leak, but the objects are still recorded by Octave and cleaned up at exit.

It looks to me like successive calls to "plot" don't delete the children of previous plots.  That's what clf does when called with no arguments.  In that case, "do_reset" will be false and we'll end up executing the code following the else:

    ## Select only the chilren with visible handles.
    kids = get (hfig, "children");
    delete (kids);

    ## Also delete the annotation axes
    hover = findall (hfig, "-depth", 1, "tag", "scribeoverlay");
    delete (hover);

If I use this modified version of the example in comment #26, I don't see a leak:

i=0;
do
i += 1;
data = rand (100_000, 1);
delete (gca);
plot (data);
drawnow;
until (i > 1e4)

Simply deleting gca might not be sufficient, as the figure may have other children that are also not cleaned up properly.

So, where should we be cleaning up previous plots?  In the newplot function?  In the C++ code that manages figures?  I'm not sure, but it should probably be fairly deep in the plotting code so that we properly clean up all types of figures, not just those generated by "plot".

John W. Eaton <jwe>
Project Administrator
Wed 09 Jan 2019 09:03:16 AM UTC, comment #32: 

I don't think this is a memory leak. This is most probably due to objects that we simply don't delete during the clf/cla process and that are only deleted at exit. What I observe is that the used memory on my system grows approximately as much as actual identified memory leaks after successive runs of Octave.

So YES, there is a significant amount of memory that is consumed (and not understood) by the Octave process during a session with successive plot/figure creations; NO this doesn't look like a memory leak since the mysterious objects are properly freed at exit.

Pantxo Diribarne <pantxo>
Project Member
Wed 09 Jan 2019 05:52:07 AM UTC, comment #31: 

Are there more options we can set in ASAN to help debug this?  I re-ran the leak script with 100 iterations under ASAN.  While running, 'top' shows memory is increasing.  However, at exit the leak analyzer doesn't report any more memory missing than if I hadn't run the script.  Apparently by the time the leak_check_at_exit is run the memory has been cleared.

Rik <rik5>
Project Administrator
Tue 08 Jan 2019 10:01:33 PM UTC, comment #30: 

And then more fun if you stop the cycle clf and restart it
(w/o restarting the octave):

<<<....>>>
       35   288028      780
       36   288028        0
       37   288732      704
       38   289512      780
       39   290292      780
       40   291072      780
       41   291856      784
       42   292636      780
       43   293416      780
       44   294200      784
^C
octave:5> clf
octave:6> i=0
i = 0
octave:7> memold=memory().MemUsedOctave;
octave:8> do
> i=i+1;
> data=rand(100000,1);
> #clf #uncomment to avoid leak
> plot(data);
> drawnow
> mem=memory().MemUsedOctave;
> mc=mem-memold;
> disp([i mem mc])
> memold=mem;
> until i>100000
        1   292948    -2032
        2   292948        0
        3   292948        0
        4   292948        0
        5   292948        0
        6   292948        0
        7   292948        0
        8   292948        0
        9   292948        0
       10   292948        0
       11   292948        0
       12   292948        0
       13   292948        0
       14   292948        0
       15   292948        0
       16   292948        0
       17   292948        0
       18   292948        0
       19   292948        0
       20   292948        0
       21   292948        0
       22   292948        0
       23   292948        0
       24   292948        0

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Tue 08 Jan 2019 08:43:50 PM UTC, comment #29: 

I changed you code to see how the memory usage changes.

i=0
memold=memory().MemUsedOctave;
do
 i=i+1;
 data=rand(100000,1);
 #clf #uncomment to avoid leak
 plot(data);
 drawnow
 mem=memory().MemUsedOctave;
mc=mem-memold;
 disp([i mem mc])
 memold=mem;
until i>100000

It has a cycle of 4 values!

Doug Stewart <dastew>
Tue 08 Jan 2019 08:28:51 PM UTC, comment #28: 

As Octave is missing MATLABs memory() function which is very helpful in debugging memory problems, I started hacking an Octave version. It only has some of the functionality of the original yet and currently works on Linux only but it does its job. Maybe something for 6.0.

I attached it to this report.
How to use it in the test script:


i=0
do
i=i+1;
data=rand(100000,1);
#clf #uncomment to avoid leak
plot(data);
drawnow
mem=memory().MemUsedOctave;
disp([i mem])
until i>100000

(file #45907)

Lars Kindermann <larskindermann>
Tue 08 Jan 2019 08:03:27 PM UTC, comment #27: 

The leak happens with all three toolkits so it is likely to be an issue with Octave's schema for properties.

Rik <rik5>
Project Administrator
Tue 08 Jan 2019 06:54:26 PM UTC, comment #26: 

Re-opening report.  Using 'top' there is still an observed accumulation of memory when running the following script

i=0;
do
i += 1;
data = rand (100_000, 1);
#clf    % uncomment to avoid leak
#cla    % This is not enough, still leaks
plot (data);
drawnow;
until (i > 1e4)

Some testing shows that clf is enough to avoid the leak, but that just clearing the axes object with cla is not.  This makes sense because a call to plot() will normally only try to clear the axes before adding in the new data.

Rik <rik5>
Project Administrator
Tue 08 Jan 2019 06:56:06 AM UTC, comment #25: 

Please reopen.

My original example for this bug (without the clf) still consumes massive amounts of memory and crash my system (4 Gigs of RAM) after about 5000 iterations. Seems that the resources used by plot() are not released automatically...  

Lars Kindermann <larskindermann>
Tue 08 Jan 2019 12:36:43 AM UTC, comment #24: 

Unfortunately, the fixes here do not resolve bug #55064.

Rik <rik5>
Project Administrator
Mon 07 Jan 2019 11:20:18 PM UTC, comment #23: 

The leak in the graphics subsystem seems to be explained and fixed.  I'm marking this bug as fixed and closing this report.

I've file a new bug about general memory leaks during the 'make check' process.  See bug #55415.

Rik <rik5>
Project Administrator
Mon 07 Jan 2019 11:12:40 PM UTC, comment #22: 

Yes, and there seams to be reports here and there about memory leaks in libfontconifg. See e.g. https://github.com/PandorasFox/i3lock-color/issues/88 which reports pretty much the same leak, and it is supposed to have been fixed in a version after 2.13 (I am using 2.12.6).

No more mysterious leak.

Pantxo Diribarne <pantxo>
Project Member
Mon 07 Jan 2019 10:15:57 PM UTC, comment #21: 

I think you are right that there is nothing to be done about libfontconfig.  The library itself seems to have a leak, and there is no real way to get rid of it.  I configured using '--without-fontconfig' and I STILL got a leak in the library.  At least it is stable at about 4KB and never changes.

As for the lost memory during initialization of the interpreter.  This seems to be intended (jwe changeset).  The code that leaks is

    // Yes, this object is intentionally not deleted in the destructor
    // so that we avoid a crash on exit for user-defined data types.
    // See bug #53156.  If that problem is properly fixed, then this
    // could be stored as an object instead of a pointer to an object
    // allocated with new.

    vals(i) = new octave_value (val);

So, are we done?  No more memory leaks to find?

Rik <rik5>
Project Administrator
Mon 07 Jan 2019 09:16:35 PM UTC, comment #20: 

About fontconfig, I observe something strange: in ft-text-renderer.cc, if I change HAVE_FONTCONFIG to HAVE_XXXXXXX (so that Octave doesn't use fontconfig at all), I observe the same direct leak.

Now about the 8 byte leaks that are observed when registering octave types, they also happen using --no-gui-libs:

Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x7fddd3f36458 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0458)
    #1 0x7fddd2748525 in octave::type_info::register_type(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, octave_value const&, bool) ../octavestable/libinterp/octave-value/ov-typeinfo.cc:154
    #2 0x7fddd27db9e4 in octave_int32_matrix::register_type(octave::type_info&) ../octavestable/libinterp/octave-value/ov-int32.cc:68
    #3 0x7fddd27918c5 in install_types(octave::type_info&) ../octavestable/libinterp/octave-value/ov.cc:2939
    #4 0x7fddd274719d in octave::type_info::type_info(int) ../octavestable/libinterp/octave-value/ov-typeinfo.cc:82
    #5 0x7fddd315c097 in octave::interpreter::interpreter(octave::application*) ../octavestable/libinterp/corefcn/interpreter.cc:384
    #6 0x7fddd1a5c0b1 in octave::application::create_interpreter() ../octavestable/libinterp/octave.cc:314
    #7 0x7fddd1a5ca26 in octave::cli_application::execute() ../octavestable/libinterp/octave.cc:389
    #8 0x562a8a333571 in main ../octavestable/src/main-cli.cc:92
    #9 0x7fddced33b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

So this is clearly no threading issue but rather a problem in Octave's interpreter initialization.

Pantxo Diribarne <pantxo>
Project Member
Mon 07 Jan 2019 08:18:25 PM UTC, comment #19: 

I ran the tst_leak.m script with one repetition and with 100 repetitions.  In each case, the amount of memory now leaked is the same.  That's good, the leak is no longer growing without bound.

There are two remaining leaks.  The first is

Direct leak of 4352 byte(s) in 17 object(s) allocated from:
    #0 0x7fada948fb50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7fad9f40e8ed  (/usr/lib/x86_64-linux-gnu/libfontconfig.so.1+0x1d8ed)

I think we have had this one before in the past.  It may required getting the debug versions of libfontconfig installed to see if there is any more information to be gained.

The second is a series of 8 byte leaks which all have the same form of starting with QApplicationPrivate::notify_helper.

Direct leak of 8 byte(s) in 1 object(s) allocated from:
    #0 0x7fada9491458 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xe0458)
    #1 0x7fada73d74f5 in octave::type_info::register_type(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, octave_value const&, bool) libinterp/octave-value/ov-typeinfo.cc:154
    #2 0x7fada7487a3a in octave_int8_scalar::register_type(octave::type_info&) libinterp/octave-value/ov-int8.cc:75
    #3 0x7fada741ed01 in install_types(octave::type_info&) libinterp/octave-value/ov.cc:2925
    #4 0x7fada73d616d in octave::type_info::type_info(int) libinterp/octave-value/ov-typeinfo.cc:82
    #5 0x7fada7de1f89 in octave::interpreter::interpreter(octave::application*) libinterp/corefcn/interpreter.cc:384
    #6 0x7fada6713801 in octave::application::create_interpreter() libinterp/octave.cc:314
    #7 0x7fada8d908b6 in octave::octave_interpreter::execute() libgui/src/main-window.cc:115
    #8 0x7fada8f60e0c in octave::octave_interpreter::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) libgui/src/moc-main-window.cc:89
    #9 0x7fada1350121 in QObject::event(QEvent*) (/usr/lib/x86_64-linux-gnu/libQt5Core.so.5+0x2ba121)
    #10 0x7fada209e82b in QApplicationPrivate::notify_helper(QObject*, QEvent*) (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5+0x15482b)

This is a very small leak, but I'm concerned because it may indicate that we don't really understand what is happening cross-thread between the Qt GUI and the main interpreter thread.

Rik <rik5>
Project Administrator
Mon 07 Jan 2019 08:52:15 AM UTC, comment #18: 

I pushed the change here:

http://hg.savannah.gnu.org/hgweb/octave/rev/6f22d0c91beb

BTW I also crossed a static analyzer detected issue in ft-text-renderer.cc.

Pantxo Diribarne <pantxo>
Project Member
Mon 07 Jan 2019 01:26:14 AM UTC, comment #17: 

@Pantxo: That definitely helps.  Leaked bytes for a simple plot went from 478K to 168K so you should definitely apply it.

Possibly of interest are these three warnings from the static analyzer.

gl-render.cc (2362)
        V1020         The function exited without calling the 'm_glfcns.glEnd' function. Check lines: 2362, 2280.
gl-render.cc (3560)
        V1020         The function exited without calling the 'm_glfcns.glEnd' function. Check lines: 3560, 3420.
gl-render.cc (3559)
        V1020         The function exited without calling the 'm_glfcns.glEnd' function. Check lines: 3559, 3420.

Rik <rik5>
Project Administrator
Sun 06 Jan 2019 10:47:37 PM UTC, comment #16: 

I attached a patch that fixes the main memory leak (introduced by me...) in freetype, the main one. I was not able to fix the one with fontconfig.

(file #45885)

Pantxo Diribarne <pantxo>
Project Member
Sun 06 Jan 2019 09:45:40 PM UTC, comment #15: 

Also:

ldd src/octave
        linux-vdso.so.1 (0x00007ffc451b5000)
        libasan.so.5 => /lib64/libasan.so.5 (0x00007f7241d3b000)
        libX11.so.6 => /lib64/libX11.so.6 (0x00007f7241bf6000)
...

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 06 Jan 2019 09:45:34 PM UTC, comment #14: 

@Dmitri: Thanks, so I definitely didn't have the sanitizer enabled. I'll compile from scratch and see what I can find.

Pantxo Diribarne <pantxo>
Project Member
Sun 06 Jan 2019 09:31:01 PM UTC, comment #13: 

You can start octave as

ASAN_OPTIONS=help=1 ./run-octave

and if asan is linked in you should get a long list of options:

ASAN_OPTIONS=help=1 ./run-octave -q -f
Available flags for AddressSanitizer:
        quarantine_size
                - Deprecated, please use quarantine_size_mb.
        quarantine_size_mb
                - Size (in Mb) of quarantine used to detect use-after-free errors. Lower value may reduce memory usage but increase the chance of false negatives.
        thread_local_quarantine_size_kb
                - Size (in Kb) of thread local quarantine used to detect use-after-free errors. Lower value may reduce memory usage but increase the chance of false negatives. It is not advised to go lower than 64Kb, otherwise frequent transfers to global quarantine might affect performance.
        redzone
                - Minimal size (in bytes) of redzones around heap objects. Requirement: redzone >= 16, is a power of two.
        max_redzone
                - Maximal size (in bytes) of redzones around heap objects.
        debug
                - If set, prints some debugging information and does additional checks.
        report_globals
...

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 06 Jan 2019 09:23:13 PM UTC, comment #12: 

Sorry Rik, I must be doing something wrong when trying to enable the address sanitizer (I don't observe any leak report at exit). How can I check that the address-sanitizer is actually enabled in octave?

Pantxo Diribarne <pantxo>
Project Member
Mon 31 Dec 2018 05:31:01 AM UTC, comment #11: 

I'm using the attached tst_leak.m file

i=0;
do
i += 1;
#disp(i);
data = rand (100_000, 1);
clf
plot (data);
drawnow;
until (i > 10)

After building with debugging and --enable-address-sanitizer-flags I can run with

./run-octave -f tst_leak.m |& tee myleak.log

The top three leaks are

Direct leak of 852720 byte(s) in 10659 object(s) allocated from:
    #0 0x7f276e9f9b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f27646bbd6c  (/usr/lib/x86_64-linux-gnu/libfreetype.so.6+0x11d6c)

Direct leak of 4352 byte(s) in 17 object(s) allocated from:
    #0 0x7f276e9f9b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f276497b8ed  (/usr/lib/x86_64-linux-gnu/libfontconfig.so.1+0x1d8ed)

Direct leak of 208 byte(s) in 1 object(s) allocated from:
    #0 0x7f276e9f9d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
    #1 0x7f273b33a828  (/usr/lib/x86_64-linux-gnu/dri/i965_dri.so+0x419828)

It gets pretty small after that.  Clearly we are not doing something correctly in the interaction with FreeType.  I'm adding Pantxo to the CC list since he has done a lot of work with FreeType and Fontconfig and might understand if we are not freeing memory correctly from text objects.

(file #45805)

Rik <rik5>
Project Administrator
Sun 30 Dec 2018 09:27:22 PM UTC, comment #10: 

There are a bunch of reports related to segfaults with Qt plotting.  I believe we need to fix this memory leak first, and then go on to see if that resolves any of the others.

Rik <rik5>
Project Administrator
Thu 27 Dec 2018 08:32:58 PM UTC, comment #9: 

I compiled octave with leak sanitizer flags. Running the loop
with LSAN_OPTIONS=leak_check_at_exit=0 gives no errors. Running
without this flag produces lots of errors and at the top I see
something like that (after 100 cycles):

==20132==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 26500000 byte(s) in 331250 object(s) allocated from:
    #0 0x7f64fa95579f in __interceptor_malloc (/lib64/liblsan.so.0+0xf79f)
    #1 0x7f64f7f59d37  (/lib64/libfreetype.so.6+0x15d37)

Number of bytes seems to increase linearly with number of cycles:

#i      bytes           objects
1       43200           540
2       310480          3881
3       577680          7221
10      2448400         30605
30      7793200         97415
100     26500000        331250

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 27 Dec 2018 07:51:48 PM UTC, comment #8: 

Now I waited long enough to see it grows. The size grows in chunks, not continiously. Not sure if this is octave's or OS's doing.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 27 Dec 2018 07:44:59 PM UTC, comment #7: 

I do not see problem running with gnuplot -- the octave process
grows a little bit at the beginning
(to 202964 SHR and 670152 RES) and then stays at that size for
as long as I am willing to wait.

(I do not have fltk compiled in, but I will shortly)

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 27 Dec 2018 06:32:44 PM UTC, comment #6: 

This seems to have been reported twice before.  See bugs #55064, #55052.

Rik <rik5>
Project Administrator
Thu 27 Dec 2018 06:05:52 PM UTC, comment #5: 

I'm pretty sure this behavior was also reported on the Maintainer's list or on another bug report.  I consider it real.  Using the test code, and watching the memory usage by the Octave process, I see it go up and up.

I'm adding jwe to the CC list.  One of the changes that was made was to switch from our own internal reference counting to using C++11 features such as std::shared_ptr.  It might be that we are somehow not signaling correctly when we are done with memory.

Also, as noted, this is a huge regression from 4.4.1 where the memory consumption is quite stable.

This happens with Qt, FLTK, AND gnuplot toolkits.  I ran with the --no-gui-libs option and still see the issue so it is not related to Qt and different threads.

Rik <rik5>
Project Administrator
Thu 27 Dec 2018 04:23:02 PM UTC, comment #4: 

might be helpful for debugging:

When clearing the figure (clf) before each plot(), the memory usage becomes much less, about a third. But still memory (which cannot be cleared any more within octave) builds up until the ram is exhausted and the system swaps and almost freezes.

i=0
do
i=i+1;disp(i)
data=rand(100000,1);
clf
plot(data);drawnow;
until false

Lars Kindermann <larskindermann>
Thu 27 Dec 2018 03:47:13 PM UTC, comment #3: 

Since there seems to be a suspicion about graphics drivers,
mine are:

Graphics: Intel® Iris Plus Graphics 640 (Kaby Lake GT3e)

Processor: Intel® Core™ i5-7260U CPU @ 2.20GHz × 4

and:
xorg-x11-drv-intel-2.99.917-39.20180618.fc29.x86_64

Michael Godfrey <godfrey>
Project Member
Wed 26 Dec 2018 10:30:13 PM UTC, comment #2: 

On my system  octave4.4.1  id ans = 80643ff8c8b7
Ubuntu 18.04
there is a memory usage that increases for each plot.

iteration      memory used

  1.                 10.5 GiB

1000              11.1 GiB
2000              11.6
3000              12.2
4000              12.7
5000              13.2 GiB

I used control c to stop and memory usage remained at 13.2 GiB
I then deleted the plot and still 13.2 GiB
I then closed Octave and it released all the used memory
back down to 10.5 GiB

Doug Stewart <dastew>
Wed 26 Dec 2018 09:56:13 PM UTC, comment #1: 

My system (fc29 octave  _octave_config_info_ ("hg_id")
dd97dc559679, graphics_toolkit qt) behaved somewhat differently:

1. loop ran to about 4500 iterations before the system
   screen froze. There was some swapping, but not enough
   to be a resource problem.

2. Only way to recover was to login from another machine.
   This took some time, but finally, shutdown -r now
   worked.

There was some resource usage which essentially locked
the system.

It would definitely be a good idea to determine the problem.

Michael Godfrey <godfrey>
Project Member
Wed 26 Dec 2018 05:30:59 PM UTC, original submission:  

Running this script for about 1500 iterations uses up all my 4 gigabytes of memory so my system starts to swap and it becomes almost inresponsive:

i=0
do
i=i+1;disp(i)
data=rand(100000,1);
plot(data);drawnow;
until false

Also if stopping early, used memory is never released, even after the figure is closed or cleared.

When trying to stop the loop with multiple ctrl-C, Octave crashes with

+verabtim+
^C^C^C^C^Cterminate called after throwing an instance of 'octave::interrupt_exception'
fatal: caught signal Abgebrochen -- stopping myself...
octave exited with signal 6
-verbatim-

No such problem in 4.1.0.

Lars Kindermann <larskindermann>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #46006:  protect_strlist3.patch added by pantxo (6KiB - text/x-patch)
file #46003:  protect_strlist2.patch added by pantxo (9KiB - text/x-patch)
file #46002:  protect_strlist.patch added by pantxo (8KiB - text/x-patch)
file #46001:  leak.diff added by rik5 (1KiB - text/x-patch)
file #45972:  tst_leak1.m added by rik5 (148B - text/x-matlab)
file #45973:  tst_leak2.m added by rik5 (184B - text/x-matlab)
file #45971:  mem_log_NOASAN.pdf added by pantxo (6KiB - application/pdf)
file #45922:  leak_log.pdf added by pantxo (31KiB - application/pdf)
file #45923:  mem_log.m added by pantxo (1009B - text/x-objcsrc)
file #45907:  memory.m added by larskindermann (3KiB - text/x-objcsrc)
file #45885:  ft_leak.patch added by pantxo (979B - text/x-patch)
file #45805:  tst_leak.m added by rik5 (99B - text/x-matlab)

 

Carbon-Copy List
  • -email is unavailable- added by robcylon7 (Posted a comment)
  • -email is unavailable- added by robcylon7 (Related to my report 55064)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by pantxo (Posted a comment)
  • -email is unavailable- added by rik5
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by rik5
  • -email is unavailable- added by dastew (Posted a comment)
  • -email is unavailable- added by godfrey (Posted a comment)
  • -email is unavailable- added by larskindermann (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 25 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-18 rik5 StatusConfirmed => Fixed
        Open/ClosedOpen => Closed
    2019-01-18 pantxo Attached File- => Added protect_strlist3.patch, #46006
    2019-01-18 pantxo Attached File- => Added protect_strlist2.patch, #46003
    2019-01-18 pantxo Attached File- => Added protect_strlist.patch, #46002
    2019-01-17 rik5 Attached File- => Added leak.diff, #46001
    2019-01-14 rik5 Attached File- => Added tst_leak1.m, #45972
        Attached File- => Added tst_leak2.m, #45973
    2019-01-14 rik5 Release5.0.0 => 5.0.1
    2019-01-14 pantxo Attached File- => Added mem_log_NOASAN.pdf, #45971
    2019-01-09 pantxo Attached File- => Added leak_log.pdf, #45922
        Attached File- => Added mem_log.m, #45923
    2019-01-09 robcylon7 Carbon-Copy- => Added rdfischetti@gmail.com
    2019-01-08 larskindermann Attached File- => Added memory.m, #45907
    2019-01-08 rik5 StatusFixed => Confirmed
        Open/ClosedClosed => Open
    2019-01-07 rik5 StatusIn Progress => Fixed
        Open/ClosedOpen => Closed
    2019-01-07 rik5 StatusPatch Submitted => In Progress
    2019-01-06 pantxo Attached File- => Added ft_leak.patch, #45885
        StatusConfirmed => Patch Submitted
    2018-12-31 rik5 Attached File- => Added tst_leak.m, #45805
        Carbon-Copy- => Added pantxo
    2018-12-27 rik5 Dependencies- => Depends on bugs #55064
    2018-12-27 rik5 Dependencies- => Depends on bugs #55052

    Back to the top


    Powered by Savane 3.5