bugGNU Octave - Bugs: bug #55052, Memory consumption of figure...

 
 

bug #55052: Memory consumption of figure command going up significantly from 4.2 to 4.4

Submitter:  None
Submitted:  Mon 19 Nov 2018 05:39:09 AM UTC
 
Category:  Plotting Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Confirmed Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       

 

( Jump to the original submission )

Fri 20 Jan 2023 05:26:23 PM UTC, comment #14: 

While investigating this problem, I also noticed a leak when just creating a figure and exiting Octave.  With current dev sources (revision eb32cf98eed1) built on my debian system, the problem showed up in the address sanitizer report like this:

Indirect leak of 1376256 byte(s) in 4 object(s) allocated from:
    #0 0x7f6d250b7ffd in __interceptor_posix_memalign ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x7f6c024ab3be  (/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.15.6+0x723be)

This leak happened for me if I started Octave and executed

figure
quit

If I executed the figure function N times and then quit, the log showed a larger number of bytes lost in N*4 objects allocated from somewhere in libQt5Widgets.

From what I could tell, the leak seemed to happening inside QOpenGLWidget.

In an attempt to find exactly where inside QOpenGLWidget the leak was happening, I tried installing debugging symbols for the Qt libraries.  But that forced an upgrade (I'm running debian testing) and then afterward, with NO changes to Octave, I'm now unable to detect the leak.  I'm not sure what changed.  I'm now using Qt 5.15.8 and Mesa 22.3.3.  Previously I believe it was Qt 5.15.2 and Mesa 20.3.5.

Unless someone can point to how we are using QOpenGLWidget incorrectly in a way that would cause this leak, then I have to conclude that it is either a bug in Qt (that may be fixed now) or a bug in the OpenGL libraries.

As part of digging into Octave's use of QOpenGLWidget, I did make the following change, but it is not what made the memory leak disappear for me.

http://hg.savannah.gnu.org/hgweb/octave/rev/d94ceed56929

John W. Eaton <jwe>
Project Administrator
Fri 20 Jan 2023 05:22:42 PM UTC, comment #13: 

On my debian system with Qt 5.15.8 and Mesa 22.3.3, I'm seeing the following results with a build from a recent revision on the stable branch (hg id 9ab31fe294c1):

>> memory
Octave is running on x86_64-pc-linux-gnu
System    RAM: 131835716 kiB,  swap: 259136504 kiB
Octave    RAM:    194512 kiB,  virt:   5663004 kiB
Free      RAM:  10830144 kiB,  swap: 259088376 kiB
Available RAM: 128627160 kiB, total: 387715536 kiB
>> figure; memory
Octave is running on x86_64-pc-linux-gnu
System    RAM: 131835716 kiB,  swap: 259136504 kiB
Octave    RAM:    258304 kiB,  virt:  10266868 kiB
Free      RAM:  10790900 kiB,  swap: 259088376 kiB
Available RAM: 128587928 kiB, total: 387676304 kiB
>> figure; memory
Octave is running on x86_64-pc-linux-gnu
System    RAM: 131835716 kiB,  swap: 259136504 kiB
Octave    RAM:    279416 kiB,  virt:  10289752 kiB
Free      RAM:  10766080 kiB,  swap: 259088376 kiB
Available RAM: 128563108 kiB, total: 387651484 kiB
>> figure; memory
Octave is running on x86_64-pc-linux-gnu
System    RAM: 131835716 kiB,  swap: 259136504 kiB
Octave    RAM:    295860 kiB,  virt:  10307184 kiB
Free      RAM:  10746820 kiB,  swap: 259088376 kiB
Available RAM: 128543848 kiB, total: 387632224 kiB
>> figure; memory
Octave is running on x86_64-pc-linux-gnu
System    RAM: 131835716 kiB,  swap: 259136504 kiB
Octave    RAM:    312404 kiB,  virt:  10323900 kiB
Free      RAM:  10723460 kiB,  swap: 259088376 kiB
Available RAM: 128520488 kiB, total: 387608864 kiB

Initially, the usage appears to be about 65MB but then is around 15MB per figure after that.  Is that too large?  I don't know.  there's a lot going on behind the scenes for each figure.

Since this bug was initially about Octave 4.4 vs. 4.2, I will note that Octave 4.2 could only use the (now obsolete) QGLWidget but 4.4 could use that or the newer QOpenGLWidget.  Maybe that change accounts for the increased memory usage?

If anyone is interested in investigating further, it is still possible for Octave 8 to use QGLWidget if QOpenGLWidget is not available.  You could also force QGLWidget to be used if you edit config.h after running configure (there's no configure option for this choice, just change it so that HAVE_QOPENGLWIDGET, HAVE_QOFFSCREENSURFACE, and HAVE_QT_OFFSCREEN are undedefined).  When I do that, I see reduced memory usage for each figure:

>> memory
Octave is running on x86_64-pc-linux-gnu
System    RAM: 131835716 kiB,  swap: 259136504 kiB
Octave    RAM:    194744 kiB,  virt:   5664384 kiB
Free      RAM:  10848360 kiB,  swap: 259088376 kiB
Available RAM: 128641652 kiB, total: 387730028 kiB
>> figure; memory
Octave is running on x86_64-pc-linux-gnu
System    RAM: 131835716 kiB,  swap: 259136504 kiB
Octave    RAM:    235316 kiB,  virt:  10245632 kiB
Free      RAM:  10821520 kiB,  swap: 259088376 kiB
Available RAM: 128614764 kiB, total: 387703140 kiB
>> figure; memory
Octave is running on x86_64-pc-linux-gnu
System    RAM: 131835716 kiB,  swap: 259136504 kiB
Octave    RAM:    246848 kiB,  virt:  10258692 kiB
Free      RAM:  10803804 kiB,  swap: 259088376 kiB
Available RAM: 128597048 kiB, total: 387685424 kiB
>> figure; memory
Octave is running on x86_64-pc-linux-gnu
System    RAM: 131835716 kiB,  swap: 259136504 kiB
Octave    RAM:    253460 kiB,  virt:  10266040 kiB
Free      RAM:  10791584 kiB,  swap: 259088376 kiB
Available RAM: 128584828 kiB, total: 387673204 kiB
>> figure; memory
Octave is running on x86_64-pc-linux-gnu
System    RAM: 131835716 kiB,  swap: 259136504 kiB
Octave    RAM:    260384 kiB,  virt:  10273712 kiB
Free      RAM:  10785972 kiB,  swap: 259088376 kiB
Available RAM: 128579216 kiB, total: 387667592 kiB

That appears to be about 7MB per figure (after the first).

Since this difference is due only to a change in Qt widgets and the old QGLWidget is considered obsolete, I don't see what we can (easily) do about it.  I don't think we really want to write our own version of QOpenGLWidget...

John W. Eaton <jwe>
Project Administrator
Wed 18 Jan 2023 10:43:44 PM UTC, comment #12: 

How do these 'memory' output numbers have to be perceived? Does the growth of "virt" also needs to be added in?

Anyway, with Octave 9.0.0 on Windows (dev from Jan 16) I see much higher figures for Octave RAM:
- with system OpenGL (Radeon / ATI): first figure ~65 MB, each consecutive figure ~25 MB
- with llvm pipe / Mesa: all figures ~45-50 MB

Philip Nienhuis <philipnienhuis>
Project Member
Wed 18 Jan 2023 07:48:30 AM UTC, comment #11: 

The additional amount of memory is about 5-10 MiB per figure for me on Windows (apart from for the very first figure which is probably because it initializes the graphics):

>> memory
Octave is running on x86_64-w64-mingw32
System    RAM:  33474268 kiB,  swap:  38455004 kiB
Octave    RAM:    126860 kiB,  virt:   1155916 kiB
Available RAM:  22396728 kiB, total:  45916224 kiB
>> figure
>> memory
Octave is running on x86_64-w64-mingw32
System    RAM:  33474268 kiB,  swap:  38455004 kiB
Octave    RAM:    195576 kiB,  virt:   1309348 kiB
Available RAM:  22397608 kiB, total:  45839920 kiB
>> figure
>> memory
Octave is running on x86_64-w64-mingw32
System    RAM:  33474268 kiB,  swap:  38455004 kiB
Octave    RAM:    205948 kiB,  virt:   1343432 kiB
Available RAM:  22419912 kiB, total:  45849268 kiB
>> figure
>> memory
Octave is running on x86_64-w64-mingw32
System    RAM:  33474268 kiB,  swap:  38455004 kiB
Octave    RAM:    211204 kiB,  virt:   1367096 kiB
Available RAM:  22420372 kiB, total:  45838948 kiB
>> figure
>> memory
Octave is running on x86_64-w64-mingw32
System    RAM:  33474268 kiB,  swap:  38455004 kiB
Octave    RAM:    219092 kiB,  virt:   1395280 kiB
Available RAM:  20207524 kiB, total:  42878436 kiB

That is about the same that Rik wrote would be way too much in comment #7.

Do you see something different?

Markus M├╝tzel <mmuetzel>
Project Member
Wed 18 Jan 2023 03:47:01 AM UTC, comment #10: 

This problem is not happening with Octave 7.

Anonymous
Sat 19 Jan 2019 06:09:37 PM UTC, comment #9: 

Another test I did was to switch the graphics toolkit to gnuplot.  In that case, each new figure results in only a few tens of kilobytes usage.  Presumably this is graphics_properties object.

In any case, it points to the problem being with the OpenGL toolkits rather than our overall graphics code.

I don't like the crazy memory usage, but I don't think this is a blocker.  It seems quite unusual to create hundreds of figures in an Octave session.  Most of the time I find there are only a few open figures and they are re-used for new plots.

Rik <rik5>
Project Administrator
Sat 19 Jan 2019 04:40:51 PM UTC, comment #8: 

With the latest sources (stable or default) the qt graphics toolkit is loaded dynamically.  So that means you should see a big jump for the first figure as Octave loads the QtOpenGL and dependent libraries.  When _init_qt_ is executed it calls mlock, so Octave won't unload the .oct file and libraries.  Even after I tried munlock and calling _shutdown_qt_ and clear all, I was unable to get Octave to unload the _init_qt_.oct file.

Apart from the memory allocated to the shared library, I also see the usage go up with figure creation.  I'm looking at the value of RSS reported by ps:

 69140  ## before qt graphics is loaded
139156  ## after figure (1)
152580  ## after figure (2)
165880  ## after figure (3)
163992  ## after close (3)
161624  ## after close (2)
157900  ## after close (1)

So it is increasing by about 13MB for each new figure and only decreasing by about 2MB, though closing the last figure seems to release more.

Here is what I see with fltk:

 67296  ## before fltk graphics is loaded
100560  ## after figure (1)
101008  ## after figure (2)
101272  ## after figure (3)
101536  ## after close (3)
102692  ## after close (2)
102692  ## after close (1)

It's no surprise that fltk is smaller than qt, but I don't understand what's going on with this case because the size appears to increase when closing figures?!?

John W. Eaton <jwe>
Project Administrator
Fri 18 Jan 2019 09:53:34 PM UTC, comment #7: 

Changing the Operating System to Any.  Each new figure on Linux is ~7MB on my machine which is way too much.  Interestingly, if you close the figures the memory is not reclaimed until Octave actually exits.

Rik <rik5>
Project Administrator
Mon 31 Dec 2018 12:06:07 AM UTC, comment #6: 

How bad does it gets if you set graphics_toolkit to gnuplot?
Perhaps it is related to but #55287

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 30 Dec 2018 11:54:49 PM UTC, comment #5: 

With Octave-6.0.0, M usage on Linux for just an empty figure handle  amounts to ~6 MB. Less than on Windows, but still IMO outrageously much.

(checked using the command in a bash shell:

ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |cut -d "" -f2 | cut -d "-" -f1 | grep octave

after each "figure" command in Octave, see below)

## Before "figure" command:
[philip@DeskPRN ~]$ ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |cut -d "" -f2 | cut -d "-" -f1 | grep octave
       161.83 Mb /usr/local/libexec/octave/6.0.0/exec/x86_64
         0.57 Mb octave

## After opening 3 figure handles:

[philip@DeskPRN ~]$ ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' |cut -d "" -f2 | cut -d "-" -f1 | grep octave
       180.06 Mb /usr/local/libexec/octave/6.0.0/exec/x86_64
         0.57 Mb octave
[philip@DeskPRN ~]$

Philip Nienhuis <philipnienhuis>
Project Member
Sun 30 Dec 2018 11:32:07 PM UTC, comment #4: 

With 6.0.0, just the command "figure" makes RAM usage go up with 37 MB.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 13 Dec 2018 05:37:20 PM UTC, comment #3: 

bug #55064 looks related.
Anyway even 35 MB RAM for a graphics handle about just a few data is just ridiculous.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 20 Nov 2018 08:11:24 AM UTC, comment #2: 

On Windows it's not so bad, with a 64bit idx 5.0.0 dev cross-build "just" 35 MB per figure :-)

Philip Nienhuis <philipnienhuis>
Project Member
Mon 19 Nov 2018 05:21:08 PM UTC, comment #1: 

I can confirm that memory consumption has been going up.  I tested on a Linux system using self-compiled binaries.

Results for a new figure

Octave 4.2.1 : 24MB
Octave 4.4.1 : 48MB
Octave 5.0.0 : 48MB

The amount might be larger on Windows, but the trend is the problem.

Rik <rik5>
Project Administrator
Mon 19 Nov 2018 05:39:09 AM UTC, original submission:  

Hello all!

Each invocation of the "figure" command in version 4.4.0 requires extra 110 MG of memory while in 4.2.1 it requires about 3-4 MG.

Thanks.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only project members can vote.

     

     

     

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-18 rik5 Operating SystemMicrosoft Windows Any
    2018-12-27 rik5 Dependencies- bugs #55287 is dependent
    2018-11-20 philipnienhuis Release4.4.1 dev
    2018-11-19 rik5 StatusNone Confirmed
        Release4.4.0 4.4.1
        SummaryMemory consumption of figure command Memory consumption of figure command going up significantly from 4.2 to 4.4

    Back to the top

    Powered by Savane 3.9