bugGNU Octave - Bugs: bug #59659, Unlimited memory usage when...

 
 

bug #59659: Unlimited memory usage when assigning function handles in a loop

Submitter:  None
Submitted:  Fri 11 Dec 2020 02:00:23 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Fixed Assigned to:  None
Originator Name:  Michael Erhard Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 6.1.0
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 20 Dec 2020 06:14:01 PM UTC, comment #18: 

The simple loop from comment #3 (version 3) no longer consumes more and more memory for me as well.

Closing as fixed.

Markus Mützel <mmuetzel>
Group administrator
Thu 17 Dec 2020 03:40:05 PM UTC, comment #17: 

Dear Michael Erhard,

Agree to Dmitri comment #16.

I am the maintainer/operator of octave.space.  The Octave builds happen fully automated and are to the best of my knowledge free of any malware and trojans.  My MS Windows test system is still running after the installation of the recent build and MS Windows defender did not complain.  If you face any problems, please report it here, in another bug report, or on Discourse.


Anyways, distrust is good.  You are free to study all build logs

  https://buildbot.octave.space/#/builders/8/builds/38
  https://buildbot.octave.space/#/builders/9/builds/34
  https://www.octave.space/data/stable/3a15c18c705f/log-octave-mxe-w64-34.zip

Like all free software there is no warranty (neither for most paid software, hidden in long license texts 😇).  I am very interested in offering a service to support Octave development and not to harm Octave users.

Kai Torben Ohlhus <siko1056>
Group Member
Thu 17 Dec 2020 03:36:17 PM UTC, comment #16: 

It is most likely a false positive.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 17 Dec 2020 02:08:05 PM UTC, comment #15: 

it is the mingw64\bin\mgwfltknox-1.3.dll
Trojan.Win32.Generic

Anonymous
Thu 17 Dec 2020 12:42:29 PM UTC, comment #14: 

I get a trojan warning during installation of 6.1.1 from wednesday?

Anonymous
Tue 15 Dec 2020 06:13:20 AM UTC, comment #13: 

Best thanks for your quickest possible reply...I'll test a nightly build on my windows operating system - also thanks for the posted link...until now, this link was out of my knowledge (this should help me also with previously posted an fixed erros).
M. Erhard

Anonymous
Tue 15 Dec 2020 05:03:28 AM UTC, comment #12: 

I guess OP can try a nightly builds from https://buildbot.octave.space/#/download
(hg id past 76c94c998d7b (stable)) - may take an extra day.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 15 Dec 2020 04:54:01 AM UTC, comment #11: 

OK, may be I see some artifacts of asan.
If I do ps -eo rss,comm | grep oct after launching octave and then
after each run of for k=1:1e6 ; fx = @(x) 1 ; endfor I get
(with ASAN compile):

$ ps -eo rss,comm | grep oct
312808 lt-octave-gui
$ ps -eo rss,comm | grep oct
730488 lt-octave-gui
$ ps -eo rss,comm | grep oct
736548 lt-octave-gui
$ ps -eo rss,comm | grep oct
737604 lt-octave-gui

With "normal" (optimized) compile I get exactly the same number
(87200) each time.
So perhaps it is OK for now.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 15 Dec 2020 04:28:44 AM UTC, comment #10: 

I no longer see an increase in memory usage when running the test loop from comment #5.

The smaller leaks at exit associated with the register_type function are due to a known problem and aren't related to the function handle leak.

John W. Eaton <jwe>
Group administrator
Tue 15 Dec 2020 02:29:33 AM UTC, comment #9: 

Effects is smaller, but I still see memory increase.
I arttach the full asan trace. 
There are few 300+ bytes leaks E.g.:


Indirect leak of 384 byte(s) in 1 object(s) allocated from:
    #0 0x7fe988e557b0 in operator new(unsigned long) (/usr/lib64/libasan.so.5+0xf17b0)
    #1 0x7fe986914b83 in octave_user_function::register_type(octave::type_info&) ../libinterp/octave-value/ov-usr-fcn.cc:207
    #2 0x7fe986950779 in install_types(octave::type_info&) ../libinterp/octave-value/ov.cc:2954
    #3 0x7fe9869065d9 in octave::type_info::type_info(int) ../libinterp/octave-value/ov-typeinfo.cc:83
    #4 0x7fe9873cc53b in octave::interpreter::interpreter(octave::application*) ../libinterp/corefcn/interpreter.cc:473
    #5 0x7fe985c3fc5b in octave::application::create_interpreter() ../libinterp/octave.cc:302
    #6 0x7fe988558c43 in octave::interpreter_qobject::execute() ../libgui/src/interpreter-qobject.cc:51
    #7 0x7fe9887f5fcb in octave::interpreter_qobject::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) libgui/src/moc-interpreter-qobject.cc:95
    #8 0x7fe983d9e275 in QObject::event(QEvent*) (/lib64/libQt5Core.so.5+0x288275)
    #9 0x7fe9846be5f4 in QApplicationPrivate::notify_helper(QObject*, QEvent*) (/lib64/libQt5Widgets.so.5+0x1675f4)


Dmitri.
--


(file #50463)

Dmitri A. Sergatskov <dasergatskov>
Tue 15 Dec 2020 02:06:27 AM UTC, comment #8: 

Thanks for the report, simplified test, and stack trace.  I pushed the following change on stable and merged with default:

http://hg.savannah.gnu.org/hgweb/octave/rev/76c94c998d7b

John W. Eaton <jwe>
Group administrator
Tue 15 Dec 2020 12:19:44 AM UTC, comment #7: 

Running with ASAN w/o any options I see:



Direct leak of 127999232 byte(s) in 999994 object(s) allocated from:
    #0 0x7fc117c607b0 in operator new(unsigned long) (/usr/lib64/libasan.so.5+0xf17b0)
    #1 0x7fc115614e29 in octave_fcn_handle::octave_fcn_handle(octave_value const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, octave_value, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, octave_value> > > const&) ../libinterp/octave-value/ov-fcn-handle.cc:2561
    #2 0x7fc11598caf2 in octave::tree_anon_fcn_handle::evaluate(octave::tree_evaluator&, int) ../libinterp/parse-tree/pt-fcn-handle.cc:178
    #3 0x7fc1159216ac in octave::tree_simple_assignment::evaluate(octave::tree_evaluator&, int) ../libinterp/parse-tree/pt-assign.cc:101
    #4 0x7fc115957a89 in octave::tree_evaluator::visit_statement(octave::tree_statement&) ../libinterp/parse-tree/pt-eval.cc:3032
    #5 0x7fc1159bd688 in octave::tree_statement::accept(octave::tree_walker&) ../libinterp/parse-tree/pt-stmt.h:124
    #6 0x7fc11595863c in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) ../libinterp/parse-tree/pt-eval.cc:3117
    #7 0x7fc115477f34 in octave::tree_statement_list::accept(octave::tree_walker&) ../libinterp/parse-tree/pt-stmt.h:201
    #8 0x7fc115952369 in octave::tree_evaluator::visit_simple_for_command(octave::tree_simple_for_command&) ../libinterp/parse-tree/pt-eval.cc:2465
    #9 0x7fc1159ad2b6 in octave::tree_simple_for_command::accept(octave::tree_walker&) ../libinterp/parse-tree/pt-loop.h:222
    #10 0x7fc11595784e in octave::tree_evaluator::visit_statement(octave::tree_statement&) ../libinterp/parse-tree/pt-eval.cc:3009
    #11 0x7fc1159bd688 in octave::tree_statement::accept(octave::tree_walker&) ../libinterp/parse-tree/pt-stmt.h:124
    #12 0x7fc11595863c in octave::tree_evaluator::visit_statement_list(octave::tree_statement_list&) ../libinterp/parse-tree/pt-eval.cc:3117
    #13 0x7fc115477f34 in octave::tree_statement_list::accept(octave::tree_walker&) ../libinterp/parse-tree/pt-stmt.h:201
    #14 0x7fc11593ffb1 in octave::tree_evaluator::eval(std::shared_ptr<octave::tree_statement_list>&, bool) ../libinterp/parse-tree/pt-eval.cc:404
    #15 0x7fc1161e611e in octave::interpreter::main_loop() ../libinterp/corefcn/interpreter.cc:1269
    #16 0x7fc1161dc2dc in octave::interpreter::execute() ../libinterp/corefcn/interpreter.cc:797
    #17 0x7fc1173640f9 in octave::interpreter_qobject::execute() ../libgui/src/interpreter-qobject.cc:87
    #18 0x7fc117600fcb in octave::interpreter_qobject::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) libgui/src/moc-interpreter-qobject.cc:95
    #19 0x7fc112bb4275 in QObject::event(QEvent*) (/lib64/libQt5Core.so.5+0x288275)
    #20 0x7fc1134d45f4 in QApplicationPrivate::notify_helper(QObject*, QEvent*) (/lib64/libQt5Widgets.so.5+0x1675f4)


(at exit, among many smaller (~100 bytes) leaks).

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 14 Dec 2020 10:52:01 PM UTC, comment #6: 

So it has nothing to do with ode solver; function handle seems to be leaking memory.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 14 Dec 2020 10:47:54 PM UTC, comment #5: 

OK I see some memory increases when I am running an optimized version. The increase is small, so perhaps it was not seen in debugging version due to a larger memory footprint.

A simple test is just:

for k=1:1e6 ; fx = @(x) 1 ; endfor

that increase RES memory by ~1.8g on my computer.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 14 Dec 2020 10:26:56 PM UTC, comment #4: 

I am running current stable (6.1.1 ; hg id
e359e0fcd6e7 (stable)) compiled with debugging (-O0 -g) and with
memory sanitizer and I do not see any memory increase during the run. Top reports (VIRT / RES / SHR).


20.0t   1.4g 136644 S 102.0   4.5   4:09.59 lt-octave-gui


No report from ASAN either. This is with sundials 5.4.0 on CentOS 8.3.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 14 Dec 2020 09:20:12 PM UTC, comment #3: 

I've take a much more deeper look in the model helping me in localizing the error/problem.
I don't think anymore, that the ode solver is the problem -> maybe you can rename the bug for clarification, it is the used root finding and/or redefintion of a function handle in every solver time step. Please see that attached example.

It's really curios...using a predefined/static function handle, the is no exzessive memory rise during the loop, also with fzero.

Only redefining the function handle inside the loop without any call to fzero increases the memory over the loop count. The same arise with the subsequent call of fzero.


#
for k = 1:1e4
  disp(k)
  # version 1: no memory rise
end
#
fx = @(x) x.^3 + x^2 - 1;
for k = 1:1e4
  disp(k)
  # version 2: no memory rise
  options = optimset('TolX',1e-12);
  fzero(fx,[0,5],options);
end
#
for k = 1:1e4
  disp(k)
  # version 3: memory rise
  fx = @(x) x.^3 + x^2 - 1 - k/1e4;
end
#
for k = 1:1e4
  disp(k)
  # version 4: memory rise
  options = optimset('TolX',1e-12);
  fx = @(x) x.^3 + x^2 - 1 - k/1e4;
  fzero(fx,[0,5],options);
end
#


Anonymous
Sat 12 Dec 2020 07:03:06 PM UTC, comment #2: 

Thanks for your quick reply. I will prepare a representive model/code for demonstration. Please be patient, until it will take until tuesday.
Michael

Anonymous
Fri 11 Dec 2020 02:49:23 PM UTC, comment #1: 

I do not see any memory leaks with "test ode15s" nor "test ode15i". So an actual code that demonstrate the problem would be helpful.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 11 Dec 2020 02:00:23 PM UTC, original submission:  

During running ode-simulations lasting several hours (>8 etc) in version 6.1.0 a continuously increasing amount of memory is needed, it can goes up do >20GB RAM per octave session, altough the generated data after simulation is only around 500MB. My own code only uses standard octave functions, no persistent or global variables are used in my code-scope.

Clearing workspace, function space, persistant variables doesn't help after finishing ode simulation. Octave gui reserves the big amount of memory until its closed and opened again. In Version 5.2.0 these behaviour doesn't happen.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50463:  asan.txt.bz2 added by dasergatskov (58KiB - application/x-bzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by jwe (Updated the item)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-12-20 mmuetzel CategoryGUI Interpreter
        StatusReady For Test Fixed
        Open/ClosedOpen Closed
        Operating SystemMicrosoft Windows Any
        SummaryUnlimited memory usage through octave gui running ode(15s) solver Unlimited memory usage when assigning function handles in a loop
    2020-12-15 dasergatskov Attached File- Added asan.txt.bz2, #50463
    2020-12-15 jwe StatusNeed Info Ready For Test
    2020-12-11 jwe StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code