bugGNU Octave - Bugs: bug #65240, The byte-code interpreter gets out...

 
 

bug #65240: The byte-code interpreter gets out of sync and does not release memory in certain cases

Submitter:  Hendrik K <koerhen>
Submitted:  Fri 02 Feb 2024 09:42:52 AM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 27 Feb 2024 01:12:02 PM UTC, comment #4: 

Seems to work. I encountered no more issues and the number crunching results are identical (with or without the byte-code interpreter active).

Thanks for the updated patch.

 Hendrik K <koerhen>
Mon 26 Feb 2024 10:21:05 PM UTC, comment #3: 

I think I found the issue.

Unwinding into an non-bytecode stack frame did not delete the "ignore output"-object, and triggered a sanity check.

+verbose+
evalin ("base","[~] = (@() error ('qwe'))()");
-verbose-
Should be a minimal example. I.e. ignoring from top scope.

This change in the original patch was wrong:
+verbose+
-        output_ignore_data::maybe_delete_ignore_data (*this, 1);
+        output_ignore_data::maybe_delete_ignore_data (*this, 0);
-verbose-

I've attached a fixed patch.

(file #55750)

 Petter <petter>
Fri 23 Feb 2024 03:02:23 AM UTC, comment #2: 

The provided patch causes a regression (i.e. coredump) in a larger program.

panic: VM internal error at 5331, !m_output_ignore_data
fatal: caught signal Aborted -- stopping myself...
Aborted (core dumped)


A trace-log of the last few commands is attached. Obviously the core dump occurred within the logistic_regression function (part of the statistics package).

Notes:
1) The logistic function uses data of a large database which is sliced and diced many times differently
2) The logistics function for certain trials does not converge and throws an error which is captured in the program and  retried in the next slice/dice.

I had another core dump with the same message in another part of the program also possibly related to a thrown error inside a subfunction (sqp function), so 2) may be indicative...



Trace:
sp  : 0x7f47c985de68
bsp : 0x7f47c985ddc8
sp i: 20
sp ii: 162
ip  : 122
code:
data: 0x7f4780a830e0
ids : 0x7f47f6b02d90
fn  : logistic_regression>logistic_regression_likelihood
Next op: 4

Trace:
sp  : 0x7f47c985de60
bsp : 0x7f47c985ddc8
sp i: 19
sp ii: 161
ip  : 123
code:
data: 0x7f4780a830e0
ids : 0x7f47f6b02d90
fn  : logistic_regression>logistic_regression_likelihood
Next op: 7

Trace:
sp  : 0x7f47c985dd78
bsp : 0x7f47c985dae8
sp i: 82
sp ii: 132
ip  : 503
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 52

Trace:
sp  : 0x7f47c985dd58
bsp : 0x7f47c985dae8
sp i: 78
sp ii: 128
ip  : 513
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 41

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 515
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 41

Trace:
sp  : 0x7f47c985dd68
bsp : 0x7f47c985dae8
sp i: 80
sp ii: 130
ip  : 517
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 110

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 519
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 4

Trace:
sp  : 0x7f47c985dd68
bsp : 0x7f47c985dae8
sp i: 80
sp ii: 130
ip  : 520
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 41

Trace:
sp  : 0x7f47c985dd70
bsp : 0x7f47c985dae8
sp i: 81
sp ii: 131
ip  : 522
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 41

Trace:
sp  : 0x7f47c985dd78
bsp : 0x7f47c985dae8
sp i: 82
sp ii: 132
ip  : 524
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 6

Trace:
sp  : 0x7f47c985dd70
bsp : 0x7f47c985dae8
sp i: 81
sp ii: 131
ip  : 526
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 111

Trace:
sp  : 0x7f47c985dd68
bsp : 0x7f47c985dae8
sp i: 80
sp ii: 130
ip  : 527
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 160

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 528
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 24

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 530
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 132

Trace:
sp  : 0x7f47c985dd58
bsp : 0x7f47c985dae8
sp i: 78
sp ii: 128
ip  : 541
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 41

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 543
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 41

Trace:
sp  : 0x7f47c985dd68
bsp : 0x7f47c985dae8
sp i: 80
sp ii: 130
ip  : 545
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 110

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 547
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 4

Trace:
sp  : 0x7f47c985dd68
bsp : 0x7f47c985dae8
sp i: 80
sp ii: 130
ip  : 548
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 41

Trace:
sp  : 0x7f47c985dd70
bsp : 0x7f47c985dae8
sp i: 81
sp ii: 131
ip  : 550
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 41

Trace:
sp  : 0x7f47c985dd78
bsp : 0x7f47c985dae8
sp i: 82
sp ii: 132
ip  : 552
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 6

Trace:
sp  : 0x7f47c985dd70
bsp : 0x7f47c985dae8
sp i: 81
sp ii: 131
ip  : 554
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 111

Trace:
sp  : 0x7f47c985dd68
bsp : 0x7f47c985dae8
sp i: 80
sp ii: 130
ip  : 555
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 164

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 556
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 25

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 558
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 132

Trace:
sp  : 0x7f47c985dd58
bsp : 0x7f47c985dae8
sp i: 78
sp ii: 128
ip  : 561
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 64

Trace:
sp  : 0x7f47c985dd58
bsp : 0x7f47c985dae8
sp i: 78
sp ii: 128
ip  : 563
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 28

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 564
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 72

Trace:
sp  : 0x7f47c985dd58
bsp : 0x7f47c985dae8
sp i: 78
sp ii: 128
ip  : 567
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 41

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 569
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 164

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 572
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 132

Trace:
sp  : 0x7f47c985dd58
bsp : 0x7f47c985dae8
sp i: 78
sp ii: 128
ip  : 575
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 20

Trace:
sp  : 0x7f47c985dd60
bsp : 0x7f47c985dae8
sp i: 79
sp ii: 129
ip  : 577
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 104

Trace:
sp  : 0x7f47c985dd68
bsp : 0x7f47c985dae8
sp i: 80
sp ii: 130
ip  : 579
code:N
data: 0x7f480cd65c00
ids : 0x7f47c6441700
fn  : logistic_regression
Next op: 19


panic: VM internal error at 5331, !m_output_ignore_data
fatal: caught signal Aborted -- stopping myself...
Aborted (core dumped)




 Hendrik K <koerhen>
Sun 04 Feb 2024 11:15:50 PM UTC, comment #1: 

Thanks for the bug report. I am able to replicate the issue.

Replacing one ~ with e.g. 'foo' seems to remove the leak.

I've found the issue to be that if there are multiple stack frames in a row that have an ignore output in them,
the stack frame will leak, due to a "lvalue list" not being deleted.

I've attached a patch that seems to solve the issue.


Fix leak of frames in bytecode interpreter with nested ignored outputs (bug #65240)

If a function had a ignore output, e.g. "[~, b] = foo ()", and so did 'foo',
the stack frames would not be destroyed due to a stray lvalue list not being
deleted.

* pt-bytecode-vm.cc: Rearange to use new field
* pt-bytecode-vm.h: (m_v_lvalue_list_childs) New field
                    Remove unused fields and member functions
                    Move ctor functionality to push_frame()
* bytecode_leaks.m: New test


(file #55667)

 Petter <petter>
Fri 02 Feb 2024 09:42:52 AM UTC, original submission:  

The byte-code interpreter seems to get out of sync in certain cases and it appears that the memory of local variables (after exiting the function) is not released.


Branch: bytecode-interpreter
Hg-id: 3904d7116c8c

(Note: The issue was originally detected with an older self-build version of octave but I included the newest commits to confirm that the issue still exists).
  


With the byte code interpreter disabled, everything works fine:

_vm_enable__(0)
octave:3> test1()
Available system memory: 55642.79 MB
Available system memory: 55646.98 MB
Available system memory: 55650.52 MB
Available system memory: 55650.41 MB
Available system memory: 55650.05 MB
Available system memory: 55655.12 MB
Available system memory: 55654.53 MB
Available system memory: 55654.67 MB
Available system memory: 55654.81 MB
Available system memory: 55658.27 MB
2.4142



With the byte-code interpreter enabled, memory does not get released.


__vm_enable__(1)
octave:3> test1()
Available system memory: 55768.84 MB
Available system memory: 55571.18 MB
Available system memory: 55377.36 MB
Available system memory: 55186.26 MB
Available system memory: 54995.40 MB
Available system memory: 54804.04 MB
Available system memory: 54608.75 MB
Available system memory: 54417.39 MB
Available system memory: 54226.53 MB
Available system memory: 54035.18 MB
2.4142



Test program test1.m


1;

function [a1,a2] = Do_Unique(a)
    [~, i_idx] = unique(a, "first");
    a1 = a(i_idx);
    a2 = sqrt(a1);
endfunction


function [retval] = OptFun(x)
   [~, refresh] = Do_Unique([1,2,1,1]);
   t_data = zeros(5000) .* x;
   retval = sum(refresh);
endfunction

for i=1:10
  a = OptFun(i);
  [~, sd] = memory(); printf("Available system memory: %7.2f MB\n",sd.SystemMemory.Available./1024./1024);
endfor
disp(a);


 Hendrik K <koerhen>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #55750:  33254.patch added by petter (9KiB - text/plain)
file #55667:  33444.patch added by petter (9KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by petter (Updated the item)
  • -email is unavailable- added by koerhen (Submitted the item)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-02-26 petter Attached File- Added 33254.patch, #55750
    2024-02-04 petter Attached File- Added 33444.patch, #55667

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.13-5190.
    Corresponding source code