Thanks for narrowing down the issue. I could replicate the crash. I think it might be due to calling __vm_compile__ with the clear flag. With the clear flag the bytecode is removed straight up with no checks if it is in use or not.

Replacing

__vm_compile__("Snippet_LogStep", "clear");

with

clear Snippet_LogStep

seems to make the problem disappear for me.

I think that maybe the clear flag should be removed. Calling "clear foo" would also remove the bytecode, but garbage collected properly. I think a recompilation with _vm_compile_ have the same issue. Maybe a recompilation call should be a noop instead.

I tried the proposed patches (https://savannah.gnu.org/bugs/?64775#comment5), but the core dump still persists.

 version("-hgid")
ans = bac9c48047cd+

But I think that I found the issue (when trying to condense the code into a shareable version):

a)
For certain parts of the application, I need to deactivate the VM.

(I use the python bridge pythonic to execute some logic in python code. Pythonic does not (yet) seem to be compatible with the VM: When the VM is active, Octave gives a warning and then dies/does not react any longer and the octave process needs to be killed. „warning: Auto-compilation of pyobject failed with message Classdef constructors are not supported by the VM yet“)

b)
I use a code snippet script for logging and execution time monitoring. This snippet script is then automatically compiled by the VM, but causes then a warning later in the application parts where the VM is not active:
„warning: Executing compiled scripts in the VM from an un-compiled function is not supported yet“

c) In order to avoid the warning, I tried to VM clear the snippet code using VM_Compile(…,“clear“) but within the snippet script itself. This was a little dodgy, but the snippet itself seemed to be a nice and especially central place. This then seems to cause the core dump, so obviously the VM is not yet sufficiently robust in such a case.

d)
I now VM clear the snippet each time when I disable the VM and outside the snippet and the core dump does not occur any longer.

e)
Note that the core dump error message in the application is „Assertion `bsp + N_LOCALS() == sp' failed.“).
But in the simplified condensed code, it is just a core dump without this message. But I also got the above error message in the simplified code when changing the code a little and as the changes prevent the core dump in the application, it seems to be have been the issue and I did not investigate further.

Snippet_LogStep.m

if VM_Compile
  disp("VM Clear in Snippet_LogStep");
  vm_compile("Snippet_LogStep", "clear");
endif

if (ParseTime && !isempty(ParseStep))
  printf("Step: %35s ( %5.1f s)\n",ParseStep,toc()-t_start);  t_start = toc();
endif

test2.m

function test2()

  global ParseTime;
  global VM_Compile

  t_start = toc();

  a = 1;
  ParseStep = "test2: Init";
  source Snippet_LogStep.m; # now Snippet_LogStep is VM compiled

  ParseStep = [  "test2 before deactivating the VM : VM is executing " num2str(__vm_is_executing__()) ];
  source Snippet_LogStep.m;

  # need to deactivate vm as pythonic is not supported by the vm
  if VM_Compile
    vm_enable (0,"local");
  endif

  # call octave functions not yet compatible with the VM

  # activate VM again
  if VM_Compile
    vm_enable (1,"local");
  endif

  ParseStep = [  "test2 after reactivating the VM: VM is executing " num2str(__vm_is_executing__()) ];
  source Snippet_LogStep.m;

endfunction

global VM_Compile = true;
global ParseTime = true;


if VM_Compile
  vm_enable (1);
endif
tic;
test2();

VM Clear in Snippet_LogStep
Step:                         test2: Init (   0.0 s)
fatal: caught signal Segmentation fault -- stopping myself...
Segmentation fault (core dumped)

Could you try to apply the two additional patches in https://savannah.gnu.org/bugs/?64775#comment5 ?

Possibly the anonymous function in the trace could have gotten a nargout of -1 from the caller, which made the stack check fail since 'n_returns_callee' would have the wrong value.

@Anonymous: as it seems to crash consistently for you but you're not able to share the code that triggers the crash, could you do a bisection with "hg bisect" pls? (I'm not able to replicate the crash myself.)

Your hd ids have + at the end indicating local changes. Could you also confirm that the local changes aren't interfering?

Well, the working VM version for my octave code is from Saturday which has the HG-ID

version ("-hgid")
ans = 2e933fe82ecb+

The octave VM version causing a core dump for my octave code has the HG-ID

version ("-hgid")
ans = e9eb8975961e+

So I have reason to believe that one of the commits between these two is the culprit and the widget patch seems unlikely...

The commit mentioned by A.R. Burgers  (554a932fc6d0) is timely BEFORE the octave version which does not cause a core dump(e9eb8975961e).
So what he has seen may or may not be related to this bug.

 

changeset:   32416:e9eb8975961e
bookmark:    @
tag:         tip
user:        Petter T. <petter.vilhelm@gmail.com>
date:        Mon Oct 16 21:09:25 2023 +0200
summary:     Don't clone nil values when breaking closure cycles on VM stack (bug #64778)

changeset:   32415:b747729c6695
user:        Petter T. <petter.vilhelm@gmail.com>
date:        Mon Oct 16 07:55:56 2023 +0200
summary:     Support nil value rhs in for loops in VM (bug #64775)

changeset:   32414:385e847edfb7
user:        Petter T. <petter.vilhelm@gmail.com>
date:        Sun Oct 15 16:50:47 2023 +0200
summary:     Fix nargout -1 for classdefs subsrefs from VM (bug #64775)

changeset:   32413:ee5d0bded6d4
user:        Markus Mützel <markus.muetzel@gmx.de>
date:        Sat Oct 14 19:13:12 2023 +0200
summary:     gui: Improve support for fractional scaling in command widget on Windows.

changeset:   32412:2e933fe82ecb
parent:      32411:768df05d04f3
parent:      32409:88ecbd109776
user:        Arun Giridhar <arungiridhar@gmail.com>
date:        Sat Oct 14 08:44:20 2023 -0400
summary:     maint: Merge away extra head

changeset:   32411:768df05d04f3
user:        A.R. Burgers <arburgers@gmail.com> and Fernando Alvarruiz <feralber@upvnet.upv.es>
date:        Sat Apr 15 09:18:39 2023 +0200
summary:     add tests for bug 63841

changeset:   32410:554a932fc6d0
parent:      32407:818698c4f296
user:        Fernando Alvarruiz <feralber@upvnet.upv.es>
date:        Sat Apr 15 10:37:37 2023 +0200
summary:     Fix nargout when calling class subsref method (bug #63841)

changeset:   32409:88ecbd109776
parent:      32407:818698c4f296
parent:      32408:6a1f89bb969c
user:        Nicholas R. Jankowski <jankowski.nicholas@gmail.com>
date:        Fri Oct 13 23:26:21 2023 -0400

There seems to be a regression related to the last VM changes compared to a version built last Saturday (I guess one of the last 2 VM patches is the culprit).
 
Now there is a core dump with the VM active (there are no problems without the VM and the code finishes with the correct result. And there were no problems with the VM active from the version built from Saturday).

Current failing version:

version ("-hgid")
ans = e9eb8975961e+

octave-gui: ./../libinterp/parse-tree/pt-bytecode-vm.cc:5888: octave_value_list octave::vm::execute_code(const octave_value_list&, int): Assertion `bsp + n_locals_callee + n_returns_callee - 1 == sp' failed.
fatal: caught signal Aborted -- stopping myself...
Aborted (core dumped)

The last VM trace statements are below

Trace:
sp  : 0x7f6456d94bc0
bsp : 0x7f6456d94b00
sp i: 24
sp ii: 45
ip  : 16
code: 0x7f652ecd8f40
data: 0x7f652ef045c0
ids : 0x7f652eb14820
fn: fn_callback
Next op: 12

Trace:
sp  : 0x7f6456d94bc8
bsp : 0x7f6456d94b00
sp i: 25
sp ii: 46
ip  : 17
code: 0x7f652ecd8f40
data: 0x7f652ef045c0
ids : 0x7f652eb14820
fn: fn_callback
Next op: 103

Trace:
sp  : 0x7f6456d94bd0
bsp : 0x7f6456d94b00
sp i: 26
sp ii: 47
ip  : 19
code: 0x7f652ecd8f40
data: 0x7f652ef045c0
ids : 0x7f652eb14820
fn: fn_callback
Next op: 2

Trace:
sp  : 0x7f6456d94bd8
bsp : 0x7f6456d94b00
sp i: 27
sp ii: 48
ip  : 21
code: 0x7f652ecd8f40
data: 0x7f652ef045c0
ids : 0x7f652eb14820
fn: fn_callback
Next op: 41

Trace:
sp  : 0x7f6456d94be0
bsp : 0x7f6456d94b00
sp i: 28
sp ii: 49
ip  : 23
code: 0x7f652ecd8f40
data: 0x7f652ef045c0
ids : 0x7f652eb14820
fn: fn_callback
Next op: 41

Trace:
sp  : 0x7f6456d94be8
bsp : 0x7f6456d94b00
sp i: 29
sp ii: 50
ip  : 25
code: 0x7f652ecd8f40
data: 0x7f652ef045c0
ids : 0x7f652eb14820
fn: fn_callback
Next op: 96

Trace:
sp  : 0x7f6476bcd1e0
bsp : 0x7f6476bcd1b8
sp i: 5
sp ii: 5
ip  : 4
code: 0x7f647653f400
data: 0x7f647653ea50
ids : 0x7f644d0d03c0
fn  : fn: fnname@<anonymous>
Next op: 20

Trace:
sp  : 0x7f6476bcd1e8
bsp : 0x7f6476bcd1b8
sp i: 6
sp ii: 6
ip  : 6
code: 0x7f647653f400
data: 0x7f647653ea50
ids : 0x7f644d0d03c0
fn  : fn: fnname@<anonymous>
Next op: 41

Trace:
sp  : 0x7f6476bcd1f0
bsp : 0x7f6476bcd1b8
sp i: 7
sp ii: 7
ip  : 8
code: 0x7f647653f400
data: 0x7f647653ea50
ids : 0x7f644d0d03c0
fn  : fn: fnname@<anonymous>
Next op: 41

Trace:
sp  : 0x7f6476bcd1f8
bsp : 0x7f6476bcd1b8
sp i: 8
sp ii: 8
ip  : 10
code: 0x7f647653f400
data: 0x7f647653ea50
ids : 0x7f644d0d03c0
fn  : fn: fnname@<anonymous>
Next op: 150

Trace:
sp  : 0x7f6476bcd1f8
bsp : 0x7f6476bcd1b8
sp i: 8
sp ii: 8
ip  : 12
code: 0x7f647653f400
data: 0x7f647653ea50
ids : 0x7f644d0d03c0
fn  : fn: fnname@<anonymous>
Next op: 4

Trace:
sp  : 0x7f6476bcd1e8
bsp : 0x7f6476bcd1b8
sp i: 6
sp ii: 6
ip  : 14
code: 0x7f647653f400
data: 0x7f647653ea50
ids : 0x7f644d0d03c0
fn  : fn: fnname@<anonymous>
Next op: 144

octave-gui: ./../libinterp/parse-tree/pt-bytecode-vm.cc:5888: octave_value_list octave::vm::execute_code(const octave_value_list&, int): Assertion `bsp + n_locals_callee + n_returns_callee - 1 == sp' failed.
fatal: caught signal Aborted -- stopping myself...
Aborted (core dumped)