With the patch I get no crash or error message on Linux (Fedora 39). I'll try on Windows later today or tomorrow

It seems like the catch of any quit_debug_exception need to happen at a lot more places, when unwinding multiple stack frames, than added in https://savannah.gnu.org/bugs/?65056

I've made an patch that seems to address the issue. I can't recreate the problem after applying the patch.

Catch debug exit exceptions in nested bytecode contexts (bug #65547)

When debugging is aborted, and there are nested bytecode interpreters,
the exit debug exception would not be handled properly.

Also, add some missed handling of exit exceptions.

* pt-bytecode-vm-internal.h: Add CATCH_DEBUG_EXCEPTION macro def.
                             Wrap tries with the macro.
                             Add missing CATCH_EXIT_EXCEPTION.
* pt-bytecode-vm.cc: Wrap tries with the CATCH_DEBUG_EXCEPTION macro

(file #55918)

In certain scenarios working in debug mode with the byte-code machine active causes a core dump/crash.
The version ID is 3cd264056d3a (from end of February 2024), so the patch for https://savannah.gnu.org/bugs/?65056 is included.


Steps to reproduce with two function files:

function ret = test1(in);
  in_1 = in * 2;
  if in_1 < 100
    ret = sin(in);
  else
    ret = cos(in);
  endif
endfunction

function ret = test2()
  ret = test1(3);
endfunction

1) Set a breakpoint in test1 (line ret = sin(in);)
2) Run test2 (e.g. via test2() on the command line)
3) change the breakpoint line to ret = cos(in);
4) Quit debug mode (using the square icon in the GUI)
--> system crash / core dump


When the byte-code machine is disabled via _vm_enable_(0) and the same steps are executed, no crash occurs.