It seems like the gui uses a 'quit_debug_exception' to signal that debugging should be canceled, which the bytecode interpreter didn't handle properly. I've attached a patch.

Handle quit_debug_exception in bytecode interpreter (bug #65056)

To allow for escaping debugging in gui properly.

* pt-bytecode-vm.cc: Catch quit_debug_exception, handle in unwind
* pt-bytecode-vm.h: New field 'm_debug_quit_all'
* pt-bytecode.h: New 'DEBUG_QUIT' error type
* pt-eval.cc: Move quit_debug_exception def to header
* pt-eval.h

> On Windows 10 with dev Octave (GUI) from Dec. 14, I get no crash with the example in comment #0. But closing the test2.m file in the editor and then reopening it, and then entering '__vm_enable__' I get an infinite loop with messages:

I think the exception escaped the VM, which probably leads to quite arbitrary execution afterwards. There would e.g. be unpopped Octave stack frames etc. Maybe the try catch in vm::call() should have a 'catch (...)' to catch these kind of bugs easier.

(file #55480)

Changing and then saving a file in debug mode with the byte-code interpreter active causes Octave to crash.

Sample steps to show the issue:
 

function ret = test2(a);
  ret = sin(a);
endfunction

1) enable the byte-code interpreter ("__vm_enable__(1)")
2) Set a breakpoint in test2
3) Call test2(1)
4) When in debug mode, change the line to ret = sin(a+1);
5) Quit the debug mode

When now saving test2 in the editor, octave crashes.

version ("-hgid")
ans = 17f613a4a843