bugGNU Octave - Bugs: bug #67040, undefined behavior in bytecode VM

 
 

bug #67040: undefined behavior in bytecode VM

Submitter:  Dmitri A. Sergatskov <dasergatskov>
Submitted:  Mon 21 Apr 2025 04:25:10 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * bytecode-evaluator Release: 
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 11 Jul 2025 02:28:27 PM UTC, comment #10: 

I realize now that my comments are off-topic here and should go to bug #66790 instead. I will move them there. Apologies.

Fernando <tutissanalio>
Fri 11 Jul 2025 09:53:31 AM UTC, comment #9: 

Sorry, the vm compilation error avoided by change in comment #7 was wrong. The correct one is:

error: VM error 3220: internal VM compiler consistency check failed, m_ignored_ip_start


(The other error happens if you change the name of the file.)

Fernando <tutissanalio>
Fri 11 Jul 2025 09:44:57 AM UTC, comment #8: 

I forgout to mention this was with Octave 11.0.0 (hg id: 67e19a2175ab)

Fernando <tutissanalio>
Fri 11 Jul 2025 09:42:43 AM UTC, comment #7: 

I played I little with fromJSON.m from io 2.7.0, but without installing the io package. I found that this change:


@@ -164,7 +164,8 @@ function [btag, block, remain] = _get_block_ (str);

     if ~( numel(idx) > 1 && ( close=-idx(2) ) > 0 );    # unless simple (non-nested) block...

-        idx   = idx([~,i] = sort(abs(idx)));  # sort idx by abs(idx)
+        [~,i] = sort(abs(idx));  # sort idx by abs(idx)
+        idx   = idx(i);

         if( ( ci=find(0 == cumsum(sign(idx)), 1) )  );
           close = -idx(ci);


eliminates the VM compilation error:

error: VM error 3080: internal VM compiler consistency check failed, external_offset == i


It seems that it also passes all the tests in the file (commenting out the check of vm enabled at the start of the function):

octave:70> test fromJSON
PASSES 152 out of 152 tests


Fernando <tutissanalio>
Tue 22 Apr 2025 10:08:36 PM UTC, comment #6: 

@Arun:
Just FTR, which Octave release, which io package? (released 2.6.4 or repo)
I suppose you ran 'pkg test io' with dev Octave and plain io-2.6.4.

In the io dev repo I simply disabled fromJSON if the bytecode interpreter VM is active - yes that gives a lot of FAILs :-)
I tried to disable the VM warnings in fromJSON.m but AFAICS there's no Octave warning ID that can be used to that end. Supposedly it can be added quite simply but as the VM is still experimental and my priority is trying to get a new io package out of the door I didn't bother to even look.

For the rest I interpret your comment #5 as confirming what I wrote in comment #4.

BTW, let's keep this bug report for undefined VM behavior, let's discuss io package JSON functions over in bug #66790.

Philip Nienhuis <philipnienhuis>
Group Member
Tue 22 Apr 2025 04:40:37 PM UTC, comment #5: 


Summary:

  PASS                              252
  FAIL                                5


For fromJSON, the bytecode interpreter failed an internal conssitency check, likely from the unconventional code in fromJSON:


>>>>> processing .../io-2.6.4/fromJSON.m
***** warning <invalid SARRAY> fromJSON('',struct);

!!!!! warning failed.
Expected <invalid SARRAY>, but got <auto-compilation of _get_block_ failed with message VM error 3220: internal VM compiler consistency check failed, m_ignored_ip_start>

***** warning <unclosed>    assert(fromJSON('{a:3,b:4'),struct("a",3,"b",4));
!!!!! warning failed.
Expected <unclosed>, but got <auto-compilation of _get_block_ failed with message VM error 3220: internal VM compiler consistency check failed, m_ignored_ip_start>

***** warning <invalid> assert( fromJSON('[😊,4]'), [NaN, 4]); % convert to NaN
!!!!! warning failed.
Expected <invalid>, but got <auto-compilation of _get_block_ failed with message VM error 3220: internal VM compiler consistency check failed, m_ignored_ip_start>


For toJSON, it was just missing Java. Those tests should probably be run only if Java is present:


>>>>> processing .../io-2.6.4/toJSON.m
***** test  ## unknown class
 obj = javaObject ("java.math.BigDecimal", 1.0);
  assert(toJSON(obj), '"[java.math.BigDecimal]"');
!!!!! test failed
javaObject: support for Java was unavailable or disabled when Octave was built
***** test  ## struct with java object
 obj = javaObject ("java.math.BigDecimal", 1.0);
  assert(toJSON(struct('a',obj)), '{"a":"[java.math.BigDecimal]"}');
!!!!! test failed
javaObject: support for Java was unavailable or disabled when Octave was built


Arun Giridhar <arungiridhar>
Group Member
Tue 22 Apr 2025 09:30:16 AM UTC, comment #4: 

Thanks for the explanation Dmitri.

Some things:

1. My code ==> code by a contributor (maybe call it "io code").

2. What I meant wiht my remark about code quality is constructs like these (a simple example from e.g., fromJSON.m):

:
  uint8(str(1)) > 126 && (str = str( find(str<127,1) : end ));   # drop leading encoding chars
:

where instead of an if block, just a statement line comprising a boolean shortcut construct is used to decide whether or not to do an assignment. Such constructs are all over the place in toJSON/fromJSON and even appear as arguments in function calls where sometimes even multiple assignments are made. I wouldn't be surprised if that sort of code trips the VM "parser", hence the

warning: auto-compilation of fromJSON failed with message VM error 3225: internal VM compiler consistency check failed, m_ignored_ip_start

warnings. I'd rather see traditional 'if' blocks (if needed, nested) with clear assignments inside. But that may be just me :-)
BTW in addition this example statement will trigger a 'regular' Octave warning if 'str' happens to be not char.

But OK all of this has little to do with the UBSAN results you showed below.

Back to VM:
I had the impression that it only worked on m-code, not binary code. But apparently I was wrong, as cell2csv in your comment #0 verbatim block is a binary (compiled) function.
There's some comfort that at least the results match ... (being right for a wrong reason?)

To repeat myself, I never hit serious issues with the VM up til now, I've been using it all the time from the moment it got implemented. It has been quite robust and gave a significant performance boost. I'd be more worried if the VM provokes memory leaks.

Bug #65420, bug #65304 and bug #65308 seem to contain not-yet-applied patches; AFAIR I've applied them in my builds.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 21 Apr 2025 11:27:48 PM UTC, comment #3: 

https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html

There are similar ASAN (address sanitizer) and TSAN (thread sanitizer).

I susually use clang when I suse those since they have a compiler support and pretty much all you have to do is to add "-fsanitize=undefined" to the "clang". With gcc one needs to link to external libraries so it gets a little messier.

I am pretty sure the problems are with VM not with your code. Your code just triggered it. I loooked at the "call-stack.cc" and saw something strange (which I thought was a type), so I "fixed" it with:

@@ -566,7 +566,7 @@
   else if (fcn->is_anonymous_function())
     {
       std::shared_ptr<stack_frame> new_frame
-        = stack_frame::create_bytecode (m_evaluator, fcn, vm,
+        = stack_frame::create_bytecode_anon (m_evaluator, fcn, vm,
                                         new_frame_idx,
                                         parent_link, static_link,
+                                        nullptr,
                                         nargout, nargin);

(the thought was that "anonymous_function" should call "bytecode_anon"), but that did not change anything.
So the problem is deeper than that.

On dev branch I do not see any UBSAN errors with your code.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 21 Apr 2025 09:57:30 PM UTC, comment #2: 

Just curious, & I have little or no clue about UBSAN: is that analyzer compatible with the coding style etc. in the bytecode interpreter VM?
From what I saw the latter comprises a lot of assembly-like machine code level shortcuts. In my impression maybe that could make code analyzers rapidly loose track of what is going on.

Over in bug #66790, you wrote:

> Is Petter still checking bugs here, or he gave up on us?

IIRC he once wrote he made the VM during paternity leave. Assuming that is over by now, and having raised kids myself, I'm quite confident he'll have little time left for Octave these days :-)
AFAIK there are still one or two older bug reports lingering on; IIRC a fix for one of them was supplied by someone else but never pushed (but I think have it in my builds).
But you can try to reach him; his email can surely be found in his contributions, or here on Savannah.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 21 Apr 2025 09:26:01 PM UTC, comment #1: 

Hmmm, I've been quite busy lately preparing a new io release and in that course, just this evening, I built bleeding edge Octave-11 for Linux and Windows.

Indeed the VM (bytecode interpreter) has a hard time coping with the code quality in esp. 'fromJSON.m' (a contributed function BTW).  But not quite as bad as what you are showing.
AFAICS that io function is the only one where I ever have seen issues with the VM. Since I retired half a year ago I don´t use Octave daily anymore but still enough to have faith in the VM. Until solidly proven otherwise, of course.

On Fedora 41 I see (with io package as currently in repo, I have more fixes awaiting pushing to repo):

GNU Octave, version 11.0.0
Copyright (C) 1993-2025 The Octave Project Developers.
This is free software; see the source code for copying conditions.
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  For details, type 'warranty'.

Octave was configured for "x86_64-pc-linux-gnu".

Home page:            https://octave.org
Support resources:    https://octave.org/support
Improve Octave:       https://octave.org/get-involved

For changes from previous versions, type 'news'.

>> pkg list
Package Name   | Version | Installation directory
---------------+---------+-----------------------
      doctest  |   0.8.0 | .../.local/share/octave/api-v60/packages/doctest-0.8.0
 econometrics  |   1.1.2 | .../.local/share/octave/api-v60/packages/econometrics-1.1.2
generate_html  |   0.3.3 | .../.local/share/octave/api-v60/packages/generate_html-0.3.3
           io  |   2.6.5 | /home/philip/.local/share/octave/api-v60/packages/io-2.6.5
        optim  |   1.6.2 | .../philip/.local/share/octave/api-v60/packages/optim-1.6.2
       signal  |   1.4.6 | .../philip/.local/share/octave/api-v60/packages/signal-1.4.6
>> __vm_enable__ ()
ans = 1
>> pkg load io
>> test toJSON
PASSES 56 out of 56 tests
>> test fromJSON
warning: auto-compilation of fromJSON failed with message VM error 3225: internal VM compiler consistency check failed, m_ignored_ip_start
warning: called from
    __test__ at line 3 column 2
    test at line 685 column 11


*****************************************************************
| fromJSON.m is incompatible with bytecode interpreter.         |
| Disable bytecode interpreter with command '__vm_enable__ (0)' |
| and try again.                                                |
*****************************************************************

***** test      ##      input validation
 assert( fromJSON(),[]); % ok, reference
!!!!! test failed
Invalid call to cond.  Correct usage is:

 -- C = cond (A)
 -- C = cond (A, P)
>> __vm_enable__ (0)
>> test fromJSON
PASSES 152 out of 152 tests

and now a full package test:

>> __vm_enable__ (1)
>> pkg test io
Testing functions in package 'io':

Integrated test scripts:
 :
 ;

Summary:

  PASS                              118
  FAIL                              139

See the file /home/philip/fntests.log for additional details.

34 (of 39) .m files have no tests.

... where those 139 FAILs are all from fromJSON.m.

With disabled VM all 257 tests pass.

So I suppose there's something in your build that makes this go awry ?

BTW I got the same results on Windows (with same io package state) but with an Octave-11 build from a few days ago. Tomorrow I'll try a newer Windows crossbuild but I expect no different behavior.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 21 Apr 2025 04:25:10 PM UTC, original submission:  

Doing "pkg load io" on bytecode-interpreter branch of octave I see:

octave:1> pkg test io
Testing functions in package 'io':

Integrated test scripts:

                                                                                  [ CPU    /  CLOCK ]
../libinterp/corefcn/stack-frame.cc:3053:37: runtime error: downcast of address 0x7fa0b75db8e0 which does not point to an object of type 'bytecode_fcn_stack_frame'
0x7fa0b75db8e0: note: object is of type 'octave::bytecode_nested_fcn_stack_frame'
 00 00 00 00  c0 e9 16 fe a1 7f 00 00  08 91 0e f4 a1 7f 00 00  00 00 00 00 ff ff ff ff  ff ff ff ff
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'octave::bytecode_nested_fcn_stack_frame'
    #0 0x7fa1fc86c6ce in octave::stack_frame::create_bytecode(octave::tree_evaluator&, octave_user_code*, octave::vm&, unsigned long, std::shared_ptr<octave::stack_frame> const&, std::shared_ptr<octave::stack_frame> const&, int, int) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/corefcn/stack-frame.cc:3053:37
    #1 0x7fa1fbf6b8c8 in octave::call_stack::push(octave::vm&, octave_user_function*, int, int) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/corefcn/call-stack.cc:564:11
    #2 0x7fa1fbd1d0a7 in octave::vm::execute_code(octave_value_list const&, int) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/parse-tree/pt-bytecode-vm.cc:2604:17
    #3 0x7fa1fbdad57b in octave::vm::call(octave::tree_evaluator&, int, octave_value_list const&, octave_user_code*, std::shared_ptr<octave::stack_frame>) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/parse-tree/pt-bytecode-vm.cc:8615:14
    #4 0x7fa1fba840ef in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/octave-value/ov-usr-fcn.cc:613:12
...<deleted>...
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../libinterp/corefcn/stack-frame.cc:3053:37
../libinterp/corefcn/stack-frame.cc:3056:10: runtime error: member call on address 0x7fa0b75db8e0 which does not point to an object of type 'octave::bytecode_fcn_stack_frame'
0x7fa0b75db8e0: note: object is of type 'octave::bytecode_nested_fcn_stack_frame'
 00 00 00 00  c0 e9 16 fe a1 7f 00 00  08 91 0e f4 a1 7f 00 00  00 00 00 00 ff ff ff ff  ff ff ff ff
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'octave::bytecode_nested_fcn_stack_frame'
    #0 0x7fa1fc86c717 in octave::stack_frame::create_bytecode(octave::tree_evaluator&, octave_user_code*, octave::vm&, unsigned long, std::shared_ptr<octave::stack_frame> const&, std::shared_ptr<octave::stack_frame> const&, int, int) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/corefcn/stack-frame.cc:3056:10
    #1 0x7fa1fbf6b8c8 in octave::call_stack::push(octave::vm&, octave_user_function*, int, int) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/corefcn/call-stack.cc:564:11
    #2 0x7fa1fbd1d0a7 in octave::vm::execute_code(octave_value_list const&, int) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/parse-tree/pt-bytecode-vm.cc:2604:17
    #3 0x7fa1fbdad57b in octave::vm::call(octave::tree_evaluator&, int, octave_value_list const&, octave_user_code*, std::shared_ptr<octave::stack_frame>) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/parse-tree/pt-bytecode-vm.cc:8615:14
    #4 0x7fa1fba840ef in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/octave-value/ov-usr-fcn.cc:613:12
    #5 0x7fa1fbd1edca in octave::vm::execute_code(octave_value_list const&, int) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/parse-tree/pt-bytecode-vm.cc:2619:50
    #6 0x7fa1fbdad57b in octave::vm::call(octave::tree_evaluator&, int, octave_value_list const&, octave_user_code*, std::shared_ptr<octave::stack_frame>) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/parse-tree/pt-bytecode-vm.cc:8615:14
    #7 0x7fa1fba840ef in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/octave-value/ov-usr-fcn.cc:613:12
...<deleted>.SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../libinterp/corefcn/stack-frame.cc:3056:10
  ..packages/io-2.6.5/x86_64-pc-linux-gnu-api-v60/cell2csv.cc-tst  pass    2/2    [ 0.029s /  0.029s]
...<deleted>...
Please help improve Octave by contributing tests for these files
(see the list in the file /home/dmitri/src/octave-io/target/fntests.log).

/usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/shared_ptr_base.h:427:9: runtime error: member call on address 0x7fa0b75db8e0 which does not point to an object of type 'octave::bytecode_nested_fcn_stack_frame'
0x7fa0b75db8e0: note: object is of type 'octave::bytecode_fcn_stack_frame'
 00 00 00 00  08 e7 16 fe a1 7f 00 00  08 91 0e f4 a1 7f 00 00  00 00 00 00 ff ff ff ff  ff ff ff ff
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'octave::bytecode_fcn_stack_frame'
    #0 0x7fa1fc8a271a in std::_Sp_counted_ptr<octave::bytecode_nested_fcn_stack_frame*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/shared_ptr_base.h:427:9
    #1 0x7fa1fb781105 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use() /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/shared_ptr_base.h:174:2
    #2 0x7fa1fbdb7fac in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/shared_ptr_base.h:1069:11
    #3 0x7fa1fbdb7fac in std::__shared_ptr<octave::stack_frame, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/shared_ptr_base.h:1531:31
    #4 0x7fa1fbdb7fac in void std::_Destroy<std::shared_ptr<octave::stack_frame>>(std::shared_ptr<octave::stack_frame>*) /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/stl_construct.h:166:19
    #5 0x7fa1fbdb7fac in void std::_Destroy<std::shared_ptr<octave::stack_frame>*>(std::shared_ptr<octave::stack_frame>*, std::shared_ptr<octave::stack_frame>*) /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/stl_construct.h:212:4
    #6 0x7fa1fbda6e08 in void std::_Destroy<std::shared_ptr<octave::stack_frame>*, std::shared_ptr<octave::stack_frame>>(std::shared_ptr<octave::stack_frame>*, std::shared_ptr<octave::stack_frame>*, std::allocator<std::shared_ptr<octave::stack_frame>>&) /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/alloc_traits.h:1045:7
    #7 0x7fa1fbda6e08 in std::vector<std::shared_ptr<octave::stack_frame>, std::allocator<std::shared_ptr<octave::stack_frame>>>::~vector() /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/bits/stl_vector.h:802:2
    #8 0x7fa1fbda6e08 in octave::vm::~vm() /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/parse-tree/pt-bytecode-vm.cc:7534:1
    #9 0x7fa1fbdad6bb in octave::vm::call(octave::tree_evaluator&, int, octave_value_list const&, octave_user_code*, std::shared_ptr<octave::stack_frame>) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/parse-tree/pt-bytecode-vm.cc:8640:1
    #10 0x7fa1fba840ef in octave_user_function::call(octave::tree_evaluator&, int, octave_value_list const&) /home/dmitri/src/dev/octave/clang_ubsan/../libinterp/octave-value/ov-usr-fcn.cc:613:12
...<deleted>...


Full log is attached.
(file bc_VM_UBSAN_log.txt)

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #57153:  bc_VM_UBSAN_log.txt added by dasergatskov (16KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tutissanalio (Posted a comment)
  • -email is unavailable- added by arungiridhar (Posted a comment)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by dasergatskov (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2025-04-21 philipnienhuis Carbon-Copy- Added philipnienhuis
    2025-04-21 dasergatskov Attached File- Added bc_VM_UBSAN_log.txt, #57153

    Back to the top

    Powered by Savane 3.15-e6e5.
    Corresponding source code