bugGNU Octave - Bugs: bug #59599, movmean crashes Octave

 
 

bug #59599: movmean crashes Octave

Submitter:  Evangelos Rozos <vrozos>
Submitted:  Wed 02 Dec 2020 06:31:35 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 6.1.0 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 09 Dec 2020 12:33:11 PM UTC, comment #27: 

I'm happy this fixed it.

Closing report.

Markus Mützel <mmuetzel>
Group administrator
Wed 09 Dec 2020 07:20:27 AM UTC, comment #26: 

I downloaded the latest build (octave-2020-12-08-00-28-w32). BINGO!


>> std (movmean (rand(40000,1), 365*20))
error: out of memory or dimension too large for Octave's index type
error: called from
    movslice at line 89 column 10
    movfun at line 209 column 29
    movmean at line 137 column 5


Evangelos Rozos <vrozos>
Mon 07 Dec 2020 03:10:40 PM UTC, comment #25: 

Nightly builds from 2020/Dec/07 or later should have been built with this change.

Markus Mützel <mmuetzel>
Group administrator
Sat 05 Dec 2020 09:47:52 PM UTC, comment #24: 

I pushed a patch to MXE Octave that enables exception handling in LLVM here:
https://hg.octave.org/mxe-octave/rev/f367e764ca55

Maybe you could test if that changes anything when the buildbots built with that change.

Markus Mützel <mmuetzel>
Group administrator
Sat 05 Dec 2020 07:55:02 PM UTC, comment #23: 

@mmuetzel, It crashes too.

comment #18:

> Could you please try if you see the same crash with a recent nightly build from Kai's buildbots? You could download those from here:
> https://buildbot.octave.space/#/download

Evangelos Rozos <vrozos>
Sat 05 Dec 2020 07:48:44 PM UTC, comment #22: 

Thanks for testing.  I think this confirms that the lead to follow up is the one Markus identified in LLVM.  It shouldn't be catching these errors in place of the standard C++ libraries.

Rik <rik5>
Group administrator
Sat 05 Dec 2020 07:00:21 PM UTC, comment #21: 

@rik5

>> log2 (sizemax ())
ans = 31.000
>> x = rand (2.1e9, 1);
error: out of memory or dimension too large for Octave's index type


No crash.

Evangelos Rozos <vrozos>
Sat 05 Dec 2020 05:54:42 PM UTC, comment #20: 

@Evangelos: I think you're system is working with 32-bit integer pointers, but can you run this to verify?


log2 (sizemax ())


This will report 31 or 63 depending on whether you have a 32-bit or 64-bit octave_idx_type.

Assuming it is the smaller, try


x = rand (2.1e9, 1);


That number is just below sizemax() for 32-bit octave_idx_type, but would require ~16GB of memory which doesn't sound possible on your 2GB machines and so this should throw an error when it can't get that much memory.

Rik <rik5>
Group administrator
Sat 05 Dec 2020 11:34:35 AM UTC, comment #19: 

This looks like it could be related:
https://llvm.org/docs/CMake.html

> LLVM_ENABLE_EH:BOOL
> Build LLVM with exception-handling support. This is necessary if you wish to link against LLVM libraries and make use of C++ exceptions in your own code that need to propagate through LLVM code. Defaults to OFF.


At the moment, we don't set that switch which means that exception handling is off in LLVM.
I'll try if I can get LLVM to build with that switch set to `On`.

Markus Mützel <mmuetzel>
Group administrator
Sat 05 Dec 2020 10:36:24 AM UTC, comment #18: 

@vrozos: John D recently updated the Windows builds to use GCC 10.2 (GCC 9.3 earlier). There is a chance - albeit slim admittedly - that this might have changed something.
Could you please try if you see the same crash with a recent nightly build from Kai's buildbots? You could download those from here:
https://buildbot.octave.space/#/download

Markus Mützel <mmuetzel>
Group administrator
Sat 05 Dec 2020 09:06:09 AM UTC, comment #17: 


x = rand (5.7e7,1);


works OK, returns a vector stored to x.


x = rand (5.7e9,1);
error: out of memory or dimension too large for Octave's index type




comment #16:

> @Evangelos: Could you try this additional test?
>


> x = rand (5.7e7,1);


>
> This will test if the Array class is also affected.

Evangelos Rozos <vrozos>
Fri 04 Dec 2020 10:54:30 PM UTC, comment #16: 

Good results (in that they at least point towards further actions to take).  It's possible that there are two problems here.  The immediate issue is LLVM and whether they are stealing errors intended for Octave.

The second issue is that the idx_vector code may be unprotected against a thrown error while the Array class is better protected.

@Evangelos: Could you try this additional test?


x = rand (5.7e7,1);


This will test if the Array class is also affected.

Rik <rik5>
Group administrator
Fri 04 Dec 2020 09:20:00 PM UTC, comment #15: 

I think LLVM is installing its own error handler

John Donoghue <lostbard>
Group Member
Fri 04 Dec 2020 07:53:10 PM UTC, comment #14: 

Thanks for the test.

So it looks like the exception is correctly caught inside Octave only if it doesn't (indirectly) use LLVM.dll? What the heck?

CC'ing John D because I hope he better understands how libraries work on Windows...

Markus Mützel <mmuetzel>
Group administrator
Fri 04 Dec 2020 06:04:40 PM UTC, comment #13: 

@mmuetzel, I renamed it into _opengl32.dll and this time no crash! I get:


> std (movmean (rand(40000,1), 365*10));
error: out of memory or dimension too large for Octave's index type
error: called from
    movfun>movfun_oncol at line 299 column 10
    movfun at line 282 column 14
    movmean at line 137 column 5



@rik5, with the following (opengl32 renamed or not), I get:


> x = ones (1e50, 1);
error: out of memory or dimension too large for Octave's index type





comment #12:

> @Evangelos Rozos: Could you please try if the crash still happens if you renamed opengl32.dll? That file should be installed at "C:\var\OCTAVE~2.0-W\mingw32\bin\opengl32.dll" for you.
> If it does, could you please try to get the backtrace following the steps in comment #5 again?

Evangelos Rozos <vrozos>
Fri 04 Dec 2020 08:14:24 AM UTC, comment #12: 

I agree that "movmean" should probably be implemented more memory-efficiently.
However, it is probably only a symptom of the underlying issue. Something seems to be wrong with handling the bad_alloc exception in core Octave that causes this function to crash Octave.

I'm wondering why the exception handling is dispatched to functions in LLVM.dll (and not a standard library). Maybe that is part of the reason this isn't working. LLVM is used by the software OpenGL driver.

@Evangelos Rozos: Could you please try if the crash still happens if you renamed opengl32.dll? That file should be installed at "C:\var\OCTAVE~2.0-W\mingw32\bin\opengl32.dll" for you.
If it does, could you please try to get the backtrace following the steps in comment #5 again?

Markus Mützel <mmuetzel>
Group administrator
Fri 04 Dec 2020 12:51:20 AM UTC, comment #11: 

Well, there are a few problems here.  It starts with movmean().  Apparently, this function is designed for small window sizes.  Using your clue from version 5.1 where Octave threw an OOM error.


movfun>movfun_oncol at line 295 column 10


The line in movfun.m is


  y(C,:) = fcn (x(slcidx));


I set a breakpoint at this line and then ran


octave:2> ypred = rand (1, 40e3);
octave:3> std (movmean (ypred, 365*4))


Inspecting the variables, C is 40e3 - window_size or ~38e3.  That's not a problem.  However, slcidx is huge.


debug> size (slcidx)
ans =

    1460   38541

debug> sizeof (slcidx)
ans = 4.5016e+08


That is ~450 MB.  Either something more clever needs to be done internally to movfun to reduce the size of that variable or the function might need to be converted to C++ so that it can use loops rather than index vectors.

But, the immediate question is why this doesn't just throw an error and return the user to the Octave prompt when the new() operator fails to allocate memory.

@Evangelos: Does a straightforward impossibility like


x = ones (1e50, 1);


throw an OOM error or does this immediately crash too?


Rik <rik5>
Group administrator
Thu 03 Dec 2020 02:51:43 PM UTC, comment #10: 

Thank you.

IIUC, a bad_alloc exception is thrown which isn't caught by Octave.
It looks like the error is thrown by `idx_vector::idx_vector_rep::idx_vector_rep<double>(Array<double> const&)`. That is probably here:
https://hg.savannah.gnu.org/hgweb/octave/file/1dbeca38c776/liboctave/array/idx-vector.cc#l325

I hope someone who understands at what level that error should have been caught is reading this. :-)

Markus Mützel <mmuetzel>
Group administrator
Thu 03 Dec 2020 01:55:15 PM UTC, comment #9: 

Please see attached file (gdblog.txt).


comment #5:

> Could you please try again, but before you continue execution with "c", try to add a breakpoint in that function with the (gdb) command "b abort".
> Does the breakpoint get hit? What does "bt" return at that point?


(file #50387)

Evangelos Rozos <vrozos>
Thu 03 Dec 2020 12:53:42 PM UTC, comment #8: 

What you are seeing on Linux is probably the OOM killer in action.
There is an open bug report for that. But I can't find it at the moment.
There is probably not much we can do to avoid the issue, unfortunately.

Could you please try the steps described in comment #5 on Windows?

Markus Mützel <mmuetzel>
Group administrator
Thu 03 Dec 2020 12:19:21 PM UTC, comment #7: 

comment #6:

> std (movmean (rand(40000,1), 365*10))


Systems with large memory (my systems are 2GB) may require larger values than the numbers above (i.e. 40000 and 365*10) to crash.

Evangelos Rozos <vrozos>
Thu 03 Dec 2020 12:14:05 PM UTC, comment #6: 

The same with GNU Octave, version 5.2.0, in Ubuntu 20.04.1 LTS


octave:1> std (movmean (rand(40000,1), 365*10))
[Thread 0x7fffefa4d700 (LWP 39893) exited]

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb) bt
No stack.


Evangelos Rozos <vrozos>
Thu 03 Dec 2020 12:11:57 PM UTC, comment #5: 

Thank you for testing.
IIRC, a program might exit with code 3 when "abort" is called by some function.

Could you please try again, but before you continue execution with "c", try to add a breakpoint in that function with the (gdb) command "b abort".
Does the breakpoint get hit? What does "bt" return at that point?

Markus Mützel <mmuetzel>
Group administrator
Thu 03 Dec 2020 11:53:29 AM UTC, comment #4: 

comment #3:

>
> Could you please check if you can catch the error with "gdb"?
> For this, you could run "__debug_octave__" at the Octave prompt and continue Octave's execution with "c" at the "(gdb)" prompt.
> Back in Octave, try to execute the commands that lead to the crash.
> When it happens, return to the gdb prompt and try to get the backtrace with "bt".
> What does that display?
>
>


(gdb) c
Continuing.
Reading symbols from C:\var\OCTAVE~2.0-W\mingw32\bin\octave-gui.exe...
(No debugging symbols found in C:\var\OCTAVE~2.0-W\mingw32\bin\octave-
(gdb) c
Continuing.
[Thread 7712.0x10e0 exited with code 0]
[Thread 7712.0x8e4 exited with code 0]
[Thread 7712.0x1970 exited with code 0]
[New Thread 7712.0x354]
[Thread 7712.0xce4 exited with code 0]

Then back to Octave and give:

std (movmean (rand(40000,1), 365*10))


It crashes and in gdb I get the following:

[Thread 7712.0x20e4 exited with code 0]
[Thread 7712.0x25ac exited with code 0]
[Thread 7712.0x10c8 exited with code 3]
[Thread 7712.0xd00 exited with code 3]
[Thread 7712.0x1124 exited with code 3]
[Thread 7712.0xed0 exited with code 3]
[Thread 7712.0x354 exited with code 3]
[Thread 7712.0x2018 exited with code 3]
[Thread 7712.0x1e30 exited with code 3]
[Thread 7712.0x770 exited with code 3]
[Inferior 1 (process 7712) exited with code 03]
(gdb) bt
No stack.

Evangelos Rozos <vrozos>
Thu 03 Dec 2020 10:41:43 AM UTC, comment #3: 

It might be or might not be an over-committing issue.
IIUC what is described in [1], Windows will over-commit memory on the stack (but not on the heap?).
Maybe something has changed in how we allocate memory between versions...

Could you please check if you can catch the error with "gdb"?
For this, you could run "__debug_octave__" at the Octave prompt and continue Octave's execution with "c" at the "(gdb)" prompt.
Back in Octave, try to execute the commands that lead to the crash.
When it happens, return to the gdb prompt and try to get the backtrace with "bt".
What does that display?


[1]: https://superuser.com/questions/1194263/will-microsoft-windows-10-overcommit-memory

Markus Mützel <mmuetzel>
Group administrator
Thu 03 Dec 2020 08:01:16 AM UTC, comment #2: 

comment #1:

> Works for me with Octave 6.1.0 on Windows 10 64bit.
>
> Are you running a 32bit OS? If so, this could be bug #58790.


Yes 32bit (see first line of my original submission). It is definitely an allocation issue. I am not sure whether it is the same with bug #58790. The key question is why Octave 5.1 successfully detects the inability to allocate whereas Octave 6.1 fails to detect it and crashes.

Evangelos Rozos <vrozos>
Wed 02 Dec 2020 07:52:57 PM UTC, comment #1: 

Works for me with Octave 6.1.0 on Windows 10 64bit.

Are you running a 32bit OS? If so, this could be bug #58790.

Markus Mützel <mmuetzel>
Group administrator
Wed 02 Dec 2020 06:31:35 PM UTC, original submission:  

The following command crashes Octave 6.1 in win32


std (movmean (ypred, 365*4))


The size of ypred is 1x40170.

The same command in Octave 5.1 returns

error: out of memory or dimension too large for Octave's index type
error: called from
    movfun>movfun_oncol at line 295 column 10
    movfun at line 278 column 14
    movmean at line 130 column 5

Evangelos Rozos <vrozos>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #50387:  gdblog.txt added by vrozos (9KiB - text/plain - BT from gdb)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by vrozos (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-12-09 mmuetzel StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2020-12-05 mmuetzel StatusConfirmed Ready For Test
    2020-12-04 mmuetzel Carbon-Copy- Added lostbard
    2020-12-03 mmuetzel StatusNeed Info Confirmed
    2020-12-03 vrozos Attached File- Added gdblog.txt, #50387
    2020-12-02 mmuetzel StatusNone Need Info

    Back to the top

    Powered by Savane 3.13-bb6a.
    Corresponding source code