I found only one way that can somewhat prevent us from being hit by memory overcommit. That is to manually check the available memory (additionally to checking whether "new" or "malloc" return a nullptr).
That would entail checking some of the fields that the new "memory" function uses to gather its output.
Not sure whether this is costly though.

That still won't make sure that Octave won't ever be hit by the OOM killer. But maybe we could prevent the OOM killer from triggering due to our own actions.

I have read that modern operating systems often over-allocate. Malloc and new can return a valid pointer even if there is not enough free memory. I.e. std::bad_alloc is never raised, or at least is not a reliable indicator of memory exhaustion. I haven't found a literature reference so far, but that could be a problem.

For example if try hard enough i can even kill Octave on linux 64 bit:

...
Allocate 44000 MByte
Allocate 45000 MByte
Allocate 46000 MByte
Killed

For this you have to run following example and choose N big enough.

used = 0;
N = 13107200;
chars = char([65:90, 97:122]);
for idx = 1 : length (chars)
  chunk = N * 8 * 10 / (1024 * 1024);
  fprintf ('Allocate %i MByte\n', used + chunk);
  x.(chars(idx)) = rand (N, 10);
  used = used + chunk;
end
fprintf ('success\n');
whos ('x');

I'm not sure this is an integer overflow when computing the total array size.
The number of elements in the array that triggers this bug is "just" 100 million (see comment #14). The largest signed 32 bit integer is approximately 2 billion.
Also, the approximately 800 MiB array that causes this bug is still about a factor 5 smaller than 4 GiB where uint32_t overflows or more than a factor 2 smaller than 2 GiB where int32_t overflows.
Additionally, this only seems to crash Octave if the memory layout is "sufficiently fragmented". An integer overflow should not depend on how much of the memory is available if I understand correctly.

But I don't know how the total array size is calculated. So there might be an intermediate result that overflows?

Yes, it is different to Octave 5.2.0:

GNU Octave Version: 7.0.0 (hg id: b9bf7a6bb8de)

Test 1.) An error is raised:

>> x=rand(30000000,10);
error: out of memory or dimension too large for Octave's index type

(IMHO the message is raised in pt-eval.cc)

Test 2.) Octave closes immediately instead of rejecting:

>> x=rand(20000000,10);
--> Octave closes immediately

Test 3.) Octave closes immediately instead of rejecting:

>> x=rand(10000000,10);
>> memory
Octave    RAM:    918916 kiB,  virt:   1829560 kiB
>> y=rand(10000000,10);
--> Octave closes immediately

The same tests in Octave 5.2.0 either result in correct rejection if there is insufficient memory or an acceptance.

BTW, the win64 target also ends without an error message if too much memory is allocated. However if one is careful and does the test on octave-cli w64, one can read "LLVM ERROR: out of memory; fatal caught signal Aborted -- stoppfing myself" immediately before the window closes.

I don't have a hg id near the RC around so i can't tell the situation there.

Link to an discussion about feasible memory consumption of the 32bit version.

bug #44380

Consistent to what was observed below.

Here, the critical section is in gmres where a chunk of memory is allocated:

resvec = zeros (max_iter_number + 1, 1);

max_iter_number is 100000000. If i run your example with "rgb = 1000x1000x3" up to and including this position, then "memory" shows:

System    RAM:   2095160 kiB,  swap:   4190320 kiB
Octave    RAM:    979228 kiB,  virt:   1951672 kiB
Available RAM:    684132 kiB, total:    870556 kiB

That is quite a chunk. How much RAM do you have?

It might be an issue that only shows on Windows 10. Or it might be specific to my system.

If I am the only one who sees this issue and only with the 32bit version of Octave (which I wouldn't use anyway), maybe we could just wait for feedback on an upcoming release candidate of Octave 6.

The tests with hg id a65cacd05892 (win64 and win32) did not reveal any new insights except bug #58800. Here the BISTs run "without closing Octave" and the reported fails are the same as mentioned below.

I played a bit and connected / disconnected an external USB sound adapter several times and suddenly audiodevinfo() returned a valid input device. Now the testsuite passes the audiotests ("skipped" is now 46 instead 50) and Octave still does not close. If i get this right the other mentioned tests complain about a numerical deviation less or equal to one machine epsilon. ok.

If i search for windows + error code 3 it says something about "directory not found". Therefore my question how Octave was installed. Maybe there are differences if the installer is run from the executable (access rights, dir-names ...). It is pure speculation but maybe you can check if that changes something. I did:

make JOBS=12 hg-octave-dist hg-octave-branch=default
make JOBS=12 nsis-installer 7z-dist

and then installed via the installer executable. And instead of "p7zip -k" i have to use "7z a" in binary-dist-rules.mk, because on Fedora there is no p7zip command.

Anyway, i will start over with a new built.

The 4 BISTs that pass for me but are skipped for you are the ones for audiorecorder.
But those aren't the ones that trigger closing the GUI here. The following works fine:

test @audiorecorder/audiorecorder.m
runtests (fullfile (OCTAVE_HOME, 'share\octave\7.0.0\m\sparse'))

This test failed for me but didn't for you:

>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\polynomial\polyfit.m
***** assert (polyfit (x, x.^2 + 2*x + 3, 3), [0, 1, 2, 3], 10*eps)
!!!!! test failed
ASSERT errors for:  assert (polyfit (x, x .^ 2 + 2 * x + 3, 3),[0, 1, 2, 3],10 * eps)

  Location  |  Observed  |  Expected  |  Reason
    (3)           2            2         Abs err 2.6645e-15 exceeds tol 2.2204e-15 by 4e-16
shared variables   scalar structure containing the fields:

    x =

      -2  -1   0   1   2

And this test failed for you but didn't for me:

>>>>> processing C:\PROGRA~2\GNUOCT~1\OCTAVE~1.0\mingw32\share\octave\7.0.0\m\specfun\cosint.m
***** assert (cosint (pi), 0.0736679120464254860, -2e-15)
!!!!! test failed
ASSERT errors for:  assert (cosint (pi),0.0736679120464254860,-2e-15)

  Location  |  Observed  |  Expected  |  Reason
     ()        0.073668     0.073668     Rel err 2.2606e-15 exceeds tol 2e-15 by 3e-16

Unfortunately, I don't think that brings us any closer to understanding what is happening.

The next step is probably for me to "bisect" by running different parts of the test suite and hope that I can find a minimal example that triggers the bug here.

1.) Find attached "hg log" that shows what i have currently patched onto hg id a65cacd05892 in the win32-target. The "tip" patches a mxe-octave problem that manifests when building on Fedora. But there is no relation ship to this problem. "ver" shows up as 7.0.0 (hg id: 6310bb807752). "hg branch" shows "default" and i used similar configure switches like you (most likely the same).

2.) For win64 target I can't say for sure because i already erased the build directory (done is done).

3.) ode also fails but sparse tests are fine. See attached log.

What i will do is to start an overnight build of hg id a65cacd05892 for win64 + win32 targets without any extra patches and with your config switches, and then i will try to reproduce the steps. Then we can definitely sure.

I have also some questions:

1.) Do you run octave in a wine environment or in a real windows (mine is Win 7)?

2.) Do run octave-gui or octave-cli (i do octave-gui)?

3.) Did you build and install via installer (mine is via installer)?



(file #49523, file #49524)

MXE Octave (hg id a65cacd05892) was configured to cross-compile Windows 32bit executables with:

./configure --with-pkg-dir=../mxe-octave-pkg --with-ccache --enable-octave=default --enable-binary-packages --enable-qt5 --enable-devel-tools --disable-windows-64 --disable-64 --disable-fortran-int64 --disable-system-opengl

When running the test suite with "__run_test_suite__" on Windows 10 2004 (64bit), Octave (hg id 6310bb807752) closes unexpectedly.
That behavior seems to be reproducible 100%.
"gdb" doesn't catch an error because Octave "just closes" (with error code 3).

I am attaching the fntests.log.

The last lines appearing in the log are:

>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\sparse\bicgstab.m
>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\sparse\cgs.m
>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\sparse\eigs.m
>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\sparse\gmres.m

Some of the tests that completed up until this point fail because of minor tolerance differences.

Other tests related to sparse matrices fail with these errors (not sure if this is related):

>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\ode\ode15i.m
***** testif HAVE_SUNDIALS_SUNLINSOL_KLU
 opt = odeset ("Jacobian", @jacfunsparse, "AbsTol", 1e-7, "RelTol", 1e-7);
 [t, y] = ode15i (@rob, [0, 100], [1; 0; 0], [-1e-4; 1e-4; 0], opt);
 assert ([t(end), y(end,:)], fref, 1e-3);
!!!!! test failed
IDASolve failed
>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\ode\ode15s.m
***** testif HAVE_SUNDIALS_SUNLINSOL_KLU
 opt = odeset ("MStateDependence", "none",
               "Mass", sparse ([1, 0, 0; 0, 1, 0; 0, 0, 0]),
               "Jacobian", @jacfunsparse);
 [t, y] = ode15s (@rob, [0, 100], [1; 0; 0], opt);
 assert ([t(end), y(end,:)], frefrob, 1e-3);
!!!!! test failed
IDASolve failed
***** testif HAVE_SUNDIALS_SUNLINSOL_KLU
 opt = odeset ("MStateDependence", "none",
               "Mass", @masssparsefuntime,
               "Jacobian", @jacfunsparse);
 [t, y] = ode15s (@rob, [0, 100], [1; 0; 0], opt);
 assert ([t(end), y(end,:)], frefrob, 1e-3);
!!!!! test failed
IDASolve failed

So maybe there are issues with one or some of the sparse related libraries (SUNDIALS, SuiteSparse, ...) for 32bit (Windows) builds.

Octave does not close if I start Octave and immediately run the tests in the "sparse" directory:

runtests (fullfile (OCTAVE_HOME, 'share\octave\7.0.0\m\sparse'))

This seems to be one of those bugs that only occur if some commands are executed in a specific order. I don't have a minimal example yet.