bugGNU Octave - Bugs: bug #58790, Octave closes unexpectedly when...

 
 

bug #58790: Octave closes unexpectedly when memory allocation fails (Windows 32bit executable)

Submitted by:  Markus Mützel <mmuetzel>
Submitted on:  Mon 20 Jul 2020 12:49:06 PM UTC  
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 02 Aug 2020 10:43:31 AM UTC, comment #18: 

Yes, it is different to Octave 5.2.0:

GNU Octave Version: 7.0.0 (hg id: b9bf7a6bb8de)

Test 1.) An error is raised:

>> x=rand(30000000,10);
error: out of memory or dimension too large for Octave's index type

(IMHO the message is raised in pt-eval.cc)

Test 2.) Octave closes immediately instead of rejecting:

>> x=rand(20000000,10);
--> Octave closes immediately

Test 3.) Octave closes immediately instead of rejecting:

>> x=rand(10000000,10);
>> memory
Octave    RAM:    918916 kiB,  virt:   1829560 kiB
>> y=rand(10000000,10);
--> Octave closes immediately

The same tests in Octave 5.2.0 either result in correct rejection if there is insufficient memory or an acceptance.

BTW, the win64 target also ends without an error message if too much memory is allocated. However if one is careful and does the test on octave-cli w64, one can read "LLVM ERROR: out of memory; fatal caught signal Aborted -- stoppfing myself" immediately before the window closes.

I don't have a hg id near the RC around so i can't tell the situation there.

Hg200 <hg200>
Sun 02 Aug 2020 05:27:50 AM UTC, comment #17: 

If I correctly read bug #44380, Octave did warn if the largest contiguous free memory block wasn't large enough to hold the requested object. It now closes unexpectedly possibly loosing data.

Might be a regression.

Markus Mützel <mmuetzel>
Project Member
Sat 01 Aug 2020 09:49:27 PM UTC, comment #16: 

Link to an discussion about feasible memory consumption of the 32bit version.

bug #44380

Consistent to what was observed below.

Hg200 <hg200>
Wed 22 Jul 2020 06:59:16 AM UTC, comment #15: 

That assignment tries to allocate about 800 MiB in memory.
IIUC, for 32bit programs the allocation size is limited by the maximum contiguous free memory "block". Currently Octave's "memory" function doesn't probe the size of that chunk but only return the total free memory (in all "blocks").

To reproduce this bug, it might be that it is necessary to "sufficiently" fragment the memory layout and then try to allocate a memory block that is larger than the largest contiguous memory block.

Matlab fails (or since it is no longer available as a 32bit executable "it failed") with an error in this case.
It looks like Octave doesn't handle that error gracefully.
It's interesting that gdb isn't able to catch that kind of error.

AFAICT, that kind of error isn't limited to 32bit Windows but could happen on any 32bit OS.
It is likely that a similar error could happen (for much larger arrays) on 64bit OSs, too.
I don't have access to a 32bit Linux to confirm that. But I'll change the title and flags of this bug anyway.

Markus Mützel <mmuetzel>
Project Member
Tue 21 Jul 2020 08:26:09 PM UTC, comment #14: 

Here, the critical section is in gmres where a chunk of memory is allocated:

resvec = zeros (max_iter_number + 1, 1);

max_iter_number is 100000000. If i run your example with "rgb = 1000x1000x3" up to and including this position, then "memory" shows:

System    RAM:   2095160 kiB,  swap:   4190320 kiB
Octave    RAM:    979228 kiB,  virt:   1951672 kiB
Available RAM:    684132 kiB, total:    870556 kiB

That is quite a chunk. How much RAM do you have?

Hg200 <hg200>
Tue 21 Jul 2020 07:34:23 PM UTC, comment #13: 

okay, now i can also reproduce!!! but i have to increase the rgb matrix from 1000x1000x3 to 2000x2000x3 (i run your second example). That closes Octave here too:

rgb = rand (2000, 2000, 3);

Hg200 <hg200>
Tue 21 Jul 2020 05:02:02 PM UTC, comment #12: 

It might be an issue that only shows on Windows 10. Or it might be specific to my system.

If I am the only one who sees this issue and only with the 32bit version of Octave (which I wouldn't use anyway), maybe we could just wait for feedback on an upcoming release candidate of Octave 6.

Markus Mützel <mmuetzel>
Project Member
Tue 21 Jul 2020 04:53:17 PM UTC, comment #11: 

I was able to condense the test case down to the following:

# from rgb2ind's BISTs
rgb = rand (1000, 1000, 3);
[ind, map] = rgb2ind (rgb);

# from gmres's BISTs
dim = 100;
A = spdiags ([[1./(2:2:2*(dim-1)) 0]; 1./(1:2:2*dim-1); ...
[0 1./(2:2:2*(dim-1))]]', -1:1, dim, dim);
A = A'*A;
b = (1:dim).'/dim;
[x, flag] = gmres (@(x) A*x, b, dim, 1e-10, 1e6,...
                   @(x) diag (diag (A)) \ x, [], []);

Sometimes, I have to execute it twice. But most of the time, Octave closes on first execution.

If I remove the call to rgb2ind, I can run the code more than 100 times without the GUI closing.

The following is basically what rgb2ind is doing and it also triggers the bug:

# from rgb2ind
rgb = rand (1000, 1000, 3);
R = rgb(:,:,1,:);
G = rgb(:,:,2,:);
B = rgb(:,:,3,:);
x = reshape (1:numel (R), size (R));

map    = unique ([R(:) G(:) B(:)], "rows");
[~, x] = ismember ([R(:) G(:) B(:)], map, "rows");

# from gmres's BISTs
dim = 100;
A = spdiags ([[1./(2:2:2*(dim-1)) 0]; 1./(1:2:2*dim-1); ...
[0 1./(2:2:2*(dim-1))]]', -1:1, dim, dim);
A = A'*A;
b = (1:dim).'/dim;
[x, flag] = gmres (@(x) A*x, b, dim, 1e-10, 1e6,...
                   @(x) diag (diag (A)) \ x, [], []);

All variables involved are dense double matrices. I'm not sure where to look next.

Markus Mützel <mmuetzel>
Project Member
Tue 21 Jul 2020 03:59:21 PM UTC, comment #10: 

The tests with hg id a65cacd05892 (win64 and win32) did not reveal any new insights except bug #58800. Here the BISTs run "without closing Octave" and the reported fails are the same as mentioned below.

Hg200 <hg200>
Mon 20 Jul 2020 09:59:15 PM UTC, comment #9: 

I deinstalled octave and run instead the postinstall + octave.vbs. Still NO problems.

Considering other issues like bug #57591 (if related or not), it is probably not so clever to start with "installing" and "deinstalling" tests, but rather with bisecting. If you can reproduce the bug - don't remove it.

Hg200 <hg200>
Mon 20 Jul 2020 07:42:17 PM UTC, comment #8: 

I played a bit and connected / disconnected an external USB sound adapter several times and suddenly audiodevinfo() returned a valid input device. Now the testsuite passes the audiotests ("skipped" is now 46 instead 50) and Octave still does not close. If i get this right the other mentioned tests complain about a numerical deviation less or equal to one machine epsilon. ok.

If i search for windows + error code 3 it says something about "directory not found". Therefore my question how Octave was installed. Maybe there are differences if the installer is run from the executable (access rights, dir-names ...). It is pure speculation but maybe you can check if that changes something. I did:

make JOBS=12 hg-octave-dist hg-octave-branch=default
make JOBS=12 nsis-installer 7z-dist

and then installed via the installer executable. And instead of "p7zip -k" i have to use "7z a" in binary-dist-rules.mk, because on Fedora there is no p7zip command.

Anyway, i will start over with a new built.

Hg200 <hg200>
Mon 20 Jul 2020 06:35:58 PM UTC, comment #7: 

I compared to one of my log files from a GUI test run. Those all stop at "sparse/gmres.m". So it didn't even include the BISTs of the files in specfun.
In the CLI test run, cosint failed with the same error for me.

Markus Mützel <mmuetzel>
Project Member
Mon 20 Jul 2020 06:26:36 PM UTC, comment #6: 

The 4 BISTs that pass for me but are skipped for you are the ones for audiorecorder.
But those aren't the ones that trigger closing the GUI here. The following works fine:

test @audiorecorder/audiorecorder.m
runtests (fullfile (OCTAVE_HOME, 'share\octave\7.0.0\m\sparse'))

This test failed for me but didn't for you:

>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\polynomial\polyfit.m
***** assert (polyfit (x, x.^2 + 2*x + 3, 3), [0, 1, 2, 3], 10*eps)
!!!!! test failed
ASSERT errors for:  assert (polyfit (x, x .^ 2 + 2 * x + 3, 3),[0, 1, 2, 3],10 * eps)

  Location  |  Observed  |  Expected  |  Reason
    (3)           2            2         Abs err 2.6645e-15 exceeds tol 2.2204e-15 by 4e-16
shared variables   scalar structure containing the fields:

    x =

      -2  -1   0   1   2

And this test failed for you but didn't for me:

>>>>> processing C:\PROGRA~2\GNUOCT~1\OCTAVE~1.0\mingw32\share\octave\7.0.0\m\specfun\cosint.m
***** assert (cosint (pi), 0.0736679120464254860, -2e-15)
!!!!! test failed
ASSERT errors for:  assert (cosint (pi),0.0736679120464254860,-2e-15)

  Location  |  Observed  |  Expected  |  Reason
     ()        0.073668     0.073668     Rel err 2.2606e-15 exceeds tol 2e-15 by 3e-16

Unfortunately, I don't think that brings us any closer to understanding what is happening.

The next step is probably for me to "bisect" by running different parts of the test suite and hope that I can find a minimal example that triggers the bug here.

Markus Mützel <mmuetzel>
Project Member
Mon 20 Jul 2020 05:46:45 PM UTC, comment #5: 

next try

(file #49525)

Hg200 <hg200>
Mon 20 Jul 2020 05:45:18 PM UTC, comment #4: 
Hg200 <hg200>
Mon 20 Jul 2020 04:37:16 PM UTC, comment #3: 

I was testing on a "physical" Windows 10 2004 (no virtualization or anything).

I was testing with the GUI, too. I unpacked the .7z archive, ran "post-install.bat" and started Octave with "octave.vbs".

I ran the test suite again with the CLI ("octave.vbs --no-gui"). The same build that still closes unexpectedly with the GUI, completes the test suite with the CLI. The overall results are similar to yours:

Summary:

  PASS                            16205
  FAIL                               10
  XFAIL (reported bug)               30
  SKIP (missing feature)             52
  SKIP (run-time condition)          46

Comparing to your summary, it shows that 46 tests are skipped on run-time condition for me, but 50 for you.
It might be that one of the four (or five?) tests that run for me and are skipped for you are causing the issue.

Could you please upload your complete fntests.log (created in the current directory by _run_test_suite_) so that we can compare which BISTs these are? (And the last one that fails for me and seems to be skipped for you.)

Markus Mützel <mmuetzel>
Project Member
Mon 20 Jul 2020 03:47:00 PM UTC, comment #2: 

1.) Find attached "hg log" that shows what i have currently patched onto hg id a65cacd05892 in the win32-target. The "tip" patches a mxe-octave problem that manifests when building on Fedora. But there is no relation ship to this problem. "ver" shows up as 7.0.0 (hg id: 6310bb807752). "hg branch" shows "default" and i used similar configure switches like you (most likely the same).

2.) For win64 target I can't say for sure because i already erased the build directory (done is done).

3.) ode also fails but sparse tests are fine. See attached log.

What i will do is to start an overnight build of hg id a65cacd05892 for win64 + win32 targets without any extra patches and with your config switches, and then i will try to reproduce the steps. Then we can definitely sure.

I have also some questions:

1.) Do you run octave in a wine environment or in a real windows (mine is Win 7)?

2.) Do run octave-gui or octave-cli (i do octave-gui)?

3.) Did you build and install via installer (mine is via installer)?

(file #49523, file #49524)

Hg200 <hg200>
Mon 20 Jul 2020 02:30:28 PM UTC, comment #1: 

In response to hg200's comment in bug #58689 https://savannah.gnu.org/bugs/index.php?58689#comment36 where he couldn't reproduce the issue for a 32bit build:

Did you use different MXE build trees for the 32bit and 64bit builds?

Did you build stable or default Octave?

Do you see errors for "test ode15i" or "test ode15s"?

Markus Mützel <mmuetzel>
Project Member
Mon 20 Jul 2020 12:49:06 PM UTC, original submission:  

MXE Octave (hg id a65cacd05892) was configured to cross-compile Windows 32bit executables with:

./configure --with-pkg-dir=../mxe-octave-pkg --with-ccache --enable-octave=default --enable-binary-packages --enable-qt5 --enable-devel-tools --disable-windows-64 --disable-64 --disable-fortran-int64 --disable-system-opengl

When running the test suite with "__run_test_suite__" on Windows 10 2004 (64bit), Octave (hg id 6310bb807752) closes unexpectedly.
That behavior seems to be reproducible 100%.
"gdb" doesn't catch an error because Octave "just closes" (with error code 3).

I am attaching the fntests.log.

The last lines appearing in the log are:

>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\sparse\bicgstab.m
>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\sparse\cgs.m
>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\sparse\eigs.m
>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\sparse\gmres.m

Some of the tests that completed up until this point fail because of minor tolerance differences.

Other tests related to sparse matrices fail with these errors (not sure if this is related):

>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\ode\ode15i.m
***** testif HAVE_SUNDIALS_SUNLINSOL_KLU
opt = odeset ("Jacobian", @jacfunsparse, "AbsTol", 1e-7, "RelTol", 1e-7);
[t, y] = ode15i (@rob, [0, 100], [1; 0; 0], [-1e-4; 1e-4; 0], opt);
assert ([t(end), y(end,:)], fref, 1e-3);
!!!!! test failed
IDASolve failed
>>>>> processing D:\SVN\Octave\test\OC20A7~1\OCTAVE~1\mingw32\share\octave\7.0.0\m\ode\ode15s.m
***** testif HAVE_SUNDIALS_SUNLINSOL_KLU
opt = odeset ("MStateDependence", "none",
               "Mass", sparse ([1, 0, 0; 0, 1, 0; 0, 0, 0]),
               "Jacobian", @jacfunsparse);
[t, y] = ode15s (@rob, [0, 100], [1; 0; 0], opt);
assert ([t(end), y(end,:)], frefrob, 1e-3);
!!!!! test failed
IDASolve failed
***** testif HAVE_SUNDIALS_SUNLINSOL_KLU
opt = odeset ("MStateDependence", "none",
               "Mass", @masssparsefuntime,
               "Jacobian", @jacfunsparse);
[t, y] = ode15s (@rob, [0, 100], [1; 0; 0], opt);
assert ([t(end), y(end,:)], frefrob, 1e-3);
!!!!! test failed
IDASolve failed

So maybe there are issues with one or some of the sparse related libraries (SUNDIALS, SuiteSparse, ...) for 32bit (Windows) builds.

Octave does not close if I start Octave and immediately run the tests in the "sparse" directory:

runtests (fullfile (OCTAVE_HOME, 'share\octave\7.0.0\m\sparse'))

This seems to be one of those bugs that only occur if some commands are executed in a specific order. I don't have a minimal example yet.

Markus Mützel <mmuetzel>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49525:  win32-target-testsuite.tar.gz added by hg200 (1MiB - application/gzip)
file #49523:  hg200_bists_run_v1.txt added by hg200 (5KiB - text/plain)
file #49524:  hg200_hglog_v1.txt added by hg200 (2KiB - text/plain)
file #49516:  fntests.7z added by mmuetzel (940KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hg200 (Updated the item)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 12 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-08-02 mmuetzel Operating SystemAny => Microsoft Windows
        SummaryOctave closes unexpectedly when memory allocation fails => Octave closes unexpectedly when memory allocation fails (Windows 32bit executable)
    2020-07-22 mmuetzel CategoryLibraries => Interpreter
        StatusNone => Confirmed
        Operating SystemMicrosoft Windows => Any
        Summary[MXE Octave] Windows 32bit build closes unexpectedly while running the test suite => Octave closes unexpectedly when memory allocation fails
    2020-07-20 hg200 Attached File- => Added win32-target-testsuite.tar.gz, #49525
    2020-07-20 hg200 Attached File- => Added hg200_bists_run_v1.txt, #49523
        Attached File- => Added hg200_hglog_v1.txt, #49524
    2020-07-20 mmuetzel Release6.0.90 => dev
        Carbon-Copy- => Added hg200
    2020-07-20 mmuetzel Attached File- => Added fntests.7z, #49516

    Back to the top


    Powered by Savane 3.5