Add a New Comment (Rich Markup)
Comment Type & Canned Response: None None > Multiple Canned Responses Fixed in development Crash with no stack trace Already fixed in newer version Fixed in stable Bad description Bad description and crash Bad stack trace Obsolete version Duplicate and not fixed Duplicate and needs more info Duplicate and fixed Need info and old
( Jump to the original submission )
Thanks, Hartmut for fixing this.
I have checked this fix, using test imerode.cc-tst after fixing this file, and it passes the tests under 32-bit Octave.
I have merged this change into the default branch (http://hg.code.sf.net/p/octave/image/rev/85eaa18701ad).
During the checking, I found more errors and fixed them (bug #62437).
Closing as fixed, Thanks :-)
As expected, I wasn't sucessful in recompiling and testing imerode.cc under w32 Octave.
Nevertheless, I consider this simple change of matrix size in two BISTs as low risk, therefore I have now pushed the code changes mentioned in comment #7 to the image repo, stable branch:
https://hg.code.sf.net/p/octave/image/rev/3ceca2e5e92d
This change should probably also be MERGED to the default branch.
I think this issue can now be CLOSED as fixed. We will see after the next next (minor) release if this patch really helped, otherwise we can still reopen this issue.
The following changed two BISTs work fine for me on Octave 7.1-w32. Only the huge matrix a was decreased a bit in size and some of the padding indizes were adapted accordingly:
%!test %! a = rand ([10 22 11 6 8 5]) > 0.2; %! se = ones ([5 3 7]); %! %! ## the image is not really indexed but this way it is padded with 1s %! assert (imerode (a, se), colfilt (a, "indexed", size (se), "sliding", @all)) %! %! assert (imerode (a, se, "valid"), convn (a, se, "valid") == nnz (se)) %! ## again, we need to pad it ourselves because convn pads with zeros %! b = true (size (a) + [4 2 6 0 0 0]); %! b(3:12, 2:23, 4:14,:,:,:) = a; %! assert (imdilate (b, se, "same"), convn (b, se, "same") > 0) %! b = true (size (a) + [8 4 12 0 0 0]); %! b(5:14, 3:24, 7:17,:,:,:) = a; %! assert (imdilate (b, se, "full"), convn (b, se, "full") > 0)
%!test %! a = rand ([10 22 11 6 8 5]) > 0.8; %! se = ones ([5 3 7]); %! assert (imdilate (a, se), convn (a, se, "same") > 0) %! assert (imdilate (a, se, "full"), convn (a, se, "full") > 0) %! assert (imdilate (a, se, "valid"), convn (a, se, "valid") > 0) %! assert (imdilate (a, se), colfilt (a, size (se), "sliding", @any))
Running those two changed BIST tests as m-code works fine under w32 Octave.
If someone else could prepare a patch and properly test this I would be happy. These BISTs are part of c-files and unfortunatelly I am not very familiar with recompiling c-files of packages and then running their BISTs under Windows.
Thanks for testing, @Hartmut.
It seems that the best solution is to reduce the size of the matrices.
It is most likely a memory fragmentation issue. In practice the largest size of an array is somewhat under 1GB.
Dmitri. --
I am able to reproduce this "out of memory" with Octave w32 under Windows (with 16 GB of RAM).
Here is the piece of code that eventually fails when running the first test mentioned in comment #0:
clear ind = rand(3780000,8); stride = 592794; limit = 2370816; ind_new = bsxfun (@plus, ind(:), (0:stride:limit));
The resulting vector ind_new should have 1.2e8 elements (when running the code witha 64bit indexing instead). I am not quite sure how much memory bsxfun internally occupies during this calculation, but it maybe needs more then the available 2.1e9, which is "only" 20-times more.
@Markus: Does the above memory consumption of bsxfun seem reasonable to you?
@Avinoam: This probably means we need to reduce the matrix size of those BIST tests, or do you have a better idea?
I'm not sure, but the index type in Octave might be signed. The maximum number of elements might be 2^31 (approx. 2.1e9). I'm not sure if it is actually an issue with the maximum index size itself or with memory (fragmentation?) with multiple very large matrices. (It could be either. I should probably have been more careful with my conclusions in comment #0.)
Depending on what the actual underlying issue is, you might be able to reproduce in the version you mentioned (Windows 32-bit).
If I see this right in the buildbot logs, then the two already mentioned failing tests of "imerode.cc-tst" are the only two tests failing, at least in the current image package 2.14.
Question 1: What is the largest number of elements that an Octave array of type "double" is (roughly) allowed to have when octave_idx_type is 32-bit? When I calculate 2^32 I get 4.3e9, is this the correct maximum number of array elements?
Currently in theses two tests the array a has roughly 1.5e6 elements and the array b has rougly 5.1e6 elements. Are these already too many, or does the index overflow only happen during the calculations (of imerode/imdilate.cc, convn or colfilt)?
Question 2: Would I be able to reproduce this error when using the "octave-7.1.0-w32-installer.exe" under Windows? Or is this 32-bit indexing something different?
Adding people to CC
Some built-in self tests of the image package are failing when executed in a version of Octave where octave_idx_type is 32-bit.
See e.g.: https://github.com/gnu-octave/octave-buildbot/runs/6277311915?check_suite_focus=true#step:13:4095
>>>>> processing D:\a\octave-buildbot\octave-buildbot\octave-2022-05-03-00-18-w32\mingw32\lib\octave\packages\image-2.14.0\i686-w64-mingw32-api-v57\imerode.cc-tst ***** test a = rand ([10 40 15 6 8 5]) > 0.2; se = ones ([5 3 7]); ## the image is not really indexed but this way it is padded with 1s assert (imerode (a, se), colfilt (a, "indexed", size (se), "sliding", @all)) assert (imerode (a, se, "valid"), convn (a, se, "valid") == nnz (se)) ## again, we need to pad it ourselves because convn pads with zeros b = true (size (a) + [4 2 6 0 0 0]); b(3:12, 2:41, 4:18,:,:,:) = a; assert (imdilate (b, se, "same"), convn (b, se, "same") > 0) b = true (size (a) + [8 4 12 0 0 0]); b(5:14, 3:42, 7:21,:,:,:) = a; assert (imdilate (b, se, "full"), convn (b, se, "full") > 0) !!!!! test failed out of memory or dimension too large for Octave's index type ***** test a = rand ([10 40 15 6 8 5]) > 0.8; se = ones ([5 3 7]); assert (imdilate (a, se), convn (a, se, "same") > 0) assert (imdilate (a, se, "full"), convn (a, se, "full") > 0) assert (imdilate (a, se, "valid"), convn (a, se, "valid") > 0) assert (imdilate (a, se), colfilt (a, size (se), "sliding", @any)) !!!!! test failed out of memory or dimension too large for Octave's index type
It might be nice to limit the size of the used matrices so that they could be used with 32-bit indices. Or if that is not possible, skip these tests if Octave's indices are 32-bit.
(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)
Attach Files: Comment:
No files currently attached
Depends on the following items: None found
Items that depend on this one: None found
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Only project members can vote.
Please enter the title of George Orwell's famous dystopian book (it's a date):
Follow 4 latest changes.
Copyright © 2022 Free Software Foundation, Inc. Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved. The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team. Source Code
Powered by Savane 3.9