bugGNU Octave - Bugs: bug #62409, [octave forge] (image) BIST errors...

 
 

bug #62409: [octave forge] (image) BIST errors on 32-bit

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Wed 04 May 2022 07:51:45 AM UTC
   
 
Category:  Octave Package Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * other Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 17 May 2022 05:16:07 AM UTC, comment #9: 

Thanks, Hartmut for fixing this.

I have checked this fix, using
test imerode.cc-tst
after fixing this file, and it passes the tests under 32-bit Octave.

I have merged this change into the default branch
(http://hg.code.sf.net/p/octave/image/rev/85eaa18701ad).

During the checking, I found more errors and fixed them
(bug #62437).

Closing as fixed, Thanks :-)

Avinoam Kalma <avinoam>
Group Member
Sun 15 May 2022 12:34:15 PM UTC, comment #8: 

As expected, I wasn't sucessful in recompiling and testing imerode.cc under w32 Octave.

Nevertheless, I consider this simple change of matrix size in two BISTs as low risk, therefore I have now pushed the code changes mentioned in comment #7 to the image repo, stable branch:

https://hg.code.sf.net/p/octave/image/rev/3ceca2e5e92d

This change should probably also be MERGED to the default branch.

I think this issue can now be CLOSED as fixed. We will see after the next next (minor) release if this patch really helped, otherwise we can still reopen this issue.

Hartmut <hardy>
Wed 11 May 2022 02:27:03 PM UTC, comment #7: 

The following changed two BISTs work fine for me on Octave 7.1-w32. Only the huge matrix a was decreased a bit in size and some of the padding indizes were adapted accordingly:


%!test
%! a = rand ([10 22 11 6 8 5]) > 0.2;
%! se = ones ([5 3 7]);
%!
%! ## the image is not really indexed but this way it is padded with 1s
%! assert (imerode (a, se), colfilt (a, "indexed", size (se), "sliding", @all))
%!
%! assert (imerode (a, se, "valid"), convn (a, se, "valid") == nnz (se))
%! ## again, we need to pad it ourselves because convn pads with zeros
%! b = true (size (a) + [4 2 6 0 0 0]);
%! b(3:12, 2:23, 4:14,:,:,:) = a;
%! assert (imdilate (b, se, "same"), convn (b, se, "same") > 0)
%! b = true (size (a) + [8 4 12 0 0 0]);
%! b(5:14, 3:24, 7:17,:,:,:) = a;
%! assert (imdilate (b, se, "full"), convn (b, se, "full") > 0)



%!test
%! a = rand ([10 22 11 6 8 5]) > 0.8;
%! se = ones ([5 3 7]);
%! assert (imdilate (a, se), convn (a, se, "same") > 0)
%! assert (imdilate (a, se, "full"), convn (a, se, "full") > 0)
%! assert (imdilate (a, se, "valid"), convn (a, se, "valid") > 0)
%! assert (imdilate (a, se), colfilt (a, size (se), "sliding", @any))


Running those two changed BIST tests as m-code works fine under w32 Octave.

If someone else could prepare a patch and properly test this I would be happy. These BISTs are part of c-files and unfortunatelly I am not very familiar with recompiling c-files of packages and then running their BISTs under Windows.


Hartmut <hardy>
Sun 08 May 2022 09:05:07 PM UTC, comment #6: 

Thanks for testing, @Hartmut.

It seems that the best solution is to reduce the size of the matrices.

Avinoam Kalma <avinoam>
Group Member
Sun 08 May 2022 05:16:54 PM UTC, comment #5: 

It is most likely a memory fragmentation issue. In practice the
largest size of an array is somewhat under 1GB.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 08 May 2022 03:54:07 PM UTC, comment #4: 

I am able to reproduce this "out of memory" with Octave w32 under Windows (with 16 GB of RAM).

Here is the piece of code that eventually fails when running the first test mentioned in comment #0:


clear
ind = rand(3780000,8);
stride = 592794;
limit = 2370816;
ind_new = bsxfun (@plus, ind(:), (0:stride:limit));


The resulting vector ind_new should have 1.2e8 elements (when running the code witha 64bit indexing instead). I am not quite sure how much memory bsxfun internally occupies during this calculation, but it maybe needs more then the available 2.1e9, which is "only" 20-times more.

@Markus: Does the above memory consumption of bsxfun seem reasonable to you?

@Avinoam: This probably means we need to reduce the matrix size of those BIST tests, or do you have a better idea?

Hartmut <hardy>
Fri 06 May 2022 06:39:45 PM UTC, comment #3: 

I'm not sure, but the index type in Octave might be signed. The maximum number of elements might be 2^31 (approx. 2.1e9).
I'm not sure if it is actually an issue with the maximum index size itself or with memory (fragmentation?) with multiple very large matrices. (It could be either. I should probably have been more careful with my conclusions in comment #0.)

Depending on what the actual underlying issue is, you might be able to reproduce in the version you mentioned (Windows 32-bit).

Markus Mützel <mmuetzel>
Group administrator
Fri 06 May 2022 06:24:25 PM UTC, comment #2: 

If I see this right in the buildbot logs, then the two already mentioned failing tests of "imerode.cc-tst" are the only two tests failing, at least in the current image package 2.14.

Question 1: What is the largest number of elements that an Octave array of type "double" is (roughly) allowed to have when octave_idx_type is 32-bit? When I calculate 2^32 I get 4.3e9, is this the correct maximum number of array elements?

Currently in theses two tests the array a has roughly 1.5e6 elements and the array b has rougly 5.1e6 elements. Are these already too many, or does the index overflow only happen during the calculations (of imerode/imdilate.cc, convn or colfilt)?

Question 2: Would I be able to reproduce this error when using the "octave-7.1.0-w32-installer.exe" under Windows? Or is this 32-bit indexing something different?

Hartmut <hardy>
Thu 05 May 2022 07:05:33 PM UTC, comment #1: 

Adding people to CC

Avinoam Kalma <avinoam>
Group Member
Wed 04 May 2022 07:51:45 AM UTC, original submission:  

Some built-in self tests of the image package are failing when executed in a version of Octave where octave_idx_type is 32-bit.

See e.g.: https://github.com/gnu-octave/octave-buildbot/runs/6277311915?check_suite_focus=true#step:13:4095

  >>>>> processing D:\a\octave-buildbot\octave-buildbot\octave-2022-05-03-00-18-w32\mingw32\lib\octave\packages\image-2.14.0\i686-w64-mingw32-api-v57\imerode.cc-tst
  ***** test
   a = rand ([10 40 15 6 8 5]) > 0.2;
   se = ones ([5 3 7]);

   ## the image is not really indexed but this way it is padded with 1s
   assert (imerode (a, se), colfilt (a, "indexed", size (se), "sliding", @all))

   assert (imerode (a, se, "valid"), convn (a, se, "valid") == nnz (se))
   ## again, we need to pad it ourselves because convn pads with zeros
   b = true (size (a) + [4 2 6 0 0 0]);
   b(3:12, 2:41, 4:18,:,:,:) = a;
   assert (imdilate (b, se, "same"), convn (b, se, "same") > 0)
   b = true (size (a) + [8 4 12 0 0 0]);
   b(5:14, 3:42, 7:21,:,:,:) = a;
   assert (imdilate (b, se, "full"), convn (b, se, "full") > 0)
  !!!!! test failed
  out of memory or dimension too large for Octave's index type
  ***** test
   a = rand ([10 40 15 6 8 5]) > 0.8;
   se = ones ([5 3 7]);
   assert (imdilate (a, se), convn (a, se, "same") > 0)
   assert (imdilate (a, se, "full"), convn (a, se, "full") > 0)
   assert (imdilate (a, se, "valid"), convn (a, se, "valid") > 0)
   assert (imdilate (a, se), colfilt (a, size (se), "sliding", @any))
  !!!!! test failed
  out of memory or dimension too large for Octave's index type


It might be nice to limit the size of the used matrices so that they could be used with 32-bit indices. Or if that is not possible, skip these tests if Octave's indices are 32-bit.

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by hardy (Posted a comment)
  • -email is unavailable- added by avinoam (Posted a comment)
  • -email is unavailable- added by avinoam
  • -email is unavailable- added by avinoam
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-05-17 avinoam StatusNone Fixed
        Open/ClosedOpen Closed
    2022-05-05 avinoam Carbon-Copy- Added carandraug
        Carbon-Copy- Added hardy

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code