bugGNU Octave - Bugs: bug #40324, Concatenating Sparse matrices...

 
 

bug #40324: Concatenating Sparse matrices throws OOM

Submitted by:  David Spies <dspyz>
Submitted on:  Mon 21 Oct 2013 04:46:21 AM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Performance
Status: FixedAssigned to: David Bateman <dbateman>
Originator Name: Open/Closed: Closed
Release: devOperating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 21 Oct 2013 08:40:27 PM UTC, comment #4:

How exactly do you "patch" numel ? This method is doing what it supposed to do for sparse matrices, but given the limitation of 32 bit integers it will never return the right value for large sparse matrices. The only "patch" I can imagine that has a chance of fixing this is to always use 64 bit integers for numel for all types, and even that won't fix

when Octave is built with 64 bit integers. Though you might argue that it'll be a while till we'll see matrices this large. In this case, the special case is justified as sparse matrices don't even use the 1x1 optimisation in pt-mat.cc

So for your question, yes I think its worth avoiding numel wherever possible when dealing with sparse matrices

David

David Bateman <dbateman>
Project MemberIn charge of this item.
Mon 21 Oct 2013 08:20:01 PM UTC, comment #3:

Do you still think it's better to keep special-casing all the calls to numel instead of patching numel itself?

Jordi GutiƩrrez Hermoso <jordigh>
Project Administrator
Mon 21 Oct 2013 07:59:19 PM UTC, comment #2:

That should of course have read horzcat.

Looking at this problem a bit further, the difference between

and

is that the first goes through the parse tree code pt-mat.cc to construct the matrix list to concatenate, and in this code there are multiple uses of numel and in particular one to test if all the values are scalar. As sparse matrices can't profit from the fact that all elements are scalar in the concatenation this test should be avoided as a call to numel calls the idx_vector::safe_numel method that throws a bad alloc in this case even though it doesn't need to.

Fixed in the changeset

http://hg.savannah.gnu.org/hgweb/octave/rev/ccc0576641f9

D.

David Bateman <dbateman>
Project MemberIn charge of this item.
Mon 21 Oct 2013 06:38:36 PM UTC, comment #1:

For a workaround horizcat and vertcat seem to work fine. I'm looking at this issue

D.

David Bateman <dbateman>
Project MemberIn charge of this item.
Mon 21 Oct 2013 04:46:21 AM UTC, original submission:

>> a = speye(100000);
>> b = [a;a];

error: out of memory or dimension too large for Octave's index type

David Spies <dspyz>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by dbateman (Posted a comment)
  • -unavailable- added by jordigh (Updated the item)
  • -unavailable- added by dspyz (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 4 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 21 Oct 2013 08:00:00 PM UTCdbatemanStatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Mon 21 Oct 2013 06:38:36 PM UTCdbatemanAssigned toNone=>dbateman
    Mon 21 Oct 2013 01:38:45 PM UTCjordighStatusNone=>Confirmed

    Back to the top


    Powered by Savane 3.1-cleanup1