bugGNU Octave - Bugs: bug #40324, Concatenating Sparse matrices...

 
 

bug #40324: Concatenating Sparse matrices throws OOM

Submitter:  David Spies <dspyz>
Submitted:  Mon 21 Oct 2013 04:46:21 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Fixed Assigned to:  dbateman
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 21 Oct 2013 08:40:27 PM UTC, comment #4: 

How exactly do you "patch" numel ? This method is doing what it supposed to do for sparse matrices, but given the limitation of 32 bit integers it will never return the right value for large sparse matrices. The only "patch" I can imagine that has a chance of fixing this is to always use 64 bit integers for numel for all types, and even that won't fix


s = speye (2^32);
q = [s, s];


when Octave is built with 64 bit integers. Though you might argue that it'll be a while till we'll see matrices this large. In this case, the special case is justified as sparse matrices don't even use the 1x1 optimisation in pt-mat.cc

So for your question, yes I think its worth avoiding numel wherever possible when dealing with sparse matrices

David

David Bateman <dbateman>
Group Member
Mon 21 Oct 2013 08:20:01 PM UTC, comment #3: 

Do you still think it's better to keep special-casing all the calls to numel instead of patching numel itself?

Jordi GutiƩrrez Hermoso <jordigh>
Group Member
Mon 21 Oct 2013 07:59:19 PM UTC, comment #2: 

That should of course have read horzcat.

Looking at this problem a bit further, the difference between


[s, s]


and


horzcat(s,s)


is that the first goes through the parse tree code pt-mat.cc to construct the matrix list to concatenate, and in this code there are multiple uses of numel and in particular one to test if all the values are scalar. As sparse matrices can't profit from the fact that all elements are scalar in the concatenation this test should be avoided as a call to numel calls the idx_vector::safe_numel method that throws a bad alloc in this case even though it doesn't need to.

Fixed in the changeset

http://hg.savannah.gnu.org/hgweb/octave/rev/ccc0576641f9

D.

David Bateman <dbateman>
Group Member
Mon 21 Oct 2013 06:38:36 PM UTC, comment #1: 

For a workaround horizcat and vertcat seem to work fine. I'm looking at this issue

D.

David Bateman <dbateman>
Group Member
Mon 21 Oct 2013 04:46:21 AM UTC, original submission:  


>> a = speye(100000);
>> b = [a;a];

error: out of memory or dimension too large for Octave's index type

David Spies <dspyz>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dbateman (Posted a comment)
  • -email is unavailable- added by jordigh (Updated the item)
  • -email is unavailable- added by dspyz (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-10-21 dbateman StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2013-10-21 dbateman Assigned toNone dbateman
    2013-10-21 jordigh StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code