bugGNU Octave - Bugs: bug #40712, Running into index errors with...

 
 

bug #40712: Running into index errors with large, 64k square sparse matrices

Submitter:  Paul Jakma <paul>
Submitted:  Tue 26 Nov 2013 04:56:38 PM UTC
   
 
Category:  Libraries Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Duplicate Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 01 Aug 2018 10:36:12 PM UTC, comment #3: 

Marking as a duplicate and closing report.

Octave is now built by default with 64-bit indices which helps avoid some of these situations.

The out-of-memory issue is real because there is no way to avoid converting the sparse matrix to a full one.  With 64-bit indices you can use logical indexing, or choose linear indexing.  For example,


nz = find (D);
D(nz) .^= -0.5;



Rik <rik5>
Group administrator
Wed 23 Nov 2016 11:09:40 PM UTC, comment #2: 

These issues are still present in Octave 4.2.0, but I think they might all be "works for me" or "duplicate":

The first issue in comment #0 is ment to expand to a full matrix, since the 0 entries need to become 0**-0.5 = Inf (similar to bug #36562).

The second issue in comment #0 sounds like a duplicate of bug #40341 (Logical indexing into sparse matrices causes OOM) to me.






Hartmut <hardy>
Sun 22 Mar 2015 02:59:34 PM UTC, comment #1: 

Thanks for your bug report. I can confirm what you are seeing with the current development version of Octave. I can also confirm that building Octave with --enable-64 allows your examples to work. So that might be another viable path to pursue for working with datasets of the size you are talking about, sparse or not.

I am not familiar enough with the sparse implementation to know if this is a bug that deserves fixing so that it works in a default Octave built without 64-bit indexes.

Mike Miller <mtmiller>
Group Member
Tue 26 Nov 2013 04:56:38 PM UTC, original submission:  

I have a 64k × 64 matrix, sparse matrix:

spstats(D)
ans =
  Compressed Column Sparse (rows = 1, cols = 64000, nnz = 15892 [25%])
...

I wanted to calculate D^-0.5, however it gives this error:

D = D**-0.5

error: memory exhausted or requested size too large for range of Octave's index type -- trying to return to prompt

It was suggested to me to on #octave use:

 D(D>0) .^= -0.5

However, this gives the same error. In the end, the following workaround was found, which does work:

Ds = diag(D);
Ds(Ds > 0) .^= -0.5;
Ds = diag(Ds);

It would be good though if the first 2 ways above worked too.

While my matrices at the moment are 64k in size, they are likely to grow. They are actually large graphs, in a 32bit node space. The 64k size will grow and grow over time. Octave has been extremely useful in helping to analyse these graphs. It'd be very useful to me if Octave's index type could support operations on them.

Paul Jakma <paul>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by hardy (Posted a comment)
  • -email is unavailable- added by mtmiller (Updated the item)
  • -email is unavailable- added by paul (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-08-01 rik5 StatusConfirmed Duplicate
        Open/ClosedOpen Closed
    2015-03-22 mtmiller CategoryOctave Function Libraries
    2014-01-19 mtmiller CategoryNone Octave Function
        StatusNone Confirmed
        Release3.6.3 dev

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code