bugGNU Octave - Bugs: bug #47175, “dimension too large” error...

 
 

bug #47175: “dimension too large” error when broadcasting to sparse matrix in octave

Submitter:  None
Submitted:  Wed 17 Feb 2016 01:17:31 AM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Documentation
Status:  In Progress Assigned to:  lachlan
Originator Name:  Dan Parshall Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * dev
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sat 02 Jul 2016 01:47:46 AM UTC, comment #4: 

Retagging from "Patch submitted" to "In progress" because the patch doesn't yet have a heuristic for estimating the number of non-zero elements, and it is not serious enough to push a partial solution into 4.2.0.

Lachlan Andrew <lachlan>
Sat 20 Feb 2016 03:38:02 AM UTC, comment #3: 

Repmat already returns sparse output for sparse input.

A big weakness with the bsxfun patch is that it builds the sparse matrix column at a time.  This will probably involve copying the entire matrix for each row processed.  That can be avoided if we know the number of non-zero elements in the final matrix.

Can someone suggest a good heuristic for estimating the number of non-zero elements in the solution?  Ideally, I don't want too many special cases for different values of
  (operator, operand1-shape/sparsity, operand2-shape/sparsity).

Lachlan Andrew <lachlan>
Sat 20 Feb 2016 03:21:37 AM UTC, comment #2: 

The attached patch should allow bsxfun to work.  It is a two-line patch relative to the patch supplied for bug #45219 and bug #40089, but that patch hasn't yet been applied.

We don't always want to return a sparse matrix from bsxfun for a mix of sparse and full.  For example, @plus and @minus should generally return a full matrix.  The attached patch skips one level of optimization for sparse matrices (maybe_optimized_builtin).  The alternative would be to enhance the first stage of optimization to consider which operations should convert full to sparse and which should convert sparse to full.

Dan, could you put this patch through its paces, like timing it and checking it with 64 bit indexing?  I only tried it on a cut-down example that fits in 32 bits.

I'll look into repmat next.

Note: For anyone trying to reproduce this, replace "rep" by "broad" in the definition of toobig.

(file #36404)

Lachlan Andrew <lachlan>
Wed 17 Feb 2016 09:50:39 AM UTC, comment #1: 

The first problem is bsxfun giving a full matrix as output


octave:14> bsxfun(@times,speye(4),ones(4,1))
ans =

   1   0   0   0
   0   1   0   0
   0   0   1   0
   0   0   0   1


Then, of course repmat(broad,...) gives a full matrix, what else?

Marco Caliari <caliari>
Group Member
Wed 17 Feb 2016 01:17:31 AM UTC, original submission:  

32-bit Octave has a limit on the maximum number of elements in an array. I have recompiled from source (following the script at https://github.com/calaba/octave-3.8.2-enable-64-ubuntu-14.04 ), and now have 64-bit indexing.

Nevertheless, when I attempt to perform elementwise multiplication using a broadcast function, I get error: out of memory or dimension too large for Octave's index type

My suspicion is that this is a bug with repmat, and that instead of producing a sparse matrix it generates a full one.  But I don't know Octave's internals well enough to be sure.

Minimal code to reproduce the problem:


function indexerror();
% both of these are formed without error
%    a = zeros (2^32, 1, 'int8');
%    b = zeros (1024*1024*1024*3, 1, 'int8');
%   sizemax    % returns 9223372036854775806

    nnz = 1000            % number of non-zero elements
    rowmax = 250000
    colmax = 100000

    irow = zeros(1,nnz);
    icol = zeros(1,nnz);
    for ind =1:nnz
        irow(ind) = round(rowmax/nnz*ind);
        icol(ind) = round(colmax/nnz*ind);
    end

    sparseMat = sparse(irow,icol,1,rowmax,colmax);

    % column vector to be broadcast
    broad = 1:rowmax;
    broad = broad(:);

    % this gives "dimension too large" error
    toobig = bsxfun(@times,sparseMat,rep);

    % so does this
    toobig2 = sparse(repmat(broad,1,size(sparseMat,2)));
    mult = sparse( sparseMat .* toobig2 );        % never made it this far
end


My current workaround is to do the multiplication inside of a loop, but it's sloooooow:

    % loop over rows, instead of using bsxfun
    mult_loop = sparse([],[],[],rowmax,colmax);
    for ind =1:length(broad);
        mult_loop(ind,:) = broad(ind) * sparseMat(ind,:);
    end


Copying from my SO question:
http://stackoverflow.com/questions/35421967/dimension-too-large-error-when-broadcasting-to-sparse-matrix-in-octave

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #36419:  bug_47175_bsxfun.cset added by lachlan (1KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by lachlan (Updated the item)
  • -email is unavailable- added by caliari (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-07-02 lachlan StatusPatch Submitted In Progress
    2016-02-22 lachlan Attached File- Added bug_47175_bsxfun.cset, #36419
    2016-02-22 lachlan Attached File#36404 Removed
    2016-02-20 lachlan Attached File- Added bug_47175_bsxfun.cset, #36404
        StatusNone Patch Submitted
        Assigned toNone lachlan
        Release3.8.2 dev

    Back to the top

    Powered by Savane 3.14-04e1.
    Corresponding source code