bugGNU Octave - Bugs: bug #47175, “dimension too large” error...


bug #47175: “dimension too large” error when broadcasting to sparse matrix in octave

Submitted by:  None
Submitted on:  Wed 17 Feb 2016 01:17:31 AM UTC  
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Documentation
Status:  In Progress Assigned to:  Lachlan Andrew <lachlan>
Originator Name:  Dan Parshall Originator Email:  -email is unavailable-
Open/Closed:  Open Release:  dev
Operating System:  GNU/Linux

Add a New Comment(Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


Sat 02 Jul 2016 01:47:46 AM UTC, comment #4:

Retagging from "Patch submitted" to "In progress" because the patch doesn't yet have a heuristic for estimating the number of non-zero elements, and it is not serious enough to push a partial solution into 4.2.0.

Lachlan Andrew <lachlan>In charge of this item.
Sat 20 Feb 2016 03:38:02 AM UTC, comment #3:

Repmat already returns sparse output for sparse input.

A big weakness with the bsxfun patch is that it builds the sparse matrix column at a time. This will probably involve copying the entire matrix for each row processed. That can be avoided if we know the number of non-zero elements in the final matrix.

Can someone suggest a good heuristic for estimating the number of non-zero elements in the solution? Ideally, I don't want too many special cases for different values of
(operator, operand1-shape/sparsity, operand2-shape/sparsity).

Lachlan Andrew <lachlan>In charge of this item.
Sat 20 Feb 2016 03:21:37 AM UTC, comment #2:

The attached patch should allow bsxfun to work. It is a two-line patch relative to the patch supplied for bug #45219 and bug #40089, but that patch hasn't yet been applied.

We don't always want to return a sparse matrix from bsxfun for a mix of sparse and full. For example, @plus and @minus should generally return a full matrix. The attached patch skips one level of optimization for sparse matrices (maybe_optimized_builtin). The alternative would be to enhance the first stage of optimization to consider which operations should convert full to sparse and which should convert sparse to full.

Dan, could you put this patch through its paces, like timing it and checking it with 64 bit indexing? I only tried it on a cut-down example that fits in 32 bits.

I'll look into repmat next.

Note: For anyone trying to reproduce this, replace "rep" by "broad" in the definition of toobig.

(file #36404)

Lachlan Andrew <lachlan>In charge of this item.
Wed 17 Feb 2016 09:50:39 AM UTC, comment #1:

The first problem is bsxfun giving a full matrix as output

Then, of course repmat(broad,...) gives a full matrix, what else?

Marco Caliari <caliari>
Project Member
Wed 17 Feb 2016 01:17:31 AM UTC, original submission:

32-bit Octave has a limit on the maximum number of elements in an array. I have recompiled from source (following the script at https://github.com/calaba/octave-3.8.2-enable-64-ubuntu-14.04 ), and now have 64-bit indexing.

Nevertheless, when I attempt to perform elementwise multiplication using a broadcast function, I get error: out of memory or dimension too large for Octave's index type

My suspicion is that this is a bug with repmat, and that instead of producing a sparse matrix it generates a full one. But I don't know Octave's internals well enough to be sure.

Minimal code to reproduce the problem:

My current workaround is to do the multiplication inside of a loop, but it's sloooooow:

Copying from my SO question:



(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #36419:  bug_47175_bsxfun.cset added by lachlan (1KiB - application/octet-stream)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by lachlan (Updated the item)
  • -email is unavailable- added by caliari (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.




    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-07-02 lachlan StatusPatch Submitted => In Progress
    2016-02-22 lachlan Attached File- => Added bug_47175_bsxfun.cset, #36419
    2016-02-22 lachlan Attached File#36404 => Removed
    2016-02-20 lachlan Attached File- => Added bug_47175_bsxfun.cset, #36404
        StatusNone => Patch Submitted
        Assigned toNone => lachlan
        Release3.8.2 => dev

    Back to the top

    Powered by Savane 3.3