bugGNU Octave - Bugs: bug #47175, “dimension too large” error...

 
 

bug #47175: “dimension too large” error when broadcasting to sparse matrix in octave

Submitted by:  None
Submitted on:  Wed 17 Feb 2016 01:17:31 AM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Documentation
Status: In ProgressAssigned to: Lachlan Andrew <lachlan>
Originator Name: Dan ParshallOriginator Email: -unavailable-
Open/Closed: OpenRelease: dev
Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 02 Jul 2016 01:47:46 AM UTC, comment #4:

Retagging from "Patch submitted" to "In progress" because the patch doesn't yet have a heuristic for estimating the number of non-zero elements, and it is not serious enough to push a partial solution into 4.2.0.

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Sat 20 Feb 2016 03:38:02 AM UTC, comment #3:

Repmat already returns sparse output for sparse input.

A big weakness with the bsxfun patch is that it builds the sparse matrix column at a time. This will probably involve copying the entire matrix for each row processed. That can be avoided if we know the number of non-zero elements in the final matrix.

Can someone suggest a good heuristic for estimating the number of non-zero elements in the solution? Ideally, I don't want too many special cases for different values of
(operator, operand1-shape/sparsity, operand2-shape/sparsity).

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Sat 20 Feb 2016 03:21:37 AM UTC, comment #2:

The attached patch should allow bsxfun to work. It is a two-line patch relative to the patch supplied for bug #45219 and bug #40089, but that patch hasn't yet been applied.

We don't always want to return a sparse matrix from bsxfun for a mix of sparse and full. For example, @plus and @minus should generally return a full matrix. The attached patch skips one level of optimization for sparse matrices (maybe_optimized_builtin). The alternative would be to enhance the first stage of optimization to consider which operations should convert full to sparse and which should convert sparse to full.

Dan, could you put this patch through its paces, like timing it and checking it with 64 bit indexing? I only tried it on a cut-down example that fits in 32 bits.

I'll look into repmat next.

Note: For anyone trying to reproduce this, replace "rep" by "broad" in the definition of toobig.

(file #36404)

Lachlan Andrew <lachlan>
Project MemberIn charge of this item.
Wed 17 Feb 2016 09:50:39 AM UTC, comment #1:

The first problem is bsxfun giving a full matrix as output

Then, of course repmat(broad,...) gives a full matrix, what else?

Marco Caliari <caliari>
Project Member
Wed 17 Feb 2016 01:17:31 AM UTC, original submission:

32-bit Octave has a limit on the maximum number of elements in an array. I have recompiled from source (following the script at https://github.com/calaba/octave-3.8.2-enable-64-ubuntu-14.04 ), and now have 64-bit indexing.

Nevertheless, when I attempt to perform elementwise multiplication using a broadcast function, I get error: out of memory or dimension too large for Octave's index type

My suspicion is that this is a bug with repmat, and that instead of producing a sparse matrix it generates a full one. But I don't know Octave's internals well enough to be sure.

Minimal code to reproduce the problem:

My current workaround is to do the multiplication inside of a loop, but it's sloooooow:

Copying from my SO question:
http://stackoverflow.com/questions/35421967/dimension-too-large-error-when-broadcasting-to-sparse-matrix-in-octave

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #36419:  bug_47175_bsxfun.cset added by lachlan (1KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by lachlan (Updated the item)
  • -unavailable- added by caliari (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 02 Jul 2016 01:47:46 AM UTClachlanStatusPatch Submitted=>In Progress
    Mon 22 Feb 2016 06:05:13 AM UTClachlanAttached File-=>Added bug_47175_bsxfun.cset, #36419
    Mon 22 Feb 2016 06:04:38 AM UTClachlanAttached File#36404=>Removed
    Sat 20 Feb 2016 03:21:37 AM UTClachlanAttached File-=>Added bug_47175_bsxfun.cset, #36404
      StatusNone=>Patch Submitted
      Assigned toNone=>lachlan
      Release3.8.2=>dev

    Back to the top


    Powered by Savane 3.1-cleanup1