bugGNU Octave - Bugs: bug #39387, With large cell-arrays of numeric...

 
 

bug #39387: With large cell-arrays of numeric scalars, cell2mat is slow and memory-hungry.

Submitter:  Olaf Till <i7tiol>
Submitted:  Mon 01 Jul 2013 05:15:42 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Fixed Assigned to:  None
Originator Name:  Olaf Till Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 18 Jul 2013 07:10:29 PM UTC, comment #2: 

Great patches!  I applied them to the development branch.  I also re-vamped the input validation in another changeset (http://hg.savannah.gnu.org/hgweb/octave/rev/19b7c7412a63) which got me another ~25% savings.

Rik <rik5>
Group administrator
Mon 01 Jul 2013 05:53:47 PM UTC, comment #1: 

Sorry, I made a mistake in a test in cellfun, leading to a test failure. Corrected patch attached ("cellfun-isnumeric-2.changeset").

(file #28474)

Olaf Till <i7tiol>
Group Member
Mon 01 Jul 2013 05:15:42 PM UTC, original submission:  

The patches will be attached as soon as I have the bug number for the commit messages.
 
Slowness of cell2mat:


octave:5> c = zeros (10000000, 3);
octave:6> tic; c = num2cell (c); toc
Elapsed time is 1.89468 seconds.
octave:7> tic; d = cell2mat (c); toc
Elapsed time is 355.079 seconds.


The latter command produced signes of thrashing due to memory exhaustion (severe disk access, slowing down of other applications).

(Same for reversed dimensions:


octave:5> c = zeros (3, 10000000);
octave:6> c = num2cell (c);
octave:7> tic; d = cell2mat (c); toc
Elapsed time is 438 seconds.

)

Slowness is partially due to a call, during argument checking:


octave:66> tic; cellfun ("isnumeric", c); toc
Elapsed time is 30.1 seconds.


which is made although there is no specialized method "isnumeric" in cellfun.

The rest of slowness and the thrashing seem to be due to the
cellfun("cat", ...)  calls, which are necessary for cell-arrays
containing non-scalar elements. The code strives to minimize the number of cat() calls (within cellfun() calls) but seemingly has the sort order of the dimensions mistaken, it should be descending (which was the intention also according to the comments in the code). After correcting this, there is no thrashing (or less?) and the function is faster:


octave:2> c = zeros (3, 10000000);
octave:3> c = num2cell (c);
octave:5> tic; d = cell2mat (c); toc
Elapsed time is 61.2661 seconds.


and with reversed dimensions


octave:6> c = zeros (10000000, 3);
octave:7> c = num2cell (c);
octave:8> tic; d = cell2mat (c); toc
Elapsed time is 55.0627 seconds.


The attached cell2mat changeset makes the above correction and
furthermore introduces a special case for scalar-only elements, where one single cat() call can be used. This special case shortens duration from (here already with the cellfun changeset (see below), so both are faster than before):


octave:5> tic; cell2mat (c); toc
Elapsed time is 27.9482 seconds.


to


octave:6> tic; cell2mat (c); toc
Elapsed time is 9.9075 seconds.


I'd suggest applying this changeset to the stable branch.

The attached cellfun changeset adds the "isnumeric" special case to cellfun. With this:


octave:7> tic; cellfun ("isnumeric", c); toc
Elapsed time is 0.326492 seconds.


(was 30 s before, see above), which adds to the speed enhancement of cell2mat.

I'd suggest applying this changeset to the default branch.

Olaf

Olaf Till <i7tiol>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #28474:  cellfun-isnumeric-2.changeset added by i7tiol (2KiB - application/octet-stream)
file #28472:  cell2mat.changeset added by i7tiol (2KiB - application/octet-stream)
file #28473:  cellfun-isnumeric.changeset added by i7tiol (2KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by i7tiol (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-07-18 rik5 StatusNone Fixed
        Open/ClosedOpen Closed
        Summary(Patches attached.) With large cell-arrays of numeric scalars, cell2mat is slow and memory-hungry. With large cell-arrays of numeric scalars, cell2mat is slow and memory-hungry.
    2013-07-01 i7tiol Attached File- Added cellfun-isnumeric-2.changeset, #28474
    2013-07-01 i7tiol Attached File- Added cell2mat.changeset, #28472
        Attached File- Added cellfun-isnumeric.changeset, #28473

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code