bugGNU Octave - Bugs: bug #42118, COW memory issues when extracting...

 
 

bug #42118: COW memory issues when extracting small slices from large arrays

Submitted by:  Jan Dohl <derjan>
Submitted on:  Mon 14 Apr 2014 04:49:55 PM UTC  
 
Category: Octave FunctionSeverity: 4 - Important
Priority: 5 - NormalItem Group: Segfault, Bus Error, etc.
Status: ConfirmedAssigned to: John W. Eaton <jwe>
Originator Name: Open/Closed: Open
Release: 3.8.1Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 08 Mar 2017 07:19:37 PM UTC, comment #7:

Sorry, I forgot to attach the patch.

(file #39945)

John W. Eaton <jwe>
Project AdministratorIn charge of this item.
Wed 08 Mar 2017 06:30:22 PM UTC, comment #6:

Array slicing can be completely disabled with the attached changeset.

I'm not sure whether there is a best way to decide to slice. Making it configurable brings in other issues, like where to store the slice limits/options, whether they should be global, etc.

John W. Eaton <jwe>
Project AdministratorIn charge of this item.
Sat 16 Jan 2016 03:58:54 AM UTC, comment #5:

This was put on hold for 4.0.0, but with a bugfix 4.0.1 coming up, should it be revisited?

Michael, you suggested cleaning memory if the reference count drops to 1. What if there are two small slices? The object would still stay in memory.

It may be better to have a separate slice counter, or even a list of slice objects, associated with array data types. If the slice counter equals the the object counter, then you could do whatever memory optimisation. The existence of this extra data shouldn't slow down any non-slice operations except for deletion (except for the slightly reduced memory locality).

One problem with jwe's idea of copying if the fraction is small, is that it is common to walk through a matrix row at a time or column at a time. These slices are likely to be small fractions of the total size (1% or 0.1%). It would be a shame to lose all copy-on-write benefits for that common case.

This is a bug example of a problem I've often thought about: whos shows the nominal memory sizes of objects, but doesn't show how much overhead there is. For example, a huge sparse array can have zero bytes of data, but still have a huge amount of column overhead. It would be great to have a version of whos that shows memory consumed by each variable. Is that feasible?

Lachlan Andrew <lachlan>
Project Member
Mon 02 Feb 2015 04:11:39 PM UTC, comment #4:

This requires some good thinking, but I think one possible approach is to check when the refcount of the underlying Array object reaches 1 and in that case optimize the memory used by the object.

For instance:
if (slice_data != rep->data || slice_len != rep->len)
do_optimize_memory ();

But I'm not sure this is technically do-able...

Michael Goffioul <goffioul>
Project Member
Mon 02 Feb 2015 03:49:32 PM UTC, comment #3:

Well it's good that there is a strategy for fixing it. It appears that this is an infrequent enough occurrence not to be a blocker for the 4.0 release. I'm marking this as WON'T FIX for the 4.0 release (http://wiki.octave.org/Bug_Fix_List_-_4.0_Release). After 4.0 we can take this up.

Rik <rik5>
Project Administrator
Mon 02 Feb 2015 06:52:15 AM UTC, comment #2:

Yes, as I recall, contiguous slices are simply additional references to the data. Maybe a reasonable compromise is to make actual copies if the size of the slice is relatively small compared to the overall size of the array? I suppose this could be configurable.

John W. Eaton <jwe>
Project AdministratorIn charge of this item.
Mon 02 Feb 2015 05:46:48 AM UTC, comment #1:

Confirmed. The behavior also affects the development branch as of 2/1/15.

Rik <rik5>
Project Administrator
Mon 14 Apr 2014 04:49:55 PM UTC, original submission:

I'm porting a Matlab script to Octave that receives a large amount of data points (hundrets of thousands to several million) every second. The task of the script is to extract the interesting part of the data which is smaller by orders of magnitude (10-100 samples). Those interesting parts are then to be saved. The script is supposed to run for several hours at a time.

A mockup version of the script is as follows:

WARNING: Do not run this on a machine which runs important processes. It will very quickly use all available memory which might cause your machine to become slow or even crash if no limits are in place.

As said before, this script will very quickly use up all available memory and eventually cause Octave to crash. However, if the process is aborted before the crash, 'whos' will just show a very small memory usage by the existing variables which I expected.

From IRC I've got the idea that this might be a COW problem and datas{i} = rxdata(1:10) actually creates a reference to the whole rxdata-array and keeps it from going out of memory. Hence, not even clearing rxdata after each run will cause the problem to go away. Overwriting one element of the copied slice with a copy of itself will however trigger the copy mechanism, remove the reference and the program works as expected.

Problem was verified with self-built Octave 3.8.1. as well as 3.6.2. from Debian Stable.

Jan Dohl <derjan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #39945:  diffs.txt added by jwe (581B - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by lachlan (Posted a comment)
  • -unavailable- added by goffioul (Posted a comment)
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by rik5
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by derjan (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 08 Mar 2017 07:19:37 PM UTCjweAttached File-=>Added diffs.txt, #39945
    Fri 01 Jul 2016 05:28:09 PM UTCrik5Severity3 - Normal=>4 - Important
    Mon 02 Feb 2015 05:47:06 AM UTCrik5Carbon-Copy-=>Added -unavailable-
    Mon 02 Feb 2015 05:46:48 AM UTCrik5StatusNone=>Confirmed
      Assigned toNone=>jwe

    Back to the top


    Powered by Savane 3.1-cleanup1