bugGNU Octave - Bugs: bug #42118, COW memory issues when extracting...

 
 

bug #42118: COW memory issues when extracting small slices from large arrays

Submitted by:  Jan Dohl <derjan>
Submitted on:  Mon 14 Apr 2014 04:49:55 PM UTC  
 
Category:  Octave Function Severity:  4 - Important
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Confirmed Assigned to:  John W. Eaton <jwe>
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Fri 09 Mar 2018 10:48:17 PM UTC, comment #9:

My first thought is that it isn't worth the maintenance burden.

Rik <rik5>
Project Administrator
Fri 09 Mar 2018 10:38:39 PM UTC, comment #8:

Ping.

Any opinions about whether we should "fix" this issue by disabling slicing as in the patch from comment #7?

Slicing can definitely help in some cases, but as we see here, it can also cause a lot of trouble. I'm not sure that adding more complexity is worth the maintenance cost.

Comments?

John W. Eaton <jwe>
Project AdministratorIn charge of this item.
Wed 08 Mar 2017 07:19:37 PM UTC, comment #7:

Sorry, I forgot to attach the patch.

(file #39945)

John W. Eaton <jwe>
Project AdministratorIn charge of this item.
Wed 08 Mar 2017 06:30:22 PM UTC, comment #6:

Array slicing can be completely disabled with the attached changeset.

I'm not sure whether there is a best way to decide to slice. Making it configurable brings in other issues, like where to store the slice limits/options, whether they should be global, etc.

John W. Eaton <jwe>
Project AdministratorIn charge of this item.
Sat 16 Jan 2016 03:58:54 AM UTC, comment #5:

This was put on hold for 4.0.0, but with a bugfix 4.0.1 coming up, should it be revisited?

Michael, you suggested cleaning memory if the reference count drops to 1. What if there are two small slices? The object would still stay in memory.

It may be better to have a separate slice counter, or even a list of slice objects, associated with array data types. If the slice counter equals the the object counter, then you could do whatever memory optimisation. The existence of this extra data shouldn't slow down any non-slice operations except for deletion (except for the slightly reduced memory locality).

One problem with jwe's idea of copying if the fraction is small, is that it is common to walk through a matrix row at a time or column at a time. These slices are likely to be small fractions of the total size (1% or 0.1%). It would be a shame to lose all copy-on-write benefits for that common case.

This is a bug example of a problem I've often thought about: whos shows the nominal memory sizes of objects, but doesn't show how much overhead there is. For example, a huge sparse array can have zero bytes of data, but still have a huge amount of column overhead. It would be great to have a version of whos that shows memory consumed by each variable. Is that feasible?

Lachlan Andrew <lachlan>
Mon 02 Feb 2015 04:11:39 PM UTC, comment #4:

This requires some good thinking, but I think one possible approach is to check when the refcount of the underlying Array object reaches 1 and in that case optimize the memory used by the object.

For instance:
if (slice_data != rep->data || slice_len != rep->len)
do_optimize_memory ();

But I'm not sure this is technically do-able...

Michael Goffioul <goffioul>
Mon 02 Feb 2015 03:49:32 PM UTC, comment #3:

Well it's good that there is a strategy for fixing it. It appears that this is an infrequent enough occurrence not to be a blocker for the 4.0 release. I'm marking this as WON'T FIX for the 4.0 release (http://wiki.octave.org/Bug_Fix_List_-_4.0_Release). After 4.0 we can take this up.

Rik <rik5>
Project Administrator
Mon 02 Feb 2015 06:52:15 AM UTC, comment #2:

Yes, as I recall, contiguous slices are simply additional references to the data. Maybe a reasonable compromise is to make actual copies if the size of the slice is relatively small compared to the overall size of the array? I suppose this could be configurable.

John W. Eaton <jwe>
Project AdministratorIn charge of this item.
Mon 02 Feb 2015 05:46:48 AM UTC, comment #1:

Confirmed. The behavior also affects the development branch as of 2/1/15.

Rik <rik5>
Project Administrator
Mon 14 Apr 2014 04:49:55 PM UTC, original submission:

I'm porting a Matlab script to Octave that receives a large amount of data points (hundrets of thousands to several million) every second. The task of the script is to extract the interesting part of the data which is smaller by orders of magnitude (10-100 samples). Those interesting parts are then to be saved. The script is supposed to run for several hours at a time.

A mockup version of the script is as follows:

WARNING: Do not run this on a machine which runs important processes. It will very quickly use all available memory which might cause your machine to become slow or even crash if no limits are in place.

As said before, this script will very quickly use up all available memory and eventually cause Octave to crash. However, if the process is aborted before the crash, 'whos' will just show a very small memory usage by the existing variables which I expected.

From IRC I've got the idea that this might be a COW problem and datas{i} = rxdata(1:10) actually creates a reference to the whole rxdata-array and keeps it from going out of memory. Hence, not even clearing rxdata after each run will cause the problem to go away. Overwriting one element of the copied slice with a copy of itself will however trigger the copy mechanism, remove the reference and the program works as expected.

Problem was verified with self-built Octave 3.8.1. as well as 3.6.2. from Debian Stable.

Jan Dohl <derjan>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #39945:  diffs.txt added by jwe (581B - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by lachlan (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by derjan (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-04-27 mtmiller Carbon-CopyRemoved 80942 => -
    2018-04-27 mtmiller Release3.8.1 => dev
    2017-03-08 jwe Attached File- => Added diffs.txt, #39945
    2016-07-01 rik5 Severity3 - Normal => 4 - Important
    2015-02-02 rik5 Carbon-Copy- => Added -email is unavailable-
    2015-02-02 rik5 StatusNone => Confirmed
        Assigned toNone => jwe

    Back to the top


    Powered by Savane 3.3