bugGNU Octave - Bugs: bug #65669, Cell array construction is...

 
 

bug #65669: Cell array construction is memory-intensive

Submitter:  Henry Shu <mlguru>
Submitted:  Wed 01 May 2024 09:36:50 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Confirmed Assigned to:  None
Originator Name:  Henry Shu Open/Closed:  * Open
Release:  * 9.1.0 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 06 May 2024 02:10:03 PM UTC, comment #32: 

It may be not as much as the actual value, but the fact that MALLOC_TRIM_THRESHOLD_ is set (it prevents dynamic threshhold escalation). From "man mallopt):


Note: Nowadays, glibc uses a dynamic mmap threshold by
              default.  The initial value of the threshold is 128*1024,
              but when blocks larger than the current threshold and less
              than or equal to DEFAULT_MMAP_THRESHOLD_MAX are freed, the
              threshold is adjusted upward to the size of the freed
              block.  When dynamic mmap thresholding is in effect, the
              threshold for trimming the heap is also dynamically
              adjusted to be twice the dynamic mmap threshold.  Dynamic
              adjustment of the mmap threshold is disabled if any of the
              M_TRIM_THRESHOLD, M_TOP_PAD, M_MMAP_THRESHOLD, or
              M_MMAP_MAX parameters is set.


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 06 May 2024 02:02:17 PM UTC, comment #31: 

E.g. this is on my machine (Centos Stream 9):


$ octave
<...deleted...>

octave:1> disp("---");
disp("Before anything is done:");
memory;

a = magic(1000);
b = cell(1000, 1000);
for r = 1:1000
  for c = 1:1000
    b{r, c} = a(r, c) + rand;  # Add something to eliminate any chance of lazy copy.
  end
end

disp("---");
disp("Large matrix and large cell are allocated:");
disp(sprintf("Matrix a uses %d kiB.", whos("a").bytes / 1024));
disp(sprintf("Cell b uses %d kiB.", whos("b").bytes / 1024));
memory;

clear a
disp("---");
disp("Only cell b is allocated:");
memory;

clear all
disp("---");
disp("Right after clear all:");
memory
---
Before anything is done:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  32517436 kiB,  swap:  34603004 kiB
Octave    RAM:     97972 kiB,  virt:   5099748 kiB
Free      RAM:  23872124 kiB,  swap:  34603004 kiB
Available RAM:  28408476 kiB, total:  63011480 kiB
---
Large matrix and large cell are allocated:
Matrix a uses 7812.5 kiB.
Cell b uses 7812.5 kiB.
Octave is running on x86_64-pc-linux-gnu
System    RAM:  32517436 kiB,  swap:  34603004 kiB
Octave    RAM:    161568 kiB,  virt:   5107564 kiB
Free      RAM:  23811472 kiB,  swap:  34603004 kiB
Available RAM:  28347876 kiB, total:  62950880 kiB
---
Only cell b is allocated:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  32517436 kiB,  swap:  34603004 kiB
Octave    RAM:    153820 kiB,  virt:   5099748 kiB
Free      RAM:  23818032 kiB,  swap:  34603004 kiB
Available RAM:  28354436 kiB, total:  62957440 kiB
---
Right after clear all:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  32517436 kiB,  swap:  34603004 kiB
Octave    RAM:    153820 kiB,  virt:   5099644 kiB
Free      RAM:  23818032 kiB,  swap:  34603004 kiB
Available RAM:  28354436 kiB, total:  62957440 kiB
octave:20>


and now with
$ MALLOC_TRIM_THRESHOLD_=100000 octave
GNU Octave, version 9.1.91

<...deleted...>

octave:1> disp("---");
disp("Before anything is done:");
memory;

a = magic(1000);
b = cell(1000, 1000);
for r = 1:1000
  for c = 1:1000
    b{r, c} = a(r, c) + rand;  # Add something to eliminate any chance of lazy copy.
  end
end

disp("---");
disp("Large matrix and large cell are allocated:");
disp(sprintf("Matrix a uses %d kiB.", whos("a").bytes / 1024));
disp(sprintf("Cell b uses %d kiB.", whos("b").bytes / 1024));
memory;

clear a
disp("---");
disp("Only cell b is allocated:");
memory;

clear all
disp("---");
disp("Right after clear all:");
memory
---
Before anything is done:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  32517436 kiB,  swap:  34603004 kiB
Octave    RAM:     95644 kiB,  virt:   5099508 kiB
Free      RAM:  23881836 kiB,  swap:  34603004 kiB
Available RAM:  28418272 kiB, total:  63021276 kiB
---
Large matrix and large cell are allocated:
Matrix a uses 7812.5 kiB.
Cell b uses 7812.5 kiB.
Octave is running on x86_64-pc-linux-gnu
System    RAM:  32517436 kiB,  swap:  34603004 kiB
Octave    RAM:    159472 kiB,  virt:   5115140 kiB
Free      RAM:  23819548 kiB,  swap:  34603004 kiB
Available RAM:  28356024 kiB, total:  62959028 kiB
---
Only cell b is allocated:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  32517436 kiB,  swap:  34603004 kiB
Octave    RAM:    151736 kiB,  virt:   5107324 kiB
Free      RAM:  23826296 kiB,  swap:  34603004 kiB
Available RAM:  28362772 kiB, total:  62965776 kiB
---
Right after clear all:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  32517436 kiB,  swap:  34603004 kiB
Octave    RAM:    143920 kiB,  virt:   5099404 kiB
Free      RAM:  23834620 kiB,  swap:  34603004 kiB
Available RAM:  28371096 kiB, total:  62974100 kiB
octave:20>

-verbatim-

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 06 May 2024 01:56:29 PM UTC, comment #30: 


Does it help if you set MALLOC_TRIM_THRESHOLD_ to some low value
prior to starting octave?

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 06 May 2024 01:17:48 PM UTC, comment #29: 


> The freed nodes should be available even though they are not collected by whoever (glibc, linux?) manages the heap, right?


Yes, that is/seems the case, as reported in comment #18 (transcribed below).  And the freed memory/nodes are available even to a subprocess: I tested this with parcellfun (version 4.0.2)

To another process though, those memory/nodes appear to be still in use.  The pain point is that we can no longer run another memory-intensive process without exiting/crashing this first Octave process.

comment #18: > E.g. with the magic square test with size = 1000, then doing that test 100 times gives the same unexplained memory usage as running it just once.

Henry Shu <mlguru>
Sun 05 May 2024 08:05:50 PM UTC, comment #28: 


> I am guessing that using a memory pool for allocating and deallocating objects would help mitigate this behavior by preventing such large allocations in the first place, so that even if Linux retains memory like it does, there wouldn't be that much to lose. But I can't be certain. There are such object pool patches in the Octave patch tracker. I used to keep them up to date with the codebase but haven't used them for a while.


I don't think those patches would help here since the pool is only like 8 objects deep.

I ran:

for i = 1:100000
    mem = memory; mem.mem_used_octave
    a = num2cell(ones(12000));
    clear a
    mem = memory; mem.mem_used_octave
    disp("")
end


and there seem to be no actual problem. Is this more than a reporting inconsistency?

The freed nodes should be available even though they are not collected by whoever (glibc, linux?) manages the heap, right? I think there are collection sweeps.

Each cell element will construct a octave_value object on the heap, so there are a lot of pointers.

Petter <petter>
Sun 05 May 2024 12:55:32 AM UTC, comment #27: 

Thank you @arungiridhar and @nrjank for helping out with this Linux conundrum.  Much truly appreciated.

For a little bit of perspective: I am doing some heavy-duty machine learning training that includes an allocation of 40+ GB of fresh data in memory repeatedly for thousands of times.  The issue as reported in this bug would be a showstopper or at least a severe inconvenience.

I will keep a cautious eye on the malloc_trim solution going forward and share here any issues encountered if any.

And if I may add, doing machine learning with Octave is much easier, more efficient, productive, and intuitive than with Python.  This is not an opinion but based on facts, and I can share more in another thread.

Henry Shu <mlguru>
Sat 04 May 2024 08:39:09 PM UTC, comment #26: 

I'm glad that solution works out for you to make the OS show the freed memory. There doesn't seem to be a direct equivalent of the C function malloc_trim in C++17, so it cannot be added as is to Octave core itself, but your solution will be available in this bug report.

If I understand, the reason only Cell types show this problem is because of the construction of a large number of temp objects during cell array construction that are freed correctly but the OS thinks it knows better. With other data types, it's one big allocation, above the glibc threshold, so it does get returned correctly.

Arun Giridhar <arungiridhar>
Group Member
Sat 04 May 2024 01:12:50 PM UTC, comment #25: 

After reading up on this, I wrote a quick oct-file


#include <octave/oct.h>
#include <malloc.h>

DEFUN_DLD (malloc_trim_oct, args, , "Just a quick test") {
  int pad = args(0).scalar_value();
  int res = malloc_trim(pad);
  octave_value_list argout(1);
  argout(0) = res;
  return argout;
}


I compiled it as


mkoctfile malloc_trim_oct.cpp -o malloc_trim_oct.oct


Running the script from comment #17 again to get the memory-hogging behavior just like before:


---
Before anything is done:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  65894144 kiB,  swap:   8387580 kiB
Octave    RAM:    120552 kiB,  virt:  21963236 kiB
Free      RAM:  60328360 kiB,  swap:   8387580 kiB
Available RAM:  64392608 kiB, total:  72780188 kiB
---
Large matrix and large cell are allocated:
Matrix a uses 7812.5 kiB.
Cell b uses 7812.5 kiB.
Octave is running on x86_64-pc-linux-gnu
System    RAM:  65894144 kiB,  swap:   8387580 kiB
Octave    RAM:    183160 kiB,  virt:  22036588 kiB
Free      RAM:  60266716 kiB,  swap:   8387580 kiB
Available RAM:  64330992 kiB, total:  72718572 kiB
---
Only cell b is allocated:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  65894144 kiB,  swap:   8387580 kiB
Octave    RAM:    175396 kiB,  virt:  22028772 kiB
Free      RAM:  60270204 kiB,  swap:   8387580 kiB
Available RAM:  64334480 kiB, total:  72722060 kiB
---
Right after clear all:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  65894144 kiB,  swap:   8387580 kiB
Octave    RAM:    175140 kiB,  virt:  22028236 kiB
Free      RAM:  60270204 kiB,  swap:   8387580 kiB
Available RAM:  64334480 kiB, total:  72722060 kiB


And it seems on my system I can now release some memory back to OS by doing:


octave:2> res = malloc_trim_oct(0); disp(res); memory;
1
Octave is running on x86_64-pc-linux-gnu
System    RAM:  65894144 kiB,  swap:   8387580 kiB
Octave    RAM:    112108 kiB,  virt:  22028156 kiB
Free      RAM:  60336028 kiB,  swap:   8387580 kiB
Available RAM:  64400560 kiB, total:  72788140 kiB


Henry Shu <mlguru>
Fri 03 May 2024 02:37:18 PM UTC, comment #24: 

Here's what I tried:

1. At the Linux shell prompt, type "cat /proc/meminfo >/tmp/base".

2. Start Octave, run one of the commands below.

3. Keeping Octave running, type "cat /proc/meminfo >/tmp/FOO" in the shell prompt, for different FOO corresponding to the commands below.

4. Exit Octave.

5. Repeat steps 2-3-4 for each of the below Octave commands.

6. At the Linux shell prompt, type "cat /proc/meminfo >/tmp/after".

Octave commands:


tic; A = magic (3e3); toc, B = num2cell (A); toc, clear; whos



clear all; tic; n = 3e3; A = magic(n); toc, B = cell(n, n); toc, for i = n:-1:1, for j = n:-1:1, B{i,j} = A(i,j); end; end; clear i j; whos; toc; clear; whos



tic; A = magic (3e3); toc, B = A; toc, clear; whos


Finally, I put the output of the various files into a spreadsheet and saw which rows (field names) are different among the different columns (individual cases). (I did not copy-paste into the spreadsheet while running the Octave tests because I didn't want LibreOffice Calc to interfere with the memory stats.)

In my case, the root cause for the difference was "AnonPages" and "Active(anon)", which were highest for the two cell cases, intermediate for the B = A case, and lowest for the base and after cases. This in turn increased the "Committed_AS" field for the two cell cases.

Reading the descriptions of the above terms on https://www.kernel.org/doc/Documentation/filesystems/proc.txt it says:

AnonPages: "Non-file backed pages mapped into userspace page tables". This seems equivalent to "dynamically allocated memory", which we already knew. https://unix.stackexchange.com/a/677020

Committed_AS: "... A process which mallocs 1G of memory but only touches 300M of it will show up as using 1G. ..."

It seems to be allocating the full size of the cell array even if it doesn't need that much during construction. The constructor code in Cell.h and Cell.cc looks pretty similar to other Array-derived classes, so the overallocation is a cell-specific unintended side-effect with local Array<octave_value> objects. (Only Cell objects use Array<object_value> types, not other Array-derived classes.) It might be the case that it creates several temporary Array<octave_value> objects for constructing each Cell element, so it multiplies usage by the factor that we see. But even though that memory is deallocated properly (ASAN makes no complaints about the cell case, at least none additional to what it usually complains about Octave), it hangs around as AnonPages where Linux thinks it might come in use. https://unix.stackexchange.com/questions/524846/why-does-my-system-use-more-ram-after-an-hour-of-usage#comment970833_524853

I am guessing that using a memory pool for allocating and deallocating objects would help mitigate this behavior by preventing such large allocations in the first place, so that even if Linux retains memory like it does, there wouldn't be that much to lose. But I can't be certain. There are such object pool patches in the Octave patch tracker. I used to keep them up to date with the codebase but haven't used them for a while.

Arun Giridhar <arungiridhar>
Group Member
Fri 03 May 2024 02:12:08 PM UTC, comment #23: 

Is this compilation warning (from Octave-10.0.0) of any help here?

:
  CXX      libinterp/octave-value/liboctave_value_la-ov-base.lo
../dev_p/libinterp/octave-value/ov-base-mat-inst.cc:129:30: warning: type attributes ignored after type is already defined [-Wattributes]
  129 | template class OCTINTERP_API octave_base_matrix<Cell>;
:


Philip Nienhuis <philipnienhuis>
Group Member
Fri 03 May 2024 11:28:55 AM UTC, comment #22: 

Good questions all. This code displays the same memory-hogging behavior as num2cell for me:


n = 1e3;  # or any size
A = magic (n);
B = cell (n, n);
for i = n:-1:1
  for j = n:-1:1
    B{i,j} = A(i.j);  # or also cell (A(i, j))
  end
end


The other test is admittedly difficult to get in the correct zone, based on a combination of array sizes and swap space available. I myself stumbled on it after many hours.

I added cout statements to the various constructors in Cell and Array to see which ones are being called. The only crucial difference I could isolate was this: though there are multiple classes that all inherit from Array (searched by grepping for "public Array" in the codebase), the others use native types for their arrays like bool or char, and only Cell uses Array<octave_type>, the only combination that includes itself. I don't know why, but somehow it is using much more temp space than the others that the OS doesn't return easily. If there's a way to move a temp-created octave_value into an array element without creating a copy, it might help but writing such a Cell constructor gave so many errors about const vs non-const.

I'll look today into mitigations. Not sure what I'll find.

Arun Giridhar <arungiridhar>
Group Member
Fri 03 May 2024 05:15:58 AM UTC, comment #21: 

regarding num2cell, maybe a better question:

if the following two cell arrays are the same:

octave:1> a = cell(10);a(:)=1;
octave:2> b = num2cell(ones(10));
octave:3> assert(a,b)  ##no error
octave:4> whos
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name        Size                     Bytes  Class
  ====   ====        ====                     =====  =====
         a          10x10                       800  cell
         b          10x10                       800  cell

Total is 200 elements using 1600 bytes


Why does the one created by num2cell persist in occupying so much more memory, and why is the amount released on a clear only equal to the size of the actual data in the cell array (the same size as the amount occupied and released by the non-num2cell one)?

see:

octave:1> memory
Octave is running on x86_64-pc-linux-gnu
System    RAM:  10138252 kiB,  swap:    945368 kiB
Octave    RAM:     89964 kiB,  virt:    442888 kiB
Free      RAM:   9135308 kiB,  swap:    474060 kiB
Available RAM:   9199088 kiB, total:   9673148 kiB

octave:2> a = cell(10000);a(:)=1;

octave:3> memory
Octave is running on x86_64-pc-linux-gnu
System    RAM:  10138252 kiB,  swap:    945368 kiB
Octave    RAM:    872428 kiB,  virt:   1224140 kiB
Free      RAM:   8364440 kiB,  swap:    474060 kiB
Available RAM:   8428228 kiB, total:   8902288 kiB

octave:4> b = num2cell(ones(10000));

octave:5> memory
Octave is running on x86_64-pc-linux-gnu
System    RAM:  10138252 kiB,  swap:    945368 kiB
Octave    RAM:   6341060 kiB,  virt:   6658448 kiB
Free      RAM:   2890076 kiB,  swap:    474060 kiB
Available RAM:   2953900 kiB, total:   3427960 kiB

octave:6> whos
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name        Size                     Bytes  Class
  ====   ====        ====                     =====  =====
         a       10000x10000              800000000  cell
         b       10000x10000              800000000  cell

Total is 200000000 elements using 1600000000 bytes

octave:7> clear a

octave:8> memory
Octave is running on x86_64-pc-linux-gnu
System    RAM:  10138252 kiB,  swap:    945368 kiB
Octave    RAM:   5559964 kiB,  virt:   5877196 kiB
Free      RAM:   3662280 kiB,  swap:    474060 kiB
Available RAM:   3726112 kiB, total:   4200172 kiB

octave:9> clear b

octave:10> memory
Octave is running on x86_64-pc-linux-gnu
System    RAM:  10138252 kiB,  swap:    945368 kiB
Octave    RAM:   4778848 kiB,  virt:   5095944 kiB
Free      RAM:   4434484 kiB,  swap:    474060 kiB
Available RAM:   4498324 kiB, total:   4972384 kiB

octave:11> whos


the data size is only about 800MB, and that's about the size of 'a' and the size of the decrease in octave-held memory when clearing 'b'.

it makes sense that num2cell uses more memory to create the cell.
 but every other operation I see in octave that temporarily uses more memory for an operation releases it immediately. i switched from magic(10000) to ones(10000) because magic has a higher temporary memory use that killed octave when I was trying to sneak up on the process1/process2 memory limit test you suggested. the only operation i've seen that doesn't release memory like this after clearing data is num2cell.

Nicholas Jankowski <nrjank>
Group Member
Fri 03 May 2024 04:33:01 AM UTC, comment #20: 

As i try to play with your example, when i try to find the right array for 2nd octave process, if it's too small I get no change but even slightly too large and it Kills the 2nd process. It seems it won't touch the first process's memory allocation no matter what i do. It just sits there stable at 4.26GB / 41.9% unused but apparently occupied memory it won't release. 

using your first line for octave proc1 , i used

a = ones (870,860,860);

in octave proc2 to get it within about 18MB remaining available.  the next tiny step took it over, and it killed process 2.  I'm not sure why it was killing process 1 in the original tests but it's killing 2 now.


what still irks me is that, if it's an OS level issue, how does it make sense that there is a difference between:

doubles:

>> mem = memory; mem.mem_used_octave
ans = 4.5352+08
>> a = ones (1000,1000,1000);
>> mem = memory; mem.mem_used_octave
ans = 8.4535+09
>> clear a
>> mem = memory; mem.mem_used_octave
ans = 4.5352+08


and cells:

>> mem = memory; mem.mem_used_octave
ans = 4.5351+08
>> a = num2cell(ones(12000));
>> mem = memory; mem.mem_used_octave
ans = 8.5177+09
>> clear a
>> mem = memory; mem.mem_used_octave
ans = 7.3657+09


forget multiple octave processes colliding.  why does one immediately release memory, and the other doesn't, and the only diff is one used num2cell to fill that 8GB?  is clear missing something with cell arrays?

Nicholas Jankowski <nrjank>
Group Member
Thu 02 May 2024 11:20:27 PM UTC, comment #19: 

After a lot of testing, I think it really is Linux memory pool behavior, not an Octave bug.

Test: In one Octave process, run this code from earlier in the thread:


tic; A = magic (1e4); toc, B = num2cell (A); toc, clear; whos


It should use about 4.6 GB after the "clear" statement. On my system, that is about 29% of RAM for Octave.

Start a second Octave process and create an array of size roughly (total RAM minus 4.6 GB), so e.g. if you have 16 GB, then create an array of size ~11.5 GB.


parr = perms (uint16 (1:12));  # this is roughly 11.5 GB final size


You can use any statement to do the second command as long as the size is in the range of (total RAM minus RAM used by first Octave process). Note that my perms() example will use some additional memory of its own while building the array, so it uses some swap on my system.

On mine, first the total memory usage went up to 90%, then the first Octave process's memory usage started falling from 29% down to low single digits, while swap usage went up to about 6GB, then when everything was over some 20 seconds later, the second Octave was using about 11.5 to 12 GB, and the first one was using only 35 MB not 4.6 GB like it had been occupying.

That sort of behavior is consistent with what I have seen from the Linux kernel. Sometimes it is slow to relinquish freed memory when the program is still running unless there really is memory pressure from other programs. But you will have to tune the size of the second array in the second process carefully. If you make it too small, there won't be any memory pressure and the apparent memory allocated to the first process won't go down -- this might look like a leak even if it is not. If you make it too big, there won't be enough RAM or swap for both processes, and one or both will crash.

I also ran it through an AddressSanitizer build to be safe. In one run I ran "A = ones (10)" and in the other, I ran "A = num2cell (ones (10))", redirecting ASAN's stderr messages to files for both cases. Comparing the two afterwards in vimdiff, the only differences were the physical memory addresses, but the line-by-line usage numbers and messages were all identical.

Arun Giridhar <arungiridhar>
Group Member
Thu 02 May 2024 07:47:49 PM UTC, comment #18: 

Yes, the memory usage is showing up independently of num2cell and also  no need to add the extra random number.

But this is interesting behavior: normally with a memory leak,  if you run it again you'll double the leak, and so on, leaking the same amount afresh each time it's run. Whatever is going on here is not like that. E.g. with the magic square test with size = 1000, then doing that test 100 times gives the same unexplained memory usage as running it just once. That memory is used in creating the cell array the first time, then presumably never released, but calling the code again only uses that hidden memory not new memory.

I'm using pmr allocation, if it helps anyone diagnose it.

I looked at other classes that inherit from Array with grep "public Array". Only Cell shows the memory behavior.

Arun Giridhar <arungiridhar>
Group Member
Thu 02 May 2024 05:36:48 PM UTC, comment #17: 

Would this script (18s to run) be an example of the bug without using num2cell?


disp("---");
disp("Before anything is done:");
memory;

a = magic(1000);
b = cell(1000, 1000);
for r = 1:1000
  for c = 1:1000
    b{r, c} = a(r, c) + rand;  # Add something to eliminate any chance of lazy copy.
  end
end

disp("---");
disp("Large matrix and large cell are allocated:");
disp(sprintf("Matrix a uses %d kiB.", whos("a").bytes / 1024));
disp(sprintf("Cell b uses %d kiB.", whos("b").bytes / 1024));
memory;

clear a
disp("---");
disp("Only cell b is allocated:");
memory;

clear all
disp("---");
disp("Right after clear all:");
memory


Running the script above produces:

---
Before anything is done:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  65894156 kiB,  swap:   8387580 kiB
Octave    RAM:    119720 kiB,  virt:  21963236 kiB
Free      RAM:  63552084 kiB,  swap:   8387580 kiB
Available RAM:  64455992 kiB, total:  72843572 kiB
---
Large matrix and large cell are allocated:
Matrix a uses 7812.5 kiB.
Cell b uses 7812.5 kiB.
Octave is running on x86_64-pc-linux-gnu
System    RAM:  65894156 kiB,  swap:   8387580 kiB
Octave    RAM:    182076 kiB,  virt:  22036588 kiB
Free      RAM:  63489952 kiB,  swap:   8387580 kiB
Available RAM:  64393860 kiB, total:  72781440 kiB
---
Only cell b is allocated:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  65894156 kiB,  swap:   8387580 kiB
Octave    RAM:    174332 kiB,  virt:  22028772 kiB
Free      RAM:  63493440 kiB,  swap:   8387580 kiB
Available RAM:  64397348 kiB, total:  72784928 kiB
---
Right after clear all:
Octave is running on x86_64-pc-linux-gnu
System    RAM:  65894156 kiB,  swap:   8387580 kiB
Octave    RAM:    174332 kiB,  virt:  22028236 kiB
Free      RAM:  63493440 kiB,  swap:   8387580 kiB
Available RAM:  64397348 kiB, total:  72784928 kiB


Henry Shu <mlguru>
Thu 02 May 2024 04:47:37 PM UTC, comment #16: 

checking with the same test as for doubles (i forgot cells concatenate similarly) to see if it's num2cell or cells in general:


whos

memory
Octave is running on x86_64-pc-linux-gnu
System    RAM:  10138252 kiB,  swap:    945368 kiB
Octave    RAM:     91136 kiB,  virt:    442888 kiB
Free      RAM:   6834940 kiB,  swap:    567084 kiB
Available RAM:   9023272 kiB, total:   9590356 kiB

a = cell(10000,10000);
a(:) = 10;
whos
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name        Size                     Bytes  Class
  ====   ====        ====                     =====  =====
         a       10000x10000              800000000  cell

Total is 100000000 elements using 800000000 bytes

memory
Octave is running on x86_64-pc-linux-gnu
System    RAM:  10138252 kiB,  swap:    945368 kiB
Octave    RAM:    872448 kiB,  virt:   1224136 kiB
Free      RAM:   6041884 kiB,  swap:    566060 kiB
Available RAM:   8222812 kiB, total:   8788872 kiB

a = cat(3,a,a);
a = cat(4,a,a);

whos
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name        Size                     Bytes  Class
  ====   ====        ====                     =====  =====
         a       10000x10000x2x2         3200000000  cell

Total is 400000000 elements using 3200000000 bytes

memory
Octave is running on x86_64-pc-linux-gnu
System    RAM:  10138252 kiB,  swap:    945368 kiB
Octave    RAM:   3217492 kiB,  virt:   3567888 kiB
Free      RAM:   3680788 kiB,  swap:    566060 kiB
Available RAM:   5862028 kiB, total:   6428088 kiB

clear all

whos

memory
Octave is running on x86_64-pc-linux-gnu
System    RAM:  10138252 kiB,  swap:    945368 kiB
Octave    RAM:     92488 kiB,  virt:    442732 kiB
Free      RAM:   6818672 kiB,  swap:    566060 kiB
Available RAM:   8999912 kiB, total:   9565972 kiB


so, it seems you're probably on the right track with num2cell.

Nicholas Jankowski <nrjank>
Group Member
Thu 02 May 2024 03:27:59 PM UTC, comment #15: 

This debugging patch to trace the allocations and deallocations:


diff -r e17b6b7de50f liboctave/array/Array.h
--- a/liboctave/array/Array.h   Wed May 01 17:18:37 2024 -0400
+++ b/liboctave/array/Array.h   Thu May 02 11:23:34 2024 -0400
@@ -33,6 +33,7 @@

 #include <algorithm>
 #include <iosfwd>
+#include <iostream>
 #include <string>

 #include "Array-fwd.h"
@@ -195,6 +196,7 @@ protected:

     pointer allocate (size_t len)
     {
+      std::cout << "Calling allocate() with len = " << len << '\n';
       pointer data = Alloc_traits::allocate (*this, len);
       for (size_t i = 0; i < len; i++)
         T_Alloc_traits::construct (*this, data+i);
@@ -203,6 +205,7 @@ protected:

     void deallocate (pointer data, size_t len)
     {
+      std::cout << "Calling deallocate() with data = " << &(*data) << " len = " << len << '\n';
       for (size_t i = 0; i < len; i++)
         T_Alloc_traits::destroy (*this, data+i);
       Alloc_traits::deallocate (*this, data, len);


followed by a bunch of manual reviewing and cancellations shows that there is exactly one extra "allocate" message without a corresponding "deallocate" (per element) when called with num2cell. I tried it with "A = 1" and then "B = num2cell (A)".

Have to dig deeper to see whether an extra copy is being made or whether a deallocation is being missed.

Arun Giridhar <arungiridhar>
Group Member
Thu 02 May 2024 02:33:46 PM UTC, comment #14: 

the main issue seems unique to linux as i can't reproduce exactly in windows, but there more 'unreleased' memory with cells, don't know if it's related:


octave:1> memory, a = magic (10000);a = cat(3, a,a); a = cat(4,a,a); a = cat(5,a,a);
Octave is running on x86_64-w64-mingw32
System    RAM:  33231568 kiB,  swap:  38212304 kiB
Octave    RAM:     43916 kiB,  virt:     90416 kiB
Available RAM:  19755168 kiB, total:  40190504 kiB
octave:2> memory
Octave is running on x86_64-w64-mingw32
System    RAM:  33231568 kiB,  swap:  38212304 kiB
Octave    RAM:   6294600 kiB,  virt:  12603744 kiB
Available RAM:  13689664 kiB, total:  27866936 kiB
octave:3> clear all
octave:4> memory
Octave is running on x86_64-w64-mingw32
System    RAM:  33231568 kiB,  swap:  38212304 kiB
Octave    RAM:     44032 kiB,  virt:     90536 kiB
Available RAM:  19956704 kiB, total:  40401148 kiB


octave:5> a = num2cell(magic(10000));
octave:6> memory
Octave is running on x86_64-w64-mingw32
System    RAM:  33231568 kiB,  swap:  38212304 kiB
Octave    RAM:   5539280 kiB,  virt:  11383372 kiB
Available RAM:  14338992 kiB, total:  29051556 kiB
octave:7> clear all
octave:8> memory
Octave is running on x86_64-w64-mingw32
System    RAM:  33231568 kiB,  swap:  38212304 kiB
Octave    RAM:     66468 kiB,  virt:    146908 kiB
Available RAM:  19833456 kiB, total:  40312140 kiB


Nicholas Jankowski <nrjank>
Group Member
Thu 02 May 2024 02:20:21 PM UTC, comment #13: 

Sorry, crossed with nrjank.

Comment #10 I can reproduce on mine. The cell array  doesn't seem to get released back but it's specific to cells.

Marking as confirmed.

Arun Giridhar <arungiridhar>
Group Member
Thu 02 May 2024 02:17:32 PM UTC, comment #12: 

comment #6:

> when you say you've tried this building from source, are you building from the default branch?
>


On the 64GB machine, I compiled Octave from the link pointed to by the official Octave website.  The link is https://gnu.mirror.constant.com/octave/octave-9.1.0.tar.gz

There, I configured/compiled Octave by following https://askubuntu.com/questions/891189/octave-4-2-1-and-intel-mkl/1502160#1502160

On the 8GB machine, I installed Octave directly from the repository ppa:ubuntuhandbook1/octave

Henry Shu <mlguru>
Thu 02 May 2024 02:16:38 PM UTC, comment #11: 

Reopening based on comments.

I'm still not able to see the behavior unless the array size is enough to exhaust the memory.

I'll test more.

Arun Giridhar <arungiridhar>
Group Member
Thu 02 May 2024 01:56:50 PM UTC, comment #10: 

a = num2cell (magic(10000))

memory:

Octave RAM: 5507572 kiB, virt: 5877044 kiB

clear all

memory:

Octave RAM: 4686456kiB, virt: 5030256 kiB


something seems different when it's a cell array.

Nicholas Jankowski <nrjank>
Group Member
Thu 02 May 2024 01:51:45 PM UTC, comment #9: 

on the same system, i just ran:

a = magic (10000);
a = cat(3, a, a);
a = cat(4, a, a);
a = cat(5, a, a);

the last one uses a similar ram usage level. i think some swapping based on system response.  `memory` reports

Octave RAM: 6178648 kiB, virt: 6692892 kiB

as soon as i 'clear all', top reports the memory is released and reported usage drops back to

Octave RAM: 45348 kiB, virt: 442736 kiB

and top reports octave-gui usage dropped back to 0.4%


something is different in the comment #0 case, and I think we may be being too hasty to attribute it to the OS memory manager. why would the case above release memory with a clear all but not the OP's case? 

how can we construct a similar simple test with cells or is there something unique to how he built the large cells?


Nicholas Jankowski <nrjank>
Group Member
Thu 02 May 2024 01:40:32 PM UTC, comment #8: 

From the video, it looks like one instance of Octave crashed because it ran out of memory. Each instance was trying to use 6+ GB on an 8GB machine. That's not an Octave bug by itself. I could run that code on a 16GB machine but if I had increased my array size it would have crashed here too.

Octave will request the memory it needs. If there isn't enough from the OS, Octave will follow the OS default behavior such as exiting the whole program unceremoniously. If you want it to only give an "insufficient memory" error and return to the prompt, there are many ways to override the default behavior through your OS mechanisms. One way is with the Linux command prlimit like in this discussion https://octave.discourse.group/t/octave-crashes-when-out-of-memory/4339

Octave will not override OS default behavior unless you specifically set such limits yourself using a technique that suits you. There is no single solution that will work for all users and environments.

Arun Giridhar <arungiridhar>
Group Member
Thu 02 May 2024 01:28:59 PM UTC, comment #7: 

adding:  exiting octave normally, all memory is instantly released and available according to top.

Nicholas Jankowski <nrjank>
Group Member
Thu 02 May 2024 01:27:26 PM UTC, comment #6: 

when you say you've tried this building from source, are you building from the default branch?

I can reproduce this under my ubuntu VM, the `whos` outputs are the same as when i ran on windows.  but I do see the persistent reported usage in top and by running the `memory` command in octave.  (Octave RAM: 7335364 kiB, even after a clear all).  I ran the script twice in the same process, and it ran fine but the top/memory reported ram usage remain unchanged.

I opened a second session (after a clear all in the first but leaving it open), and I also saw similar issues to what @mlguru is reporting.  running the script starts the new octave-gui process increasing mem use without the first process having its memory claim reduced. when the system hit 100% usage (the first process being at ~70%, the new one hitting ~28%) the first process crashed. the second process completed now occupying 70% of memory.   This is a 16GB machine i've only allocated 10GB to the VM, so that was the magic number to trigger the crash.  If I fire up a third terminal the same situation happens (octave-gui process #2 will crash when #3 brings mem usage to ~100%.)

crashing the cli version i see

octave:4> Killed

before being returned to the shell. On the fourth try (with session 3 being --gui) it seemed to have crashed the VM, or maybe it actually started swapping as both client and host were unresponsive.  eventually it crashed the session #3 gui, reporting 'octave exited with signal 9'

Anyway, so it seems in ubuntu (22.04) the os is allocating the memory and even though octave doesn't need it anymore it's not being released to the new process.  A Kill is being issued to the old octave process when new memory is needed.  Is there a way we can be sure Octave isn't at all attributable for the issue by not releasing the memory?

Nicholas Jankowski <nrjank>
Group Member
Thu 02 May 2024 11:47:44 AM UTC, comment #5: 

Octave does seem to misbehave.

On my Lenovo IdeaPad Flex 15 with 8GB RAM, I opened two separate Octave sessions.  I ran the script in one session, followed by running the same script in the other session.  I see two misbehaviors.

Misbehavior 1: The script in the 1st session completed in 8.8 seconds.  It took 15.6 seconds in the 2nd session.

Misbehavior 2: The 1st Octave session died while the script was running in the 2nd session.

A video is attached and also on Youtube: https://youtu.be/GDgCGlB0qxU

This machine is Ubuntu 22.04.4 x86_64, with kernel 6.5.0-28-generic.

Please let me know if I can help reproduce these issues.  Thanks.

(file #55998)

Henry Shu <mlguru>
Thu 02 May 2024 01:59:53 AM UTC, comment #4: 

From what I could see in the video, the memory usage inside Octave was low (last line of whos gives the bytes used), so I'm inclined to close this as invalid / not an Octave bug.

Most probably, because your system has 64 GB, Linux is slow to give up cached RAM because it figures the cache might as well hang around unless something else needs to really use that memory. If you're bothered by it, look up swappiness, but I've never needed to touch that kernel parameter, and if you're new to OS settings you shouldn't touch it for a while, at least not without reading up on it.

Even if it looks like the memory isn't being returned, if you need to use another program while Octave is running, Linux will allocate that memory to the other program on demand.

I'll close this as not an Octave bug / normal Linux behavior. If Octave misbehaves comment here and we can reopen it.

Arun Giridhar <arungiridhar>
Group Member
Wed 01 May 2024 11:39:58 PM UTC, comment #3: 

I'm a novice for OS-related stuff.  I made a video showing my `top` (ordered by memory with Ctrl-m) and Octave session simultaneously.  I also did the vm.drop_caches command in the end.  The video is attached with this bug, and also on Youtube: https://youtu.be/IIiy9oiRXUg

From the video, could you let me know if my apparent memory consumption is just cache or actual usage?  Please let me know if you need anything else from me.  Thank you.

My machine has 64G RAM.  So 8.6% (as reported by `top`) of it is about 5.5GB.


(file #55996)

Henry Shu <mlguru>
Wed 01 May 2024 10:57:46 PM UTC, comment #2: 

Works for me on Linux (not Ubuntu though).

I modified the test code to add some whos and printf statements:

gatherFun = @(n) zeros(n, 160000, "single");
numPts = randi([3 11], [1600 1]);
numRuns = 2;
for curRun = 1:numRuns
  printf ("Starting iteration# %d\n", curRun)

  myData = cellfun(gatherFun, num2cell(numPts), "UniformOutput", false);

  # Manual release of memory
  for chunk = 1:numel(myData)
    myData{chunk} = 0;
  endfor

  whos
  printf ("Ending iteration# %d\n", curRun)
end

whos
clear all


This is what I get with both Octave 9 and Octave 10 on Linux:

Starting iteration# 1
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name           Size                     Bytes  Class
  ====   ====           ====                     =====  =====
         chunk          1x1                          8  double
         curRun         1x1                          8  double
         gatherFun      1x1                          0  function_handle
         myData      1600x1                      12800  cell
         numPts      1600x1                      12800  double
         numRuns        1x1                          8  double

Total is 3204 elements using 25624 bytes

Ending iteration# 1
Starting iteration# 2
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name           Size                     Bytes  Class
  ====   ====           ====                     =====  =====
         chunk          1x1                          8  double
         curRun         1x1                          8  double
         gatherFun      1x1                          0  function_handle
         myData      1600x1                      12800  cell
         numPts      1600x1                      12800  double
         numRuns        1x1                          8  double

Total is 3204 elements using 25624 bytes

Ending iteration# 2
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name           Size                     Bytes  Class
  ====   ====           ====                     =====  =====
         chunk          1x1                          8  double
         curRun         1x1                          8  double
         gatherFun      1x1                          0  function_handle
         myData      1600x1                      12800  cell
         numPts      1600x1                      12800  double
         numRuns        1x1                          8  double

Total is 3204 elements using 25624 bytes

octave:8> whos
octave:9>


Can you confirm that the memory you see in `top` is actually used by Octave and not just cached by the OS? You can try changing the system's swappiness. Or before that you could try something like "sudo sync ; sudo sysctl -w vm.drop_caches=3" if you understand what those commands do.

Arun Giridhar <arungiridhar>
Group Member
Wed 01 May 2024 10:23:50 PM UTC, comment #1: 

unable to recreate on windows 10 in octave 9.1.0, so may be unique to linux:

variables in scope after the "myData = cellfun(..." line

variables in scope: top scope

  Attr   Name           Size                     Bytes  Class
  ====   ====           ====                     =====  =====
         ans            1x2                         16  double
         chunk          1x1                          8  double
         curRun         1x1                          8  double
         gatherFun      1x1                          0  function_handle
         myData      1600x1                 7139200000  cell
         numPts      1600x1                      12800  double
         numRuns        1x1                          8  double

Total is 3206 elements using 7139212840 bytes


variables in scope right before the clear all:



variables in scope: top scope

  Attr   Name           Size                     Bytes  Class
  ====   ====           ====                     =====  =====
         ans            1x1                          8  double
         chunk          1x1                          8  double
         curRun         1x1                          8  double
         gatherFun      1x1                          0  function_handle
         myData      1600x1                      12800  cell
         numPts      1600x1                      12800  double
         numRuns        1x1                          8  double

Total is 3205 elements using 25632 bytes


and Task managr confirms that memory usage.

Nicholas Jankowski <nrjank>
Group Member
Wed 01 May 2024 09:36:50 PM UTC, original submission:  

The following code script will create about 8GB of data (all zeros) held by the cell myData.  This consumption of 8GB in memory persists even after calling clear all.  (One can use, for example, top to verify this.)  There seems to be no way to clear this memory other than exiting the Octave session.

Somehow, if numRuns = 1, the consumption of 8GB in memory would be cleared in the end.  But no such luck with numRuns = 2.

This would be a major hindrance for crunching big data.  I would appreciate a fix or workaround without the need to invoke a separate Octave session for every call of the script below.  Thank you!


gatherFun = @(n) zeros(n, 160000, "single");
numPts = randi([3 11], [1600 1]);
numRuns = 2;
for curRun = 1:numRuns
  myData = cellfun(gatherFun, num2cell(numPts), "UniformOutput", false);

  # Manual release of memory
  for chunk = 1:numel(myData)
    myData{chunk} = 0;
  endfor
end

clear all




My machine: Ubuntu 22.04 x86_64, kernel 6.5.0-27-generic

I used the following to install Octave.  But this bug is observed even when I compiled Octave from source.


sudo add-apt-repository -y ppa:ubuntuhandbook1/octave
sudo apt update
sudo apt -y install octave octave-dev


Henry Shu <mlguru>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by petter (Posted a comment)
  • -email is unavailable- added by arungiridhar (Potential test for object pools)
  • -email is unavailable- added by arungiridhar (Posted a comment)
  • -email is unavailable- added by nrjank (Posted a comment)
  • -email is unavailable- added by mlguru (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-05-03 arungiridhar SummaryMemory Leak when Using Cell Cell array construction is memory-intensive
        Carbon-Copy- Added petter
    2024-05-02 arungiridhar StatusNone Confirmed
    2024-05-02 arungiridhar StatusInvalid / Not an Octave Bug None
        Open/ClosedClosed Open
    2024-05-02 mlguru Attached File- Added possible_memory_leak_video2.mp4, #55998
    2024-05-02 arungiridhar StatusNone Invalid / Not an Octave Bug
        Open/ClosedOpen Closed
    2024-05-01 mlguru Attached File- Added possible_memory_leak_video1.mp4, #55996

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code