patchGNU Octave - Patches: patch #9924, Suggestion for a memory() function


patch #9924: Suggestion for a memory() function

Submitted by:  Lars Kindermann <larskindermann>
Submitted on:  Mon 20 Apr 2020 03:20:31 PM UTC  
Category:  Core : new function Priority:  5 - Normal
Status:  Done Privacy:  Public
Assigned to:  None Open/Closed:  Closed

Add a New Comment (Rich Markup)

You are not logged in

Please log in, so followups can be emailed to you.


( Jump to the original submission)

Sun 07 Jun 2020 05:27:25 PM UTC, comment #32: 

I cross-built for Windows 64bit and 32bit. The results look reasonable in both cases.

Closing as done.

Markus Mützel <mmuetzel>
Project Member
Thu 28 May 2020 07:03:25 PM UTC, comment #31: 

Many thanks to you all for this new function! Looks great.

Pantxo Diribarne <pantxo>
Project Member
Wed 27 May 2020 06:51:35 PM UTC, comment #30: 

I pushed the patches here:

If some more work should be needed (e.g. documentation or getrusage on Linux) or someone would like to implement that function on Mac, this could be done here or in new bug reports.

Marking as ready for test.

Markus Mützel <mmuetzel>
Project Member
Sun 24 May 2020 07:22:36 PM UTC, comment #29: 

The attached patch includes the better fallback value for available RAM.

(file #49162)

Markus Mützel <mmuetzel>
Project Member
Sun 24 May 2020 04:32:22 PM UTC, comment #28: 

[comment #27:]

> I don't have a "FreeRAM" value in meminfo.
> Should that be "MemFree"? Or is that a value that existed on that older kernels?

You're correct, it should read "MemFree" indeed!

Lars Kindermann <larskindermann>
Sun 24 May 2020 10:42:56 AM UTC, comment #27: 

@Lars: Thanks for that code snippet.
I don't have a "FreeRAM" value in meminfo. Should that be "MemFree"? Or is that a value that existed on that older kernels?

Markus Mützel <mmuetzel>
Project Member
Sun 24 May 2020 10:28:55 AM UTC, comment #26: 

@markus, re comment #22

>> I also added a fallback to MemFree if MemAvailable doesn't exist in
>> proc/meminfo. Is this ok? Or is there something better we can do?

This is from the kernel patch comment that indrocuded the MemAvailable field:

"Many load balancing and workload placing programs check /proc/meminfo to
estimate how much free memory is available.  They generally do this by
adding up "free" and "cached", which was fine ten years ago, but is
pretty much guaranteed to be wrong today."

So I added the "ten year old way" to memory.m as a fallback:

  if (isfield(meminfo_numeric,"MemAvailable"))
    AvailableRAM = meminfo_numeric.MemAvailable * kiB;
    # on kernels from before 2014 MemAvailable is not present
    # this is a rough estimate to be used instead
    AvailableRAM = (meminfo_numeric.FreeRAM + meminfo_numeric.Cached) * kiB;

Lars Kindermann <larskindermann>
Sun 24 May 2020 10:02:47 AM UTC, comment #25: 

I moved reading the pseudo files on Linux to a local function and removed some variables that are no longer used in patch9924_memory_v2.patch. It still applies on top of patch9924_wmemory.patch.

Additionally, I added some documentation in patch9924_memory_doc.patch. I am not sure I used the correct terminology and what I wrote makes at least have sense. A review and corrections would be very welcome.
I hope to also have covered the potential issue Lars mentioned in comment #23 correctly.

There is no "Ready to Review" status on the patch tracker. So I'll leave the status on "In Progress". Although I think most of the work should be finished that is necessary for an initial push to the repository.

(file #49159, file #49160)

Markus Mützel <mmuetzel>
Project Member
Sat 23 May 2020 08:39:21 PM UTC, comment #24: 

I'm surprised an OS as rock-solid as Linux can get this far in trouble in the first place, due to just user actions ...

Philip Nienhuis <philipnienhuis>
Project Member
Sat 23 May 2020 08:05:41 PM UTC, comment #23: 

I continued to work on the linux code and while testing I realized there is a severe problem in providing the matlab-compatible function.

After allocating all of the memory the MemAvailableAllArrays or MaxPossibleArrayBytes fields suggest, the system will be most probably dead.

After executing

[usr, sys] = memory;

my system went (as expectd) into heavy swapping, but because really all memory was used up, it became totally inaccessible. All attemps to kill octave failed. And because the memory usage was still legit in theory, the OOM killer did not step in. After two hours I forcefully powered down the system under heavy disk load - with the result that the partition table was destroyed and I lost my data.

We should at least place some warning into the documentation about this...

Lars Kindermann <larskindermann>
Thu 21 May 2020 06:43:36 PM UTC, comment #22: 

I went ahead and removed the linux tag, adapted the code to the coding standards, added calling the _wmemory_ function on Windows and created a patch with Lars as the author.
It applies on top of the wmemory patch.

I also added a fallback to MemFree if MemAvailable doesn't exist in
proc/meminfo. Is this ok? Or is there something better we can do?

We should probably flesh out the documentation a little bit more, i.e. explain the meaning of the fields in the returned structures.

Querying the actual size of the user address space on Linux as well as support on Mac would be nice. But that could also be added at a later point imho.

(file #49140)

Markus Mützel <mmuetzel>
Project Member
Thu 21 May 2020 02:32:13 PM UTC, comment #21: 

Oh, sorry, never meant to use psutil in Octave. My interest was in obtaining robust values for the amount of total/available memory so was curious to see what they were relying on:

Guillaume <gyom>
Thu 21 May 2020 01:57:44 PM UTC, comment #20: 

@Guillaume: AFAICT, psutil is written in Python. I'm not sure if we can use this in Octave.

I came across "getrlimit" [1]. I think we could query the process' address space size on posix compatible systems (Linux and Mac) with "RLIMIT_AS".


Markus Mützel <mmuetzel>
Project Member
Thu 21 May 2020 09:06:07 AM UTC, comment #19: 

Was looking at implementing memory() unaware of this patch so very pleased to come across it.
Of interest for a cross platform availability, I found this recent discussion in Numba:
that references psutil:

Guillaume <gyom>
Sun 03 May 2020 10:18:03 AM UTC, comment #18: 

The code should not fail on systems with older kernels. Would it be possible to handle that gracefully?

Markus Mützel <mmuetzel>
Project Member
Sat 02 May 2020 03:25:33 PM UTC, comment #17: 

I just realized that the /proc/meminfo/memavailable field was introduced to the kernel just in 2014:

Thus on older kernels the memory function will fail. Do you think this should be checked in the code?

Lars Kindermann <larskindermann>
Sat 02 May 2020 03:10:57 PM UTC, comment #16: 

@Lars: That sounds reasonable.
Maybe we could use underscores "_" in the field name to make it even clearer that this is an Octave extension. AFAIK, jwe is no big fan of CamelCase.
So what about "RAM_available_all_arrays"?

The 3 GiB for the total user address space on 32bit systems seems to be hard coded. If I correctly understood, that might be different for kernels compiled with non-default settings. Is there an interface to query the actual value on Linux?
For Windows, this is probably the value of sys.TotalVirtual.

Markus Mützel <mmuetzel>
Project Member
Sat 02 May 2020 02:52:30 PM UTC, comment #15: 

32Bit systems with PAE can use up to 64GB of RAM, but every process is limited to a virtual address space of 4GB, 1GB is typically assigned to the kernel, 3GB to the userspace. This user/kernel split is defined when compiling the kernel, with the 3GB/1GB being the default:

So the RAM that is available to each octave process is limited by the total RAM available on the system or the virtual address space left for the process, whatever is smaller.

That's why I think the RAM available to the process is more important than the free RAM on the system. I added that computation to a new version of memory.m, so user.RamAvailableAllArrays and syst.PhysicalMemory.Available are different now.

Maybe to avoid confusion, the field name could contain the term Octave, e.g. RamAvailableOctave istead of RamAvailableAllArrays?

(file #48984)

Lars Kindermann <larskindermann>
Sat 02 May 2020 12:11:04 PM UTC, comment #14: 

@Markus: fair enough.

This will typically be  the case for 32 Bit systems with more than 4GB of RAM

AFAIK 32-bit systems can't make use of any RAM > 4 GB (maybe unless PAE extensions are used).

Philip Nienhuis <philipnienhuis>
Project Member
Fri 01 May 2020 05:33:32 PM UTC, comment #13: 

From the code of memory.m, I was under the impression that user.RamAvailableAllArrays and syst.PhysicalMemory.Available would always have the same value.
Do I correctly understand that the value for user.RamAvailableAllArrays will have to be determined differently in the future?
Maybe we could add that field (possibly with a more Octave-y name) once it is clear how we can get the corresponding information.

Markus Mützel <mmuetzel>
Project Member
Fri 01 May 2020 04:27:59 PM UTC, comment #12: 

As I don't have any windows system at hand, I could not add the windows stuff so far.

About compatibility: I would strongly recommend to keep at least the "RamAvailableAllArrays" field, maybe renaming it.

This is basically what I was writing the function for and most people probably want to know about memory: How much RAM you can use right now. Matlabs MemAvailable... fields are useless in most cases as putting your data into swap will slow down anything incredibly.

This value cannot always be derived from other fields in the system struct, as suggested. The Ram available to a process can be smaller than the free Ram on the system. This will typically be  the case for 32 Bit systems with more than 4GB of RAM (I implemented this in the RAMAvailabe... field) and when the process is restricted by the kernel in its memory use, e.g. using cgroups. (not yet implemented, there is no corresponding field in the /proc/pid/status pseudofile)

Lars Kindermann <larskindermann>
Fri 01 May 2020 01:47:07 PM UTC, comment #11: 

What still needs to be done as far as I can see, is (not in any particular order):

  • decide on the non-Matlab extensions in Lars' function (additional fields and "linux" flag),
  • add a branch for Windows platforms to Lars' function (effectively a "translation" from _wmemory_ to the "memory" output structures),
  • add code for Mac platforms (or does the Linux approach also work on Macs?),
  • and make a patch for the default branch out of all this.

Did I miss something?

Some Octave extensions over Matlab bit us in the back in the past. I believe it is best to keep it simple and close to Matlab's behavior unless there is a big advantage in deviating.
So I'd vote for removing the additional fields and the "linux" flag from Lars' function and for not implementing the RAM usage check for other processes.
If that is needed, we could use dedicated, separate functions for this. But these features should be working cross-platform if we do so.

Markus Mützel <mmuetzel>
Project Member
Thu 30 Apr 2020 06:24:39 PM UTC, comment #10: 

Patch from comment #6 works fine on Windows 7.

I was wondering, the wmemory.m I submitted could also report RAM usage of other programs running. A scenario where that could be handy is e.g., distributed computing using master/slaves constructs.
Could that also be added? (just a wish)

Other than that, what is needed to proceed?

Philip Nienhuis <philipnienhuis>
Project Member
Sat 25 Apr 2020 11:57:12 AM UTC, comment #9: 

Ah I overlooked that, I think it does. I hope to be able to try the patch soon.

Philip Nienhuis <philipnienhuis>
Project Member
Sat 25 Apr 2020 11:41:49 AM UTC, comment #8: 

Philip, I believe that the information you ask for is contained in the proc structure (in particular proc.WorkingSetSize).

Please, let me know if you need something different.

Markus Mützel <mmuetzel>
Project Member
Sat 25 Apr 2020 11:30:59 AM UTC, comment #7: 

Markus, could info about RAM usage of individual processes (at least the calling invocation of Octave itself) be included as well, like in my wmemory patch of patch #9746 ?

Philip Nienhuis <philipnienhuis>
Project Member
Wed 22 Apr 2020 02:08:27 PM UTC, comment #6: 

The attached patch adds the function to Octave core as described in comment #5.
I also removed the type prefixes from the field names of the sys structure. So the output here is now:

>> [proc, sys] = __wmemory__ ()

proc =

  scalar structure containing the fields:

    PageFaultCount = 88269
    PeakWorkingSetSize = 1.8433e+08
    WorkingSetSize = 1.5598e+08
    QuotaPeakPagedPoolUsage = 960136
    QuotaPagedPoolUsage = 960088
    QuotaPeakNonPagedPoolUsage = 60424
    QuotaNonPagedPoolUsage = 59752
    PagefileUsage = 1.3471e+08
    PeakPagefileUsage = 1.7294e+08

sys =

  scalar structure containing the fields:

    MemoryLoad = 52
    TotalPhys = 3.4225e+10
    AvailPhys = 1.6182e+10
    TotalPageFile = 3.6372e+10
    AvailPageFile = 1.2927e+10
    TotalVirtual = 1.4074e+14
    AvailVirtual = 1.4073e+14
    AvailExtendedVirtual = 0

(file #48900)

Markus Mützel <mmuetzel>
Project Member
Wed 22 Apr 2020 12:57:14 PM UTC, comment #5: 

The attached file can be compiled with

mkoctfile -lpsapi

On the system here, I get these results when I call it:

>> [proc, sys] = __wmemory__ ()

proc =

  scalar structure containing the fields:

    PageFaultCount = 584161
    PeakWorkingSetSize = 1.8735e+09
    WorkingSetSize = 1.5450e+08
    QuotaPeakPagedPoolUsage = 954832
    QuotaPagedPoolUsage = 915912
    QuotaPeakNonPagedPoolUsage = 60064
    QuotaNonPagedPoolUsage = 54904
    PagefileUsage = 1.4248e+08
    PeakPagefileUsage = 2.3375e+09

sys =

  scalar structure containing the fields:

    dwMemoryLoad = 38
    ullTotalPhys = 3.4225e+10
    ullAvailPhys = 2.0961e+10
    ullTotalPageFile = 3.6372e+10
    ullAvailPageFile = 1.2759e+10
    ullTotalVirtual = 1.4074e+14
    ullAvailVirtual = 1.4073e+14
    ullAvailExtendedVirtual = 0

This might be enough to get all the information needed to fill the return values of "memory".
Apart from MaxPossibleArrayBytes on 32bit systems. But I don't know if we'd need that for a first version...

I don't think we should include that as an .oct file in Octave. It would probably be better as a DEFUN in with fallback code for non-Windows systems.
But I think the interface (i.e. the two returned structs) is good enough. So if you want, you could start incorporating this in memory.m.

(file #48899)

Markus Mützel <mmuetzel>
Project Member
Tue 21 Apr 2020 06:12:46 PM UTC, comment #4: 

When the memory function was introduced in Matlab, systems were 32Bit at most. On 32Bit systems the virtual address space of every process can be 4GB at most, on linux typically 3GB are made available to the user, 1GB is reserved for the kernel. So the availabe memory (MemAvailableAllArrays) = min(availabe virtual memory, availabe system memory). And MaxPossibleArrayBytes is the largest contiguous memory block, which tends to become smaller by fragmentation. So the Ram available to a process can be smaller than the physical memory availabe on the system.

I'm not aware of an api call to get the lagest contiguous memory block, so I don't compute MaxPossibleArrayBytes on 32Bit. You could try to malloc increasingly smaller blocks until it succeeds...

On 64Bit systems it starts pretty easy. You have a virtual address space of 2^63Bytes (smaller on actual hardware). All available ram and swap can be mapped into this. Fragmentation is avoided by mapping a new array to a new location. So MaxPossibleArrayBytes, MemAvailableAllArrays, SystemMemory.Available are typically about the same. And RamAvailableAllArrays can be the same as PhysicalMemoryAvailable.
But the kernel can impose restrictions of ram /swap usage for every process. So not all the available Ram might be availabe to octave. But that's not implemented yet.

Lars Kindermann <larskindermann>
Tue 21 Apr 2020 04:27:53 PM UTC, comment #3: 

For Windows, it's probably possible to query the memory usage of Octave with "GetProcessMemoryInfo" [1] for the handle returned by "GetCurrentProcess" [2]. Information about the system memory could probably be retrieved with "GlobalMemoryStatusEx" [3].
Maybe I could provide the code for a preliminary .oct file that returns that data if that is of interest for you.

"memory" already has a second output argument including information about the total and physical memory. So imho, there is no need to add a field "RamAvailableAllArrays" in the first output argument. But maybe I'm missing the point...

I was not sure what the difference was between "MaxPossibleArrayBytes" and "MemAvailableAllArrays". It seems to be the same when I call "memory" on Windows 10 64bit here. Maybe the answer is here: [4].
Also SystemMemory.Available seems to be lower than PhysicalMemory.Available here (32 GiB RAM). Any idea what that means?
Virtual memory (2-8 GiB dynamic, currently 2 GiB) seems to be not included in the results. But that might be a local issue.

>> [a, b] = memory ()

a =

  struct with fields:

    MaxPossibleArrayBytes: 2.0897e+10
    MemAvailableAllArrays: 2.0897e+10
            MemUsedMATLAB: 1.6874e+09

b =

  struct with fields:

    VirtualAddressSpace: [1×1 struct]
           SystemMemory: [1×1 struct]
         PhysicalMemory: [1×1 struct]

>> b.VirtualAddressSpace

ans =

  struct with fields:

    Available: 1.4072e+14
        Total: 1.4074e+14

>> b.SystemMemory

ans =

  struct with fields:

    Available: 2.0897e+10

>> b.PhysicalMemory

ans =

  struct with fields:

    Available: 2.2551e+10
        Total: 3.4225e+10

The Matlab documentation contains this [5]:

> MATLAB computes the value for Memory Used By MATLAB by walking the MATLAB process memory structures and summing all the sections that have physical storage allocated in memory or in the paging file on disk.

Not exactly sure what that means and if we should care about this. Querying that info from the OS is probably good enough...


Markus Mützel <mmuetzel>
Project Member
Tue 21 Apr 2020 08:54:22 AM UTC, comment #2: 

I already tried to make the memory.m as compatible as possible. It returns all the structs and fields matlab provides, with some additions. The way to gather the informnation is indeed similar to the memory.m from the mcgyver_utils.

Your wmemory.m patch could easily be included in my version, too. It should just fill the fields of the structs declared there.

As I don't have a Windows system at hand at the moment, I can try that at some time in the near future.

Lars Kindermann <larskindermann>
Mon 20 Apr 2020 07:54:43 PM UTC, comment #1: 

This looks like a sort of duplicate, or extension, of patch #9746. As I wrote there it's not too hard to extend the wmemory.m there to also report various RAM sizes using Windows' "systeminfo" command:

>> outval = system ("systeminfo");
Total Physical Memory:     7 887 MB
Available Physical Memory: 5 126 MB
Virtual Memory: Max Size:  15 772 MB
Virtual Memory: Available: 12 719 MB
Virtual Memory: In Use:    3 053 MB

(note: echoed to screen, NOT returned in outval)

.. and obviously Windows has built-in binary library functions that give that info right away, rather than complete with the entire OS hotfix history that "systeminfo" helpfully echos as well. After all, "systeminfo" gets its info from /somewhere/.

As outlined in the relevant maintainers ML thread where you first posted today, at
there's a memory.m (GPL AFAICS) which seems to work on MacOSX, Linux and Windows. In that memory.m my patch turns out to be absorbed.

What do you think, could that file be combined with yours or vice versa, and the trick I used above be added somehow?
That way we can have one comprehensive function for exploring RAM usage. Making it Matlab-compatible can be done in a next step by calling it by Matlab-compatible wrapper functions.

Philip Nienhuis <philipnienhuis>
Project Member
Mon 20 Apr 2020 03:20:31 PM UTC, original submission:  

Some time ago I wrote a mostly matlab compatible memory() function for two main reasons:

- I often work with datasets far bigger than my pc memory and I have to split them into chunks which just fit into ram.

- I wanted to track down a memory leak that occured in preparing 5.0.

So far I did not think about submitting it to the octave kernel because it works on linux only. But I just realized that matlabs memory() is available on windows only. So I think a linux only memory() for octave
is reasonable.

The function is octave only code, a single m-file which reads the linux /proc/pid/statm, /proc/pid/status and /proc/meminfo pseudo files to retrieve exactly the same information you get with the "top" command.

I use it frequently for more than a year on 64Bit amd64 systems with kernels 3.2 - 5.5 and have tested a little bit on i386 32Bit systems, too (Debian & Ubuntu).

But I am not sure if it will work in every situation on 32Bit systems as memory management and limitations are much more complicated there. Would it be better to provide it for 64Bit only?

Should a warning or an error be thrown on other architectures?

And there are some compatibility issues with the matlab function.

E.g. naming of fields: Should "MemUsedMATLAB" become "MemUsedOctave" or should both names be provided?

Matlab's MemAvailableAllArrays and MaxPossibleArrayBytes return the available memory amount including swap. Which renders the system almost unusable when fully allocated.

So I introduced few new, octave-only fields, e.g. "RamAvailableAllArrays" which returns the available ram, which is what you really want to know for efficient work. Is this ok in regard to usability / compatibility?

A very first version of memory.m is attached.

Any comments?

Lars Kindermann <larskindermann>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

Attached Files
file #49162:  patch9924_memory_v3.patch added by mmuetzel (9KiB - application/octet-stream)
file #49159:  patch9924_memory_v2.patch added by mmuetzel (9KiB - application/octet-stream)
file #49160:  patch9924_memory_doc.patch added by mmuetzel (3KiB - application/octet-stream)
file #49140:  patch9924_memory.patch added by mmuetzel (10KiB - application/octet-stream)
file #48984:  memory.m added by larskindermann (8KiB - text/x-objcsrc)
file #48900:  patch9924_wmemory.patch added by mmuetzel (4KiB - application/octet-stream)
file #48899: added by mmuetzel (3KiB - application/octet-stream)
file #48881:  memory.m added by larskindermann (7KiB - text/x-objcsrc)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by pantxo (Posted a comment)
  • -email is unavailable- added by gyom (Posted a comment)
  • -email is unavailable- added by philipnienhuis
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by larskindermann (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 13 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-06-07 mmuetzel StatusReady For Test => Done
        Open/ClosedOpen => Closed
    2020-05-27 mmuetzel StatusIn Progress => Ready For Test
    2020-05-24 mmuetzel Attached File- => Added patch9924_memory_v3.patch, #49162
    2020-05-24 mmuetzel Attached File- => Added patch9924_memory_v2.patch, #49159
        Attached File- => Added patch9924_memory_doc.patch, #49160
    2020-05-21 mmuetzel Attached File- => Added patch9924_memory.patch, #49140
        StatusNone => In Progress
    2020-05-05 philipnienhuis Carbon-Copy- => Added philipnienhuis
    2020-05-02 larskindermann Attached File- => Added memory.m, #48984
    2020-04-22 mmuetzel Attached File- => Added patch9924_wmemory.patch, #48900
    2020-04-22 mmuetzel Attached File- => Added, #48899
    2020-04-20 larskindermann Attached File- => Added memory.m, #48881

    Back to the top

    Powered by Savane 3.6