bugGNU Octave - Bugs: bug #44380, Facing memory limit of less than 1...

 
 

bug #44380: Facing memory limit of less than 1 GB

Submitter:  Puronjoy B <puronjoy>
Submitted:  Fri 27 Feb 2015 10:08:15 AM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Performance
Status:  Invalid / Not an Octave Bug Assigned to:  None
Originator Name:  Puronjoy Open/Closed:  * Closed
Release:  * 3.6.4 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 04 Jan 2016 07:21:02 PM UTC, comment #21: 

I don't know what Windows does behind the scenes for managing virtual memory.  In any case, I don't think this is an Octave bug.  All Octave does is request memory from C++ operator new and occasionally the C library malloc function.  Those are outside the scope of the Octave project.  Given that the limits appear to fluctuate, it seems to me that the limits have something to do with other processes that might be running on your system, or whatever is happening with the operating system's memory management.

John W. Eaton <jwe>
Group administrator
Mon 04 Jan 2016 02:54:16 PM UTC, comment #20: 

@Andreas, I'm running on windows and as I understand it, with memory virtualisation, all applications see a continuous memory space even if it is physically fragmented. So I don't see how that is the problem.

Tom <tomd1982>
Mon 04 Jan 2016 11:07:07 AM UTC, comment #19: 

@Tom: Your operating system isn't able to find a continuous memory block with the requested size which can be addressed by an 32bit pointer.

The size of the biggest continuous memory block depends also on memory fragmentation an thus which processes on you system are started, how many memory they have allocated and freed.

So to conclude: Octave asks the operating system to allocate a continuous memory block but the operating system isn't able to fulfil this. In my eyes clearly not an Octave bug

Andreas Weber <andy1978>
Group Member
Mon 04 Jan 2016 09:43:52 AM UTC, comment #18: 

I think that even though I may have misunderstood the documentation, surely this is still a valid bug?. A 32 bit install limited on some sessions to 0.6GB memory, the amount of allocatable memory varying from user to user and session to session? Do those not constitute bugs?

Tom <tomd1982>
Thu 17 Dec 2015 05:33:47 PM UTC, comment #17: 

Although you may be running Octave on a 64-bit system, the Windows version distributed from gnu.org is a 32-bit application, with 32-bit pointers.  So the part of the documentation you are quoting doesn't apply.  I suppose this could be a bit clearer.  In the future, we will probably be distributing a 64-bit applicatino with 32-bit integers for indexing as is the norm on Linux systems now.  You are, of course, free to build your own 64-bit build.  If you'd like to see this happen faster, then contact me privately for commercial support options.

John W. Eaton <jwe>
Group administrator
Thu 17 Dec 2015 02:42:37 PM UTC, comment #16: 

I think the documentation needs to be ammended to reflect this bug if it isn't going to be fixed, from the page linked in this discussion

"
On 64-bit systems, Octave is limited to (approximately) the following array sizes when using the default 32-bit indexing mode:

double:         16GB
single:          8GB
uint64, int64:  16GB
uint32, int32:   8GB
uint16, int16:   4GB
uint8, int8:     2GB
"

However the windows installer for octave 4 does not produce an install that can do this. Also, fyi, 0.6GB is not the limit, it varies from session to session, sometimes I can reach 1.3GB, other times only 0.8GB.

Tom <tomd1982>
Thu 17 Dec 2015 09:39:07 AM UTC, comment #15: 

I don't understand how this is closed as invalid. It's obviously a bug. In the octave documentation it says

"
On 64-bit systems, Octave is limited to (approximately) the following array sizes when using the default 32-bit indexing mode:

double:         16GB
single:          8GB
uint64, int64:  16GB
uint32, int32:   8GB
uint16, int16:   4GB
uint8, int8:     2GB
"

The idea that it is solved because there are build options to circumvent memory limits is kind of ridiculous because this isn't about the memory limits of octave, this is about a bug, I shouldn't have to compile as 64bit just to allocate a 1.5GB double array, as I should be able to do ten times that on the 32 bit install.

Tom <tomd1982>
Sun 08 Mar 2015 11:26:41 AM UTC, comment #14: 

Closing with "invalid" as there are build options to circumvent memory limits.
FYI Matlab 32 bit has the same limits, ML 64 bit is needed to overcome them

Philip Nienhuis <philipnienhuis>
Group Member
Tue 03 Mar 2015 02:50:53 AM UTC, comment #13: 

The limits on data use for 64-bit systems are not as simple as just less than 2^32 bytes.  More memory than that can be allocated, and you can index approximately 2^31 elements (the index value is signed).  For double complex, that's approximately 16*2^31, or about 32GB.  See more detail here:

http://www.gnu.org/software/octave/doc/interpreter/Compiling-Octave-with-64_002dbit-Indexing.html#Compiling-Octave-with-64_002dbit-Indexing

John W. Eaton <jwe>
Group administrator
Tue 03 Mar 2015 01:42:51 AM UTC, comment #12: 

I'd say give the bug submitter a day or two to confirm and then close the report as "invalid" later this week.

For your 64-bit build without 64-bit indexing, how can it access more than 2^32 = 4 GB, assuming 32-bit indexing?

Dan Sebald <sebald>
Mon 02 Mar 2015 07:50:24 PM UTC, comment #11: 

Turns out I already have (more or less by accident) a 64-bit Octave-3.9.1+ without 64-bit indexing - it seems I simply forgot that I configured MXE with -enable-windows-64 rather than --enable-64 (which implies --enable-windows-64). OK I had a week skiing holiday in between and had built it on my other box, those may be some reasons I forgot it.

Anyway that version can easily accommodate a 30000x30000 double array (7.2 GB) as well on my Win64 8 GB RAM box.

With an older "pure" 32-bit 3.9.0+ it turns out that rand(8650) (598580000 bytes) is near the max size it can allocate.

The conclusion is that to be able to use RAM > 0.6 GB on 64-bit machines running 64-bit Windows, an Octave is needed that has been configured & built with --enable-windows-64 (and runs only on 64-bit Windows).
For "pure" 32-bit Octave builds, the max array size is in the order of 450 to 600 MB.

As to how to further process this bug report, I tend to close it with either:
- "fixed" (as we have --enable-windows-64 build options) or
- "invalid" (as there are Octave options to circumvent the memory limits) or
- "won't fix" (as there's no way out for pure 32-bit builds, like 3.6.4).
What do you think, Dan?

Philip Nienhuis <philipnienhuis>
Group Member
Mon 02 Mar 2015 07:06:52 PM UTC, comment #10: 

@comment #7:
what is the result of the Octave command:
octave_config_info ("USE_64_BIT_IDX_T")
?

Philip Nienhuis <philipnienhuis>
Group Member
Mon 02 Mar 2015 06:25:10 PM UTC, comment #9: 

@comment #8: AFAIK Octave doubles occupy 8 bytes.
And yes, for 32-bit programs Windows memory fragmentation an be very bad, see my comment #2.

@comment #7:
What exactly do you mean with "64-bit Octave 3.8.2" ?
Where did you get your Octave version?

We have three Octave-"bit width" versions on Windows:
- plain 32-bit, runs on 32-bit Windows
- 32-bit indexing but only running on 64-bit Windows
- 64-bit indexing,   ---^^ditto^^---

I think you're running the upper 32-bit version, isn't it? Given your results, I doubt you have a 64-bit version.


I haven't come around to building the middle type, I only used 64-bit indexing builds till now, but I'll have time this week to try.

FYI, with Octave-3.9.1+ w. 64-bit indexing and Matlab 2015b prerelease (64-bit) on 64-bit Win7 8 GB RAM, arrays of 30000x30000 double ("rand (30000)") could easily be accommodated (as expected, of course) within a mere 10 seconds, total memory occupied by Octave = 7.2 GB, by ML ~7.6 GB.
I also tried "rand (40000)" but then it took 15 minutes to get it accommodated, as it overflowed physical RAM capacity and in the end ~5 GB was in the swap file on hard disk (w. 99 % RAM usage).

Philip Nienhuis <philipnienhuis>
Group Member
Sun 01 Mar 2015 06:05:21 PM UTC, comment #8: 

OK, so in comment #5 you used 3.6.4 and could not allocate the equivalent of (assuming four bytes per double vector value, although maybe it should be eight bytes for 64-bit, not that important because it's a rough guess) 4*256MB or 1GB memory.  In comment #7 you used 3.8.2 and was able to go a little further with the vector size, but still nowhere near the amount of memory on your system.  Would Windows fragmentation be that bad?

I doubt the "Run in separate memory space" has any bearing on this.  It sounds like that is for throw-back programs that are 16-bit (DOS) and should still be runnable under Windows:

http://www.sevenforums.com/tutorials/13239-run-command-add-run-separate-memory-space.html

Apparently there were limitations in DOS such that apps couldn't really use features of modern OS's.

The reason you can't uncheck the box is that your compiled Octave is 64-bit and its the modern OS so automatically is run in its own separate memory space.  That's my takeaway, as I'm not a Windows user.

At the same time, you might be right to look for an OS-related configuration related to memory.  Again, fragmentation of what should be at least 30 GB of Memory seams odd.  You could try stretching/maxing memory usage in other applications, say an editor or a spreadsheet.  Just copy the whole contents and paste twice, repeat--something like that.

Dan Sebald <sebald>
Sun 01 Mar 2015 08:47:35 AM UTC, comment #7: 

On 64-bit Octave 3.8.2 on Windows, I find that the memory limit is stretched to 1.45 GB rather than being under 1 GB. I tested by allocating large matrices after using "clear", and checking memory utilization with "whos".

Puronjoy B <puronjoy>
Sun 01 Mar 2015 08:33:21 AM UTC, comment #6: 

I have also noticed that under "Advanced properties" selected from the menu obtained on right-clicking the Octave icon, there is a box for "Run in separate memory space" which is ticked by default, and cannot be modified. Could this have something to do with the memory problem?

Puronjoy B <puronjoy>
Sun 01 Mar 2015 08:12:11 AM UTC, comment #5: 

I am indeed referring to 64-bit Windows and 64-bit Linux.
I have faced the problem with experimental Octave 3.8.2 on Windows as well. The code provided above yields the following output:

...
vecsize =  268435456
error: out of memory or dimension too large for Octave's index type

Puronjoy B <puronjoy>
Sat 28 Feb 2015 07:09:12 PM UTC, comment #4: 

Let's use "rand" to ensure memory is actually used, and put a pause in there so the effect on memory might be monitored externally.


vecsize = 1;
while (1)
  vecsize = 2 * vecsize
  x = rand(vecsize, 1);
  pause();
  clear x;
end


The pause will stop the program at a point where the variable x is still resident in memory.  On my system, I pause and then sit and wait for the system disk usage monitor to update (every five seconds or so).  I'm able to watch the memory usage slowly grow to 87.5%.  I then stop there to avoid going into swap land.

Dan Sebald <sebald>
Sat 28 Feb 2015 06:33:55 PM UTC, comment #3: 

OK, 64 bit then...  Could you write some test script that illustrates what the problem is?  I'm running on linux, but if there were a text script that failed when memory was consumed, it could point to something.

For example, on my machine, the following


vecsize = 1;
while (1)
  vecsize = 2 * vecsize
  x = ones(vecsize, 1);
  clear x;
end


produces


vecsize =  2
vecsize =  4
vecsize =  8
[snip]
vecsize =  268435456
vecsize =  536870912
vecsize =    1.0737e+09


at which point all my RAM is consumed and swap space looked like it was on the path of reaching the max before the OS slowed to halt because of all the swapping.

If I'm understanding the problem, on Windows your finding that vector size to be quite small, or the memory never gets totally consumed?

Dan Sebald <sebald>
Sat 28 Feb 2015 05:06:11 PM UTC, comment #2: 

A while ago I posted in the help-octave list or in a  bug report about memory limits; as I'm abroad w.limited web access (typing on an old tablet) I can't search now. It was IIRC some 6-8 months ago.

 Your problem is memory fragmentation of the OS itself. I compared Matlab 32b and Octave 32b on a 64bit Win7 w. 8 GB RAM, both could only create an array of IIRC max. ~440 MB immediately after booting (i.e., as first user program). Nothing to be done about it.
With an Octave build f.64 bits it could be better.

BTW you use an old Octave version (3.6.4); there are 64-bit indexing versions of 3.8.2 f. Windows available. Search the wiki for the URL

Philip Nienhuis <philipnienhuis>
Group Member
Fri 27 Feb 2015 05:54:53 PM UTC, comment #1: 

Are you talking about 64-bit Windows and 64-bit Linux?  Along with the 64-bit compilation of the software?  32-bit accesses somewhere around 3.8 GB and with the OS using some of that memory could leave only 3 GB to work with.  And perhaps the 32-bit libraries impose some additional limitations, so even if you have a 64-bit OS, a 32-bit app might have limitations.

Dan Sebald <sebald>
Fri 27 Feb 2015 10:08:15 AM UTC, original submission:  

I am running GNU Octave on a dual-booting PC with 32 GB RAM, but the Windows installer (Win 8) of GNU Octave (3.6.4) limits RAM usage to less than 1 GB. On Linux, I find no corresponding limit for the current version of GNU Octave (3.8), and am able to load large multi-GB matrices until the entire RAM is used up (thereafter, swap memory is used). What is the way to solve the memory issue on Windows?

Puronjoy B <puronjoy>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by andy1978 (Posted a comment)
  • -email is unavailable- added by tomd1982 (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by puronjoy (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2015-03-08 philipnienhuis CategoryOctave Package Configuration and Build System
        StatusNone Invalid / Not an Octave Bug
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code