bugGNU Octave - Bugs: bug #50603, Document limitations of various...

 
 

bug #50603: Document limitations of various file formats available for save()

Submitter:  None
Submitted:  Tue 21 Mar 2017 05:09:56 PM UTC
   
 
Category:  Documentation Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Documentation
Status:  Confirmed Assigned to:  None
Originator Name:  David Nichols Originator Email:  -email is unavailable-
Open/Closed:  * Open Release:  * 4.2.1
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 23 Mar 2017 04:28:31 PM UTC, comment #9: 

"Using the stream version of libz seems like a really good project to get working."

I agree.  It looks like there are a few parameters to set, and I assume the read side of things could use stream support as well.

Dan Sebald <sebald>
Wed 22 Mar 2017 09:22:42 PM UTC, comment #8: 

It appears that one can shift this to "documentation bug," changing things so that the documentation for save points out that some of the matlab formats are unable to contain data objects which Octave easily creates/saves/loads.

Changing the output to hdf5 sounds like a good idea - use a format designed to hold the kinds of objects 64-bit computers use.

David Nichols <david_nichols>
Wed 22 Mar 2017 09:15:57 PM UTC, comment #7: 

Using the stream version of libz seems like a really good project to get working.

For the original reporter, if you just need to save/restore large data files you might try the '-hdf5' option rather than '-mat'.  That could be a good workaround.

Rik <rik5>
Group administrator
Wed 22 Mar 2017 06:58:22 PM UTC, comment #6: 

Yes, roughly the stream buffer (z_stream) is more memory efficient.  It looks something like


deflateInit((z_streamp strm, int level));
while (memory_left) {
    <point to new memory or copy memory to some known small buff>
    deflate((z_streamp strm, int flush));
}
deflateEnd((z_streamp strm));


and the size of the small buffer is indicated to the deflate() routine via the z_stream structure that is pointed to, in element "avail_in".  deflate() must keep an internal state between calls to it so that new output data can be generated when adequate input data comes along...probably random just when that is the case depending on compression behavior.  Then deflateEnd must look at the state, what data remains, then wrap things up as far as remaining compressed data goes.

Dan Sebald <sebald>
Wed 22 Mar 2017 06:13:24 PM UTC, comment #5: 

OK, I see that.  It's the code near this comment:


          // destLen must be at least 0.1% larger than source buffer
          // + 12 bytes.  Reality is it must be larger again than that.


you are referring to.

Is the compress() function originating from zlib.h?  If so, zlib

http://www.zlib.net/manual.html

categorizes this function as a utility:

"The following utility functions are implemented on top of the basic stream-oriented functions. To simplify the interface, some default options are assumed (compression level and memory usage, standard memory allocation functions)."

Maybe the "basic stream-oriented functions" have a small memory footprint.

Making a change (and testing) might be a bit to tackle at the moment.  Well, definitely not a change to make shortly before a release.

Dan Sebald <sebald>
Wed 22 Mar 2017 04:50:56 PM UTC, comment #4: 

@Dan: I think you're right.  We are using the lzip compression library and there should be some option to set the chunk size that we operate on.  Or maybe Octave is being bad and just passing a pointer to the entire buffer.  The file to check is libinterp/corefcn/ls-mat5.cc.  The mat5 is a bit of a misnomer as this file handles mat5, mat6, and mat7 formats.

Rik <rik5>
Group administrator
Wed 22 Mar 2017 07:31:11 AM UTC, comment #3: 

Having just tried the -v7 option on an 8G memory machine, I suggest being very careful to have no other applications running such that almost all memory is free.  Otherwise, the system will get caught in a swap-memory trap and have a really difficult time getting out of it.

First, let me say that the compression option should in theory work faster, not slower.  The reason is that, as I pointed out in a previous post, barely any CPU is used for the -mat option and all the time spent is the disk churning away.  Compression should utilize all the bandwidth of the CPU(s) creating less memory that needs to be stored to disk and hence less disc activity.

However, my guess is that the -v7 compression may not be done quite right in Octave, or at least it isn't done in a memory efficient way.  And it should be memory efficient because the large data elements are what we'd like compressed.

When I ran with the -v7 option, because the x variable was 4G memory, it seemed like Octave wanted to do the compression with some large data buffer to work with.  So it needed far more memory than my machine had available.  And I think because the needed memory chunk was so big, it wasn't a little bit of memory that went into the swap drive, it was 50%.  This brought the system to a standstill.  Using Cntrl-C on Octave was problematic as well, because then Octave attempts to store variables to octave-workspace, which of course is big:

-rw-r--r--. 1 sebald sebald 3115411537 Mar 22 01:46 octave-workspace

Killing the process did clear big swaths of memory, but still everything else was stuck in swap memory and remained slow.

It may be worth looking at the -v7 compression and see how memory intensive it is.  If it is using a lot of memory--on the order of the elements being saved--then it would definitely be worth attempting some algorithm that can perhaps work on smaller hunks of data at a time while streaming to disk, thereby keeping the extra memory required for compression small.

Dan Sebald <sebald>
Wed 22 Mar 2017 06:03:05 AM UTC, comment #2: 

Have a look at Matlab's own documentation (https://www.mathworks.com/help/matlab/ref/save.html).

They have a chart with the limits of each format.  The '-v6' option, which is what Octave defaults to for '-mat', is limited to 2^31 bytes per variable.

At least on 64-bit Linux, I am able to write in -v6 format (3.1G) and -v7 format (4.4M).

As a workaround, you might try saving in -v7 format.  It is much slower, because it does compression, but the eventual size of the element to save is pretty small.

Rik <rik5>
Group administrator
Tue 21 Mar 2017 06:19:47 PM UTC, comment #1: 

944*944*437 * 8 bytes/double = 3.1 G which is near the boundary of 32 bit memory space.  Are you running a 32-bit build of Octave?  Or a 64-bit build of Octave?

I'm using 64-bit build on a 64-bit version of Linux (otherwise, even the operating system wouldn't access but 4G of memory) and things seem to work fine (uses barely any CPU while the disk churns away):


octave:2> 2^32
ans =    4.2950e+09
octave:3> x = ones([944 944 437]);
octave:5> 944*944*437 * 8
ans =    3.1154e+09
octave:6> save -mat "junk.mat"
octave:7> clear
octave:8> load "junk.mat"
octave:9> size(x)
ans =

   944   944   437



$ ls -l junk.mat
-rw-r--r-- 1 sebald sebald 3115411712 Mar 21 13:12 junk.mat


Dan Sebald <sebald>
Tue 21 Mar 2017 05:09:56 PM UTC, original submission:  

Trying to save a large variable to a file produces an incorrect result.  For instance, the commands below produce a small (192 byte) file which does not load correctly.  (The same thing occurs with actual data, so it's not that save is smart and tries to compress things.) Note that the data object in question is larger than 2^31 bytes.



>> x=ones([944 944 437]);
>> save -mat "junk.mat"
>> clear
>> load "junk.mat"
error: load: reading matrix data for 'x'


Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by david_nichols (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-03-22 rik5 CategoryInterpreter Documentation
        Item GroupIncorrect Result Documentation
        StatusNone Confirmed
        Operating SystemMicrosoft Windows Any
        Summarysave -mat fails for large (# of bytes) variables Document limitations of various file formats available for save()

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code