bugGNU Octave - Bugs: bug #50603, Document limitations of various...

 
 

bug #50603: Document limitations of various file formats available for save()

Submitted by:  None
Submitted on:  Tue 21 Mar 2017 05:09:56 PM UTC  
 
Category: DocumentationSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Documentation
Status: ConfirmedAssigned to: None
Originator Name: David NicholsOriginator Email: -unavailable-
Open/Closed: OpenRelease: 4.2.1
Operating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Thu 23 Mar 2017 04:28:31 PM UTC, comment #9:

"Using the stream version of libz seems like a really good project to get working."

I agree. It looks like there are a few parameters to set, and I assume the read side of things could use stream support as well.

Dan Sebald <sebald>
Wed 22 Mar 2017 09:22:42 PM UTC, comment #8:

It appears that one can shift this to "documentation bug," changing things so that the documentation for save points out that some of the matlab formats are unable to contain data objects which Octave easily creates/saves/loads.

Changing the output to hdf5 sounds like a good idea - use a format designed to hold the kinds of objects 64-bit computers use.

David Nichols <david_nichols>
Wed 22 Mar 2017 09:15:57 PM UTC, comment #7:

Using the stream version of libz seems like a really good project to get working.

For the original reporter, if you just need to save/restore large data files you might try the '-hdf5' option rather than '-mat'. That could be a good workaround.

Rik <rik5>
Project Administrator
Wed 22 Mar 2017 06:58:22 PM UTC, comment #6:

Yes, roughly the stream buffer (z_stream) is more memory efficient. It looks something like

and the size of the small buffer is indicated to the deflate() routine via the z_stream structure that is pointed to, in element "avail_in". deflate() must keep an internal state between calls to it so that new output data can be generated when adequate input data comes along...probably random just when that is the case depending on compression behavior. Then deflateEnd must look at the state, what data remains, then wrap things up as far as remaining compressed data goes.

Dan Sebald <sebald>
Wed 22 Mar 2017 06:13:24 PM UTC, comment #5:

OK, I see that. It's the code near this comment:

you are referring to.

Is the compress() function originating from zlib.h? If so, zlib

http://www.zlib.net/manual.html

categorizes this function as a utility:

"The following utility functions are implemented on top of the basic stream-oriented functions. To simplify the interface, some default options are assumed (compression level and memory usage, standard memory allocation functions)."

Maybe the "basic stream-oriented functions" have a small memory footprint.

Making a change (and testing) might be a bit to tackle at the moment. Well, definitely not a change to make shortly before a release.

Dan Sebald <sebald>
Wed 22 Mar 2017 04:50:56 PM UTC, comment #4:

@Dan: I think you're right. We are using the lzip compression library and there should be some option to set the chunk size that we operate on. Or maybe Octave is being bad and just passing a pointer to the entire buffer. The file to check is libinterp/corefcn/ls-mat5.cc. The mat5 is a bit of a misnomer as this file handles mat5, mat6, and mat7 formats.

Rik <rik5>
Project Administrator
Wed 22 Mar 2017 07:31:11 AM UTC, comment #3:

Having just tried the -v7 option on an 8G memory machine, I suggest being very careful to have no other applications running such that almost all memory is free. Otherwise, the system will get caught in a swap-memory trap and have a really difficult time getting out of it.

First, let me say that the compression option should in theory work faster, not slower. The reason is that, as I pointed out in a previous post, barely any CPU is used for the -mat option and all the time spent is the disk churning away. Compression should utilize all the bandwidth of the CPU(s) creating less memory that needs to be stored to disk and hence less disc activity.

However, my guess is that the -v7 compression may not be done quite right in Octave, or at least it isn't done in a memory efficient way. And it should be memory efficient because the large data elements are what we'd like compressed.

When I ran with the -v7 option, because the x variable was 4G memory, it seemed like Octave wanted to do the compression with some large data buffer to work with. So it needed far more memory than my machine had available. And I think because the needed memory chunk was so big, it wasn't a little bit of memory that went into the swap drive, it was 50%. This brought the system to a standstill. Using Cntrl-C on Octave was problematic as well, because then Octave attempts to store variables to octave-workspace, which of course is big:

-rw-r--r--. 1 sebald sebald 3115411537 Mar 22 01:46 octave-workspace

Killing the process did clear big swaths of memory, but still everything else was stuck in swap memory and remained slow.

It may be worth looking at the -v7 compression and see how memory intensive it is. If it is using a lot of memory--on the order of the elements being saved--then it would definitely be worth attempting some algorithm that can perhaps work on smaller hunks of data at a time while streaming to disk, thereby keeping the extra memory required for compression small.

Dan Sebald <sebald>
Wed 22 Mar 2017 06:03:05 AM UTC, comment #2:

Have a look at Matlab's own documentation (https://www.mathworks.com/help/matlab/ref/save.html).

They have a chart with the limits of each format. The '-v6' option, which is what Octave defaults to for '-mat', is limited to 2^31 bytes per variable.

At least on 64-bit Linux, I am able to write in -v6 format (3.1G) and -v7 format (4.4M).

As a workaround, you might try saving in -v7 format. It is much slower, because it does compression, but the eventual size of the element to save is pretty small.

Rik <rik5>
Project Administrator
Tue 21 Mar 2017 06:19:47 PM UTC, comment #1:

944944437 * 8 bytes/double = 3.1 G which is near the boundary of 32 bit memory space. Are you running a 32-bit build of Octave? Or a 64-bit build of Octave?

I'm using 64-bit build on a 64-bit version of Linux (otherwise, even the operating system wouldn't access but 4G of memory) and things seem to work fine (uses barely any CPU while the disk churns away):

Dan Sebald <sebald>
Tue 21 Mar 2017 05:09:56 PM UTC, original submission:

Trying to save a large variable to a file produces an incorrect result. For instance, the commands below produce a small (192 byte) file which does not load correctly. (The same thing occurs with actual data, so it's not that save is smart and tries to compress things.) Note that the data object in question is larger than 2^31 bytes.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by david_nichols (Posted a comment)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by sebald (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 22 Mar 2017 09:35:54 PM UTCrik5CategoryInterpreter=>Documentation
      Item GroupIncorrect Result=>Documentation
      StatusNone=>Confirmed
      Operating SystemMicrosoft Windows=>Any
      Summarysave -mat fails for large (# of bytes) variables=>Document limitations of various file formats available for save()

    Back to the top


    Powered by Savane 3.1-cleanup1