bugGNU Octave - Bugs: bug #55427, save -v7 exits with "error...

 
 

bug #55427: save -v7 exits with "error compressing data element"

Submitter:  Philip Nienhuis <philipnienhuis>
Submitted:  Wed 09 Jan 2019 12:57:54 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error or Warning
Status:  Fixed Assigned to:  None
Originator Name:  Philip Nienhuis Open/Closed:  * Closed
Release:  * 6.2.90 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 05 Jul 2021 03:38:56 PM UTC, comment #19: 

Marking as fixed and closing report.

Rik <rik5>
Group administrator
Sun 04 Jul 2021 08:01:30 PM UTC, comment #18: 

Indeed, loading the individual .mat files directly into Matlab gives the exact same RAM usage as when loading them through a combined -v7 .mat file prepared by Octave.
With that issue checked I suppose this bug has been fixed & can be closed.
I can still check next week with the original .mat files that provoked this bug report just to be sure, but I doubt if that'll yield surprises.

Philip Nienhuis <philipnienhuis>
Group Member
Sun 04 Jul 2021 10:16:17 AM UTC, comment #17: 

It occurred to me that I could also try to load the individual .mat files directly in Matlab; I'll do that tonight.

My aim was primarily to check if Octave correctly saves the various compressed data.
Your explanation (thanks!) almost makes for another feature request: Octave showing its overhead for storing data structures :-) Could be enlightening if not confronting info.
I already figured years ago that the "gis" struct in my example is probably quite memory-inefficient; each field contains structs containing o.a., struct arrays of grids each with fairly different dimensions. There must be a lot of "air" in the data structure in RAM. But I didn't expect a factor of 4 even in Matlab ...
(OTOH that struct is very efficient for how it is used, but that's quite another story)

Philip Nienhuis <philipnienhuis>
Group Member
Sat 03 Jul 2021 08:36:26 PM UTC, comment #16: 

I think the difference in sizes may be due to differences in storage calculations.  Octave, for example, only accounts for the storage of the actual data, but for the overhead of structs themselves.  For example, I tried this test code


x.a.b = ones (1e6,1);
whos x


In Octave, this returns 8,000,000 (1 million elements at 8 bytes per element.  In Matalb, this returns, 8,000,336 (8MB of data + 336 of overhead for the struct).

Similarly for


y = {ones(10,1, 'uint8')};
whos y


Octave reports just 10 bytes, while Matlab reports 114 bytes.

You could always check to make sure the data itself is intact by saving a struct "var1" from Matlab, loading the struct in Octave, renaming the variable ("var2 = var1;"), saving the struct in Octave, loading the old and new structs in Matlab and then use "assert (var1, var2)" to verify that they are the same.


You could run a test and save the structure

Rik <rik5>
Group administrator
Sat 03 Jul 2021 07:24:28 PM UTC, comment #15: 

Using the 6.2.93 nightly from July 3, I tried "save -v7" with a bunch of large data structures I have lying around on my laptop:

>> whos
Variables visible from the current scope:

variables in scope: top scope

   Attr Name        Size                     Bytes  Class
   ==== ====        ====                     =====  =====
        ans        95x1                        804  cell
        divers      1x148                158144931  struct
        gis         1x1                   55582705  struct
        gr          1x1                   65228745  struct

Total is 245 elements using 278957185 bytes


(ans = "fieldnames (gr)")
During "save -v7 test.mat divers gis gr" I got no errors (nice!) but after loading the resulting 58 MB file in Matlab r2014a, I get:

>> whos
  Name         Size                 Bytes  Class     Attributes

  ans         95x1                  12248  cell
  divers       1x148            158468902  struct
  gis          1x1              213452354  struct
  gr           1x1               81220402  struct


where to my surprise some structs occupy much more memory (in bytes) than in Octave.
I'd have to dive deeper into it to find out why (the deeply nested) structs 'gis' and 'gr' got blown up that much, but that'll be a lengthy affair I'm afraid. Or maybe Octave is just much more memory-efficient?
(Running "pack" at the Matlab command line didn't change even just one byte of the occupied memory sizes, as can be expected when loading a .mat file in a pristine Matlab session.)

When I'm back I can try bigger struct arrays - at work we got some 140+ MB .mat files created with Matlab using "-v7". It's those files with which I hit this bug.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 02 Jul 2021 08:13:35 PM UTC, comment #14: 

Wow, awesome if this bug is fixed. Up til now I had to save large struct arrays with -v6, read them in Matlab and save from there with -v7. I think the 40 MB limit more or less matches my own experience with this bug.
I'll try the fix when I'm back, somewhere mid July.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 01 Jul 2021 08:41:10 PM UTC, comment #13: 

replacing this line


uLongf destLen = srcLen * 101 / 100 + 12;


with


uLongf destLen = double(srcLen) * 101 / 100 + 12;


also makes the standalone program work, with this result:


srcLen=209715200 destLen=211812364
after compression: destLen=181079622


the srcLen * 101 apparently overflows, as you already diagnosed.
Your solution looks fine, good find the compressBound.

The -5 error return is Z_BUF_ERROR

A.R. Burgers <arb>
Thu 01 Jul 2021 03:17:36 PM UTC, comment #12: 

I pushed a change to stable here:
https://hg.savannah.gnu.org/hgweb/octave/rev/c4f9513b09f1

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Thu 01 Jul 2021 11:50:48 AM UTC, comment #11: 

The uLongf type that is used for the input and output buffer size is a unsigned long integer. That is 64 bit on Linux, but it is 32 bit on Win64.
IIUC, the calculation that we use for estimating the destination buffer size overflows if the source buffer is 2^32/101 bytes or larger (that is just over 40 MiB if I'm not mistaken).

I'm currently cross compiling with a changeset. If using `compressBound` works, I'll probably push to the stable branch.

Markus Mützel <mmuetzel>
Group administrator
Thu 01 Jul 2021 11:14:20 AM UTC, comment #10: 

Thanks for the test program.

Instead of estimating the size of the destination buffer with our own calculation, we could maybe use `compressBound`.
It looks like this function was added to zlib in version 1.2.0 (in 2003). So we probably can use it without a configure check: https://github.com/madler/zlib/blob/master/ChangeLog#L998

That fixes the error for me with the test program by @arb on mingw64.

Markus Mützel <mmuetzel>
Group administrator
Thu 01 Jul 2021 07:49:38 AM UTC, comment #9: 

don't know if it helps, but this stand-alone program reproduces the issue.


#include <string>
#include <iostream>
#include <random>
#include <zlib.h>

int main(int argc, char *argv[])
{
  int n_mega = 200;
  size_t startLen = n_mega * 1024 * 1024;
  std::string buf_str(startLen, 'x');
  uLongf srcLen = buf_str.length();
  uLongf destLen = srcLen * 101 / 100 + 12;
  char * out_buf = new char[destLen];

  if (1) {
    // make buf_str more difficult to compress by
    // filling it with random numbers
    std::random_device rd;
    std::mt19937 gen(rd());
    std::uniform_int_distribution <unsigned char> distrib(1, 116);

    for (size_t n = 0; n < startLen; ++n) {
      buf_str[n] = distrib(gen);
    }
  }

  std::cout << "srcLen=" << srcLen << " destLen=" << destLen << std::endl;

  int retval = compress (reinterpret_cast       <Bytef *> (out_buf),        &destLen,
                         reinterpret_cast <const Bytef *> (buf_str.c_str()), srcLen);
  if (retval != Z_OK) {
    std::cerr << "save: error compressing data element. retval=" << retval << std::endl;
    return 0;
  }
  std::cout << "after compression: destLen=" << destLen << std::endl;

  return 0;
}


this is the output from the mingw64 console with the error code:


$ g++ -o zz zz.cc -lz
$ ./zz
srcLen=209715200 destLen=40013672
save: error compressing data element. retval=-5


A.R. Burgers <arb>
Fri 11 Jan 2019 10:22:17 AM UTC, comment #8: 

What I'm wondering is why essentially the same library behaves differently on Linux and Windows. It's built from the same source code after all. Perhaps configure options?

I glanced through the zlib sources (zlib-1.2.11-1.src.tar.gz) but couldn't find any mention of size limits relevant to the issue at hand. But admittedy my C++ proficiency is quite lacking...

Philip Nienhuis <philipnienhuis>
Group Member
Thu 10 Jan 2019 09:50:47 PM UTC, comment #7: 

It's not meant to be a Matlab compatible workaround, I just wanted to see if Octave's two different uses of zlib worked. And as I suspected it looks like 'compress' has problems on Windows when calling it on a single large data object, but using the stream-based 'deflate' method does work on Windows.

Therefore a method of fixing this for Windows might be to rewrite ls-mat5.cc to use 'deflate' in a loop instead of 'compress'.

Mike Miller <mtmiller>
Group Member
Thu 10 Jan 2019 09:44:05 PM UTC, comment #6: 

Thanks Mike, that does work on Windows. But it is terribly slow;
on Linux:

>> x = rand (1e5, 1e3);
>> tic; save -v6 foo6.mat; toc
Elapsed time is 0.58468 seconds.
>> tic; save -v7 foo7.mat; toc
Elapsed time is 33.2968 seconds.
>> tic; save -z fooz.mat; toc
Elapsed time is 209.423 seconds.
>>


I tried to read the resulting file in Matlab (r2019a prerelease) but that doesn't work - on Windows, Matlab just hangs forever with CPU usage < 1 %. On linux, I get this:

>> load('foo6.mat')
>> tic; load foo6.mat; toc
Elapsed time is 0.313205 seconds.
>> tic; load foo7.mat; toc
Elapsed time is 4.665843 seconds.
>> tic; load fooz.mat; toc
Error using load
Unable to read MAT-file /home/philip/fooz.mat. Not a binary MAT-file. Try load -ASCII to read as
text.

>>


.. so -z is not a good option as far as Matlab compatiility is concerned.

Can something be done about the internal memory allocation you mentioned? On Linux it seems to be configured differently.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 10 Jan 2019 06:34:55 PM UTC, comment #5: 

This may simply have to do with internal memory allocation in the zlib 'compress' function, which compresses a single buffer at once. It might be necessary to switch to using the stream based 'deflate' function to handle objects over a certain size.

Does 'save -z' work for you on the same system that fails on 'save -v7'? The v7 format uses 'compress' to compress each element in the Matlab file format, while 'save -z' uses zlib's 'gzwrite' to write a normal file stream through the 'deflate' function.

Mike Miller <mtmiller>
Group Member
Thu 10 Jan 2019 05:40:19 PM UTC, comment #4: 

On Linux it works for me as well (and fast), also the original file (577 MB in 'whos') in my report comment #0.

But not so on Windows, as I indicated in the original report.

Status back to "None" until someone on Windows can confirm e.g., with the example in comment #1.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 10 Jan 2019 10:07:56 AM UTC, comment #3: 

It may well be a Windows-specific thing:

GNU Octave, version 6.0.0
:
Octave was configured for "x86_64-w64-mingw32".
:

>> x = rand (1e5, 1e3);
>> save -v7 foo.mat
error: save: error compressing data element
>>


While saving, RAM usage goes up ~1.5 GB before "save" gives up; total max RAM in use by all processes = 5.11 GB out of 8 GB available so I surmise OOM wouldn't play a role.

But:

>> x = zeros (1e5, 1e3);
>> save -v7 foo2.mat
>>


goes fine and file foo2.mat is about 760 KB.

@Rik comment #2:
I doubt whether it is directly related to the specific file format. If the nested struct array is just smaller, yet complexity is the same, it'll work fine. AFAICS it's just the data object size that matters.

FTR, Octave is configured with --enable-64 and fortran 64 indexing (can't find the latter literally in __octave_config_info__ , I only see 'F77_INTEGER_8_FLAG=-fdefault-integer-8').

I have a feeling that thus bug isn't so much Octave's fault but rather due to the compression library on MinGW that is invoked by Octave.

(Release => dev)

Philip Nienhuis <philipnienhuis>
Group Member
Wed 09 Jan 2019 11:07:51 PM UTC, comment #2: 

Works for me too, but on Ubuntu not Windows.

Can you try the simple example that Mike provided?  If that works then it is something specific to the file you have.  It could be that certain formats, like structs, are not handled correctly while ordinary objects, like arrays, are.

Rik <rik5>
Group administrator
Wed 09 Jan 2019 09:50:20 PM UTC, comment #1: 

I can't reproduce this on Debian with


x = rand (1e5, 1e3);
save -v7 foo.mat


The resulting file foo.mat is about 750 MB instead of the 800 MB saved with -v6. And if the matrix is all zero instead of random, the resulting file is about 800 kB.

While compressing Octave's resident memory goes up to about 3 GB, which is released before the save command returns.

Mike Miller <mtmiller>
Group Member
Wed 09 Jan 2019 12:57:54 PM UTC, original submission:  

"save -v7" isn't able to write data to file if the data is larger than in the order of ~100-150 Mbytes (according to "whos").
As the data question (here a 1x1 struct) contain nested structs, struct arrays and small cellstr arrays and double arrays it may be a bit larger than "whos" reports.

"save -v6" goes fine; the size of the .mat-file on disk more or less equals the size reported by "whos" (e.g.,

Attr Name Size     Bytes  Class
==== ==== ====     =====  =====
     D    1x1  584238640  struct


... leads to a file of 574,963 KB.

Needless to say Matlab happliy runs the same code fine and creates fairly big .mat-files w/o issue; the above struct becomes a 127 MB file.

According to the Matlab docs the difference between -v6 and -v7 is merely data compression and Unicode support.

Octave has no issues reading a big mat-file created by Matlab with -v7 option; it is specifically writing to file that goes wrong.

If required I can upload the .mat file somewhere.

Philip Nienhuis <philipnienhuis>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by arb (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by philipnienhuis (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-07-05 rik5 StatusReady For Test Fixed
        Open/ClosedOpen Closed
    2021-07-01 mmuetzel StatusNone Ready For Test
    2021-07-01 mmuetzel Releasedev 6.2.90
    2019-01-10 philipnienhuis StatusWorks For Me None
    2019-01-10 philipnienhuis Release5.0.1 dev
    2019-01-09 rik5 StatusNone Works For Me

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code