bugGNU Octave - Bugs: bug #55427, save -v7 exits with "error...

 
 

bug #55427: save -v7 exits with "error compressing data element"

Submitted by:  Philip Nienhuis <philipnienhuis>
Submitted on:  Wed 09 Jan 2019 12:57:54 PM UTC  
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error
Status:  None Assigned to:  None
Originator Name:  Philip Nienhuis Open/Closed:  Open
Release:  dev Operating System:  Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Fri 11 Jan 2019 10:22:17 AM UTC, comment #8:

What I'm wondering is why essentially the same library behaves differently on Linux and Windows. It's built from the same source code after all. Perhaps configure options?

I glanced through the zlib sources (zlib-1.2.11-1.src.tar.gz) but couldn't find any mention of size limits relevant to the issue at hand. But admittedy my C++ proficiency is quite lacking...

Philip Nienhuis <philipnienhuis>
Project Member
Thu 10 Jan 2019 09:50:47 PM UTC, comment #7:

It's not meant to be a Matlab compatible workaround, I just wanted to see if Octave's two different uses of zlib worked. And as I suspected it looks like 'compress' has problems on Windows when calling it on a single large data object, but using the stream-based 'deflate' method does work on Windows.

Therefore a method of fixing this for Windows might be to rewrite ls-mat5.cc to use 'deflate' in a loop instead of 'compress'.

Mike Miller <mtmiller>
Project Administrator
Thu 10 Jan 2019 09:44:05 PM UTC, comment #6:

Thanks Mike, that does work on Windows. But it is terribly slow;
on Linux:

I tried to read the resulting file in Matlab (r2019a prerelease) but that doesn't work - on Windows, Matlab just hangs forever with CPU usage < 1 %. On linux, I get this:

.. so -z is not a good option as far as Matlab compatiility is concerned.

Can something be done about the internal memory allocation you mentioned? On Linux it seems to be configured differently.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 10 Jan 2019 06:34:55 PM UTC, comment #5:

This may simply have to do with internal memory allocation in the zlib 'compress' function, which compresses a single buffer at once. It might be necessary to switch to using the stream based 'deflate' function to handle objects over a certain size.

Does 'save -z' work for you on the same system that fails on 'save -v7'? The v7 format uses 'compress' to compress each element in the Matlab file format, while 'save -z' uses zlib's 'gzwrite' to write a normal file stream through the 'deflate' function.

Mike Miller <mtmiller>
Project Administrator
Thu 10 Jan 2019 05:40:19 PM UTC, comment #4:

On Linux it works for me as well (and fast), also the original file (577 MB in 'whos') in my report comment #0.

But not so on Windows, as I indicated in the original report.

Status back to "None" until someone on Windows can confirm e.g., with the example in comment #1.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 10 Jan 2019 10:07:56 AM UTC, comment #3:

It may well be a Windows-specific thing:

While saving, RAM usage goes up ~1.5 GB before "save" gives up; total max RAM in use by all processes = 5.11 GB out of 8 GB available so I surmise OOM wouldn't play a role.

But:

goes fine and file foo2.mat is about 760 KB.

@Rik comment #2:
I doubt whether it is directly related to the specific file format. If the nested struct array is just smaller, yet complexity is the same, it'll work fine. AFAICS it's just the data object size that matters.

FTR, Octave is configured with --enable-64 and fortran 64 indexing (can't find the latter literally in __octave_config_info__ , I only see 'F77_INTEGER_8_FLAG=-fdefault-integer-8').

I have a feeling that thus bug isn't so much Octave's fault but rather due to the compression library on MinGW that is invoked by Octave.

(Release => dev)

Philip Nienhuis <philipnienhuis>
Project Member
Wed 09 Jan 2019 11:07:51 PM UTC, comment #2:

Works for me too, but on Ubuntu not Windows.

Can you try the simple example that Mike provided? If that works then it is something specific to the file you have. It could be that certain formats, like structs, are not handled correctly while ordinary objects, like arrays, are.

Rik <rik5>
Project Administrator
Wed 09 Jan 2019 09:50:20 PM UTC, comment #1:

I can't reproduce this on Debian with

The resulting file foo.mat is about 750 MB instead of the 800 MB saved with -v6. And if the matrix is all zero instead of random, the resulting file is about 800 kB.

While compressing Octave's resident memory goes up to about 3 GB, which is released before the save command returns.

Mike Miller <mtmiller>
Project Administrator
Wed 09 Jan 2019 12:57:54 PM UTC, original submission:

"save -v7" isn't able to write data to file if the data is larger than in the order of ~100-150 Mbytes (according to "whos").
As the data question (here a 1x1 struct) contain nested structs, struct arrays and small cellstr arrays and double arrays it may be a bit larger than "whos" reports.

"save -v6" goes fine; the size of the .mat-file on disk more or less equals the size reported by "whos" (e.g.,

... leads to a file of 574,963 KB.

Needless to say Matlab happliy runs the same code fine and creates fairly big .mat-files w/o issue; the above struct becomes a 127 MB file.

According to the Matlab docs the difference between -v6 and -v7 is merely data compression and Unicode support.

Octave has no issues reading a big mat-file created by Matlab with -v7 option; it is specifically writing to file that goes wrong.

If required I can upload the .mat file somewhere.

Philip Nienhuis <philipnienhuis>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by philipnienhuis (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-10 philipnienhuis StatusWorks For Me => None
    2019-01-10 philipnienhuis Release5.0.1 => dev
    2019-01-09 rik5 StatusNone => Works For Me

    Back to the top


    Powered by Savane 3.4