Thu 26 May 2016 03:31:26 AM UTC, comment #9:
Agree. It was an interesting problem while it lasted. I think the bug can be considered to be in either EFMtool or in the JMatIO library that is bundled in its jar file and is what I assume is used to write mat-files.
|
Thu 26 May 2016 12:21:04 AM UTC, comment #8:
I tracked it down to something similar. I wasn't as handy as you at being able to uncompress just a part of the file. But single-stepping with gdb shows that the tag for the bad file is 13,448 bytes. And in ls-mat5.cc there is
destLen is the recently uncompressed length tag.
Just for fun, I modified ls-mat5.cc to have
which gives a bigger buffer. The uncompress call sets destLen to the true size, so this doesn't do anything other than waste memory most of the time.
With that in place, I get a little farther and the new error is
This is again probably because the element lengths are off. I'm not sure how Matlab is managing to cope with that, but that is a curiosity. I think we should mark this as "Won't Fix" since it is some strangeness is the EFM tool code.
|
Wed 25 May 2016 11:23:09 PM UTC, comment #7:
I was curious so I worked on this for a half hour. Here's what I've found as to the cause of this error.
Both files contain zlib-compressed data. The mat v5 file specification has a size field outside of the compression, and then a size of the data element inside of the compression.
The file produced by Matlab is entirely consistent. The file size is 1176 bytes, 136 bytes of standard file header, and 1040 bytes of zlib-compressed data. Uncompressing 1040 bytes results in exactly 6896 bytes, which is also exactly what the value of the size field is in the first few bytes of the compressed block.
The file produced by EFMtool uncompresses into more data than it claims should be there. The file is 1362 bytes, again 136 bytes of header and 1226 bytes of zlib-compressed data. Uncompressing the trailing 1226 bytes yields 14,152 bytes. However, the data size field in the first few bytes of the compressed block say the data should be 13,448 bytes.
This doesn't seem like an error on the surface, but I'm guessing that Octave attempts to call uncompress of 14,152 bytes into a buffer of length 13,448, and gets the Z_BUF_ERROR return value because the output buffer wasn't large enough to finish decompressing.
|
Wed 25 May 2016 10:47:16 PM UTC, comment #6:
I should add that we've been discussing this on #octave. It's pretty low priority, more of a curiosity. Yes, the original big-endian file is produced by the non-Matlab Java software. The files it generates are able to be read by Matlab but not by Octave. If no one is interested in investigating, it can't be determined what's wrong with the file, or if it can be blamed on the other software, then this can certainly be closed Won't Fix.
|
Wed 25 May 2016 10:44:44 PM UTC, comment #5:
Nevermind, I see the point of the error is when uncompressing part of the file after reading the length from the same file. So the endianness may indeed be an issue giving a wrong value for the size of the compressed data element.
|
Wed 25 May 2016 10:38:54 PM UTC, comment #4:
The new little endian version of the file loads fine for me.
Was the original file generated by Matlab, or was it generated outside of Matlab in Java by the efmtool? It might be that the non-Matlab software doesn't truly have the file format correct. When you load it in to Matlab and re-save it then it gets written out in a completely correct format.
|
Wed 25 May 2016 10:29:43 PM UTC, comment #3:
Here attached is the same .mat file after loading it in Matlab and saving it back to the supposedly same .mat format. It gives no issues when loading it in Octave
(file #37264)
|
Wed 25 May 2016 10:20:23 PM UTC, comment #2:
If there were simply a problem with endianness, wouldn't invalid data be read successfully? This seems more like a field that we are expecting to read isn't long enough or is missing in the file.
The Z_BUF_ERROR error code means "there was not enough room in the output buffer" according to http://www.zlib.net/manual.html.
|
Wed 25 May 2016 09:57:54 PM UTC, comment #1:
When you load the mat file in Matlab, are there any special variables types it is using: tables, structs, cell arrays, etc.
Using the Unix utility 'file' shows something interesting
My computer, which uses an Intel CPU, reports "little endian" for any .mat file I create with Octave. If there isn't a special variable type being used, then it might be that the .mat file was created on a big endian machine and is trying to be read on a little endian one. Of course, that should be possible, but it might be an indication of where to look.
|
Wed 25 May 2016 09:41:33 PM UTC, original submission:
The attached .mat file produced by EFMtool [1] cannot be imported in Octave:
Installation details:
Octave 4.0.2 from Debian testing amd64
zlib 1.2.8
[1] http://www.csb.ethz.ch/tools/software/efmtool.html
|