bugGNU Octave - Bugs: bug #48018, This .mat file version 5 cannot be...

 
 

bug #48018: This .mat file version 5 cannot be loaded in Octave (while it can in Matlab)

Submitted by:  None
Submitted on:  Wed 25 May 2016 09:41:33 PM UTC  
 
Category: Octave FunctionSeverity: 2 - Minor
Priority: 3 - LowItem Group: Matlab Compatibility
Status: Wont FixAssigned to: None
Originator Name: Cristian VersariOriginator Email: -unavailable-
Open/Closed: ClosedRelease: 4.0.2
Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Thu 26 May 2016 03:31:26 AM UTC, comment #9:

Agree. It was an interesting problem while it lasted. I think the bug can be considered to be in either EFMtool or in the JMatIO library that is bundled in its jar file and is what I assume is used to write mat-files.

Mike Miller <mtmiller>
Project Administrator
Thu 26 May 2016 12:21:04 AM UTC, comment #8:

I tracked it down to something similar. I wasn't as handy as you at being able to uncompress just a part of the file. But single-stepping with gdb shows that the tag for the bad file is 13,448 bytes. And in ls-mat5.cc there is

destLen is the recently uncompressed length tag.

Just for fun, I modified ls-mat5.cc to have

which gives a bigger buffer. The uncompress call sets destLen to the true size, so this doesn't do anything other than waste memory most of the time.

With that in place, I get a little farther and the new error is

This is again probably because the element lengths are off. I'm not sure how Matlab is managing to cope with that, but that is a curiosity. I think we should mark this as "Won't Fix" since it is some strangeness is the EFM tool code.

Rik <rik5>
Project Administrator
Wed 25 May 2016 11:23:09 PM UTC, comment #7:

I was curious so I worked on this for a half hour. Here's what I've found as to the cause of this error.

Both files contain zlib-compressed data. The mat v5 file specification has a size field outside of the compression, and then a size of the data element inside of the compression.

The file produced by Matlab is entirely consistent. The file size is 1176 bytes, 136 bytes of standard file header, and 1040 bytes of zlib-compressed data. Uncompressing 1040 bytes results in exactly 6896 bytes, which is also exactly what the value of the size field is in the first few bytes of the compressed block.

The file produced by EFMtool uncompresses into more data than it claims should be there. The file is 1362 bytes, again 136 bytes of header and 1226 bytes of zlib-compressed data. Uncompressing the trailing 1226 bytes yields 14,152 bytes. However, the data size field in the first few bytes of the compressed block say the data should be 13,448 bytes.

This doesn't seem like an error on the surface, but I'm guessing that Octave attempts to call uncompress of 14,152 bytes into a buffer of length 13,448, and gets the Z_BUF_ERROR return value because the output buffer wasn't large enough to finish decompressing.

Mike Miller <mtmiller>
Project Administrator
Wed 25 May 2016 10:47:16 PM UTC, comment #6:

I should add that we've been discussing this on #octave. It's pretty low priority, more of a curiosity. Yes, the original big-endian file is produced by the non-Matlab Java software. The files it generates are able to be read by Matlab but not by Octave. If no one is interested in investigating, it can't be determined what's wrong with the file, or if it can be blamed on the other software, then this can certainly be closed Won't Fix.

Mike Miller <mtmiller>
Project Administrator
Wed 25 May 2016 10:44:44 PM UTC, comment #5:

Nevermind, I see the point of the error is when uncompressing part of the file after reading the length from the same file. So the endianness may indeed be an issue giving a wrong value for the size of the compressed data element.

Mike Miller <mtmiller>
Project Administrator
Wed 25 May 2016 10:38:54 PM UTC, comment #4:

The new little endian version of the file loads fine for me.

Was the original file generated by Matlab, or was it generated outside of Matlab in Java by the efmtool? It might be that the non-Matlab software doesn't truly have the file format correct. When you load it in to Matlab and re-save it then it gets written out in a completely correct format.

Rik <rik5>
Project Administrator
Wed 25 May 2016 10:29:43 PM UTC, comment #3:

Here attached is the same .mat file after loading it in Matlab and saving it back to the supposedly same .mat format. It gives no issues when loading it in Octave

(file #37264)

Anonymous
Wed 25 May 2016 10:20:23 PM UTC, comment #2:

If there were simply a problem with endianness, wouldn't invalid data be read successfully? This seems more like a field that we are expecting to read isn't long enough or is missing in the file.

The Z_BUF_ERROR error code means "there was not enough room in the output buffer" according to http://www.zlib.net/manual.html.

Mike Miller <mtmiller>
Project Administrator
Wed 25 May 2016 09:57:54 PM UTC, comment #1:

When you load the mat file in Matlab, are there any special variables types it is using: tables, structs, cell arrays, etc.

Using the Unix utility 'file' shows something interesting

My computer, which uses an Intel CPU, reports "little endian" for any .mat file I create with Octave. If there isn't a special variable type being used, then it might be that the .mat file was created on a big endian machine and is trying to be read on a little endian one. Of course, that should be possible, but it might be an indication of where to look.

Rik <rik5>
Project Administrator
Wed 25 May 2016 09:41:33 PM UTC, original submission:

The attached .mat file produced by EFMtool [1] cannot be imported in Octave:

Installation details:
Octave 4.0.2 from Debian testing amd64
zlib 1.2.8

[1] http://www.csb.ethz.ch/tools/software/efmtool.html

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #37264:  efms_01.mat added by None (1KiB - application/octet-stream)
file #37263:  efms_0.mat added by None (1KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 26 May 2016 03:31:26 AM UTCmtmillerStatusNone=>Wont Fix
      Open/ClosedOpen=>Closed
    Wed 25 May 2016 10:47:56 PM UTCmtmillerSeverity3 - Normal=>2 - Minor
      Priority5 - Normal=>3 - Low
    Wed 25 May 2016 10:29:43 PM UTCNoneAttached File-=>Added efms_01.mat, #37264
    Wed 25 May 2016 09:41:33 PM UTCNoneAttached File-=>Added efms_0.mat, #37263

    Back to the top


    Powered by Savane 3.1-cleanup1