bugGNU Octave - Bugs: bug #48018, This .mat file version 5 cannot be...

 
 

bug #48018: This .mat file version 5 cannot be loaded in Octave (while it can in Matlab)

Submitter:  None
Submitted:  Wed 25 May 2016 09:41:33 PM UTC
   
 
Category:  Octave Function Severity:  2 - Minor
Priority:  3 - Low Item Group:  Matlab Compatibility
Status:  Wont Fix Assigned to:  None
Originator Name:  Cristian Versari Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * 4.0.2
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 26 May 2016 03:31:26 AM UTC, comment #9: 

Agree. It was an interesting problem while it lasted. I think the bug can be considered to be in either EFMtool or in the JMatIO library that is bundled in its jar file and is what I assume is used to write mat-files.

Mike Miller <mtmiller>
Group Member
Thu 26 May 2016 12:21:04 AM UTC, comment #8: 

I tracked it down to something similar.  I wasn't as handy as you at being able to uncompress just a part of the file.  But single-stepping with gdb shows that the tag for the bad file is 13,448 bytes.  And in ls-mat5.cc there is


      destLen = tmp[1] + 8;
      std::string outbuf (destLen, ' ');

      // FIXME: find a way to avoid casting away const here!

      int err = uncompress (reinterpret_cast<Bytef *>
                            (const_cast<char *> (outbuf.c_str ())),
                            &destLen, reinterpret_cast<Bytef *> (inbuf),
                            element_length);


destLen is the recently uncompressed length tag.

Just for fun, I modified ls-mat5.cc to have


      destLen = tmp[1] + 8 + 4096;


which gives a bigger buffer.  The uncompress call sets destLen to the true size, so this doesn't do anything other than waste memory most of the time.

With that in place, I get a little farther and the new error is


error: load: invalid array flags subelement


This is again probably because the element lengths are off.  I'm not sure how Matlab is managing to cope with that, but that is a curiosity.  I think we should mark this as "Won't Fix" since it is some strangeness is the EFM tool code.


Rik <rik5>
Group administrator
Wed 25 May 2016 11:23:09 PM UTC, comment #7: 

I was curious so I worked on this for a half hour. Here's what I've found as to the cause of this error.

Both files contain zlib-compressed data. The mat v5 file specification has a size field outside of the compression, and then a size of the data element inside of the compression.

The file produced by Matlab is entirely consistent. The file size is 1176 bytes, 136 bytes of standard file header, and 1040 bytes of zlib-compressed data. Uncompressing 1040 bytes results in exactly 6896 bytes, which is also exactly what the value of the size field is in the first few bytes of the compressed block.

The file produced by EFMtool uncompresses into more data than it claims should be there. The file is 1362 bytes, again 136 bytes of header and 1226 bytes of zlib-compressed data. Uncompressing the trailing 1226 bytes yields 14,152 bytes. However, the data size field in the first few bytes of the compressed block say the data should be 13,448 bytes.

This doesn't seem like an error on the surface, but I'm guessing that Octave attempts to call uncompress of 14,152 bytes into a buffer of length 13,448, and gets the Z_BUF_ERROR return value because the output buffer wasn't large enough to finish decompressing.

Mike Miller <mtmiller>
Group Member
Wed 25 May 2016 10:47:16 PM UTC, comment #6: 

I should add that we've been discussing this on #octave. It's pretty low priority, more of a curiosity. Yes, the original big-endian file is produced by the non-Matlab Java software. The files it generates are able to be read by Matlab but not by Octave. If no one is interested in investigating, it can't be determined what's wrong with the file, or if it can be blamed on the other software, then this can certainly be closed Won't Fix.

Mike Miller <mtmiller>
Group Member
Wed 25 May 2016 10:44:44 PM UTC, comment #5: 

Nevermind, I see the point of the error is when uncompressing part of the file after reading the length from the same file. So the endianness may indeed be an issue giving a wrong value for the size of the compressed data element.

Mike Miller <mtmiller>
Group Member
Wed 25 May 2016 10:38:54 PM UTC, comment #4: 

The new little endian version of the file loads fine for me.

Was the original file generated by Matlab, or was it generated outside of Matlab in Java by the efmtool?  It might be that the non-Matlab software doesn't truly have the file format correct.  When you load it in to Matlab and re-save it then it gets written out in a completely correct format.

Rik <rik5>
Group administrator
Wed 25 May 2016 10:29:43 PM UTC, comment #3: 

Here attached is the same .mat file after loading it in Matlab and saving it back to the supposedly same .mat format. It gives no issues when loading it in Octave

(file #37264)

Anonymous
Wed 25 May 2016 10:20:23 PM UTC, comment #2: 

If there were simply a problem with endianness, wouldn't invalid data be read successfully? This seems more like a field that we are expecting to read isn't long enough or is missing in the file.

The Z_BUF_ERROR error code means "there was not enough room in the output buffer" according to http://www.zlib.net/manual.html.

Mike Miller <mtmiller>
Group Member
Wed 25 May 2016 09:57:54 PM UTC, comment #1: 

When you load the mat file in Matlab, are there any special variables types it is using: tables, structs, cell arrays, etc.

Using the Unix utility 'file' shows something interesting


file efms_0.mat
efms_0.mat: Matlab v5 mat-file (big endian) version 0x0100


My computer, which uses an Intel CPU, reports "little endian" for any .mat file I create with Octave.  If there isn't a special variable type being used, then it might be that the .mat file was created on a big endian machine and is trying to be read on a little endian one.  Of course, that should be possible, but it might be an indication of where to look.

Rik <rik5>
Group administrator
Wed 25 May 2016 09:41:33 PM UTC, original submission:  

The attached .mat file produced by EFMtool [1] cannot be imported in Octave:


octave>load('efms_0.mat')
error: load: error uncompressing data element (buf error from zlib)


Installation details:
Octave 4.0.2 from Debian testing amd64
zlib 1.2.8


[1] http://www.csb.ethz.ch/tools/software/efmtool.html

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #37264:  efms_01.mat added by None (1KiB - application/octet-stream)
file #37263:  efms_0.mat added by None (1KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-05-26 mtmiller StatusNone Wont Fix
        Open/ClosedOpen Closed
    2016-05-25 mtmiller Severity3 - Normal 2 - Minor
        Priority5 - Normal 3 - Low
    2016-05-25 None Attached File- Added efms_01.mat, #37264
    2016-05-25 None Attached File- Added efms_0.mat, #37263

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code