bugGNU Octave - Bugs: bug #42562, 3.8.1 can't load mat file...

 
 

bug #42562: 3.8.1 can't load mat file (>maltab r2010b)

Submitter:  Juan Pablo Carbajal <juanpi>
Submitted:  Sun 15 Jun 2014 08:02:55 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Matlab Compatibility
Status:  Fixed Assigned to:  None
Originator Name:  juanpi Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 16 Jun 2014 09:11:14 PM UTC, comment #20: 

I see I typed to fast and got row and column vectors reversed in the last post, but the answer still stands.  You can use either one for export.

Rik <rik5>
Group administrator
Mon 16 Jun 2014 09:10:03 PM UTC, comment #19: 

Q1 : It's really up to you whether you prefer row (Nx1) or column (1xN) vectors.  Octave/Matlab makes the distinction because you might want to form the Matrix/Vector product

A*x

which will only work if x is oriented correctly.

Q2 : Correct.  We've worked around the 1-D case and it should be part of our 3.8.2 bug fix release in a few weeks so there is no longer any reason for you to hurry a patch into production.

Rik <rik5>
Group administrator
Mon 16 Jun 2014 08:50:32 PM UTC, comment #18: 

Author of R.matlab here:

Thanks for reporting on this out.  I take it as R.matlab::writeMat() incorrectly writes R vectors as one-dimensional arrays (i.e. the dimension array subelement contains a single value) whereas the MAT file format requires at least two elements.  I must admit that I've missed:

"Dimensions Array Subelement
This subelement specifies the size of each dimension of an n-dimensional array in an n-sized array of 32-bit values (miINT32). All numeric arrays have at least two dimensions. The Dimensions Array subelement is common to all MATLAB array types." (page 1-17 in http://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf)


Q1. Should R vectors (=one dimensional arrays) be written as nx1 or 1xn arrays?

Q2. Should I read this thread as you indeed updated Octave to workaround the one-dimensional case?

/Henrik

PS. It'll probably be a while before I fix this in R.matlab, because I have to careful about cases related to automatic translation/coercing of vectors-to-matrices in writeMat() and readMat().

Henrik Bengtsson <henrikb>
Mon 16 Jun 2014 04:11:59 PM UTC, comment #17: 

Rik,

Thank you.

I think the point is that even this "malformed" objects are read in by matlab, as you said. But they weren't loaded by GNU Octave.

I will tell the community of R (which seems to have problems loading empty objects as 0x0 matrices), but I guess their answer will be something like: "matlab and scipy load these files without problems."

Thanks again.

Juan Pablo Carbajal <juanpi>
Group Member
Mon 16 Jun 2014 03:56:26 PM UTC, comment #16: 

This is really a problem with R and Python.  Matlab objects are always at least 2-D.  Even an empty object is implemented as a 2-D 0x0 object.  Hence, when the file is read in by Matlab and written back out again it generates a proper scalar object which is 2-D 1x1.

We can help out our open source friends though since the code change is minimal.  I made the change on the stable branch in this changeset (http://hg.savannah.gnu.org/hgweb/octave/rev/94d791a03737).  It will be part of the 3.8.2 bug fix release.

Rik <rik5>
Group administrator
Mon 16 Jun 2014 11:49:02 AM UTC, comment #15: 

I implemented the suggestion from anonymous.

The attached patch should solve the problem.

Please check and close the bug.

Thanks


(file #31565)

Juan Pablo Carbajal <juanpi>
Group Member
Mon 16 Jun 2014 08:44:55 AM UTC, comment #14: 

Juan Pablo,

a possible shorter version of your patch.


diff --git a/libinterp/corefcn/ls-mat5.cc b/libinterp/corefcn/ls-mat5.cc
--- a/libinterp/corefcn/ls-mat5.cc
+++ b/libinterp/corefcn/ls-mat5.cc
@@ -668,6 +668,7 @@

       int ndims = dim_len / 4;
       dims.resize (ndims);
+      dims(1) = 1;           // dims.length () is always min. 2, so set n x 1 if ndims == 1
       for (int i = 0; i < ndims; i++)
         {
           int32_t n;


Anonymous
Mon 16 Jun 2014 03:02:05 AM UTC, comment #13: 

To find out ndims directly from a dim_vector it is usually sufficient to take the length of the dim_vector.  This will not tell you whether one of the dimensions is zero though so a 1x0 object still has 2 dimensions.


dim_vector dv;
ndims = dv.length ();


Rik <rik5>
Group administrator
Mon 16 Jun 2014 02:13:45 AM UTC, comment #12: 

The attached change set fixes the problem, but is not elegant.

I have tested it with row vectors, columns, matrices and NDarrays created in R.

Once the dim_vector is resized (line 670 ls-mat5.cc), the loop should use the information stored inside it (since when resized it is check that if dims < 2 then dim == 2), but I couldn't find the proper way of doing it.

How can one get ndims from dim_vector directly?

Thanks


(file #31561)

Juan Pablo Carbajal <juanpi>
Group Member
Mon 16 Jun 2014 01:36:19 AM UTC, comment #11: 

OK, I think I found the problem.

I created a file as explained below using R.


hexdump -C test.mat
00000000  4d 41 54 4c 41 42 20 35  2e 30 20 4d 41 54 2d 66  |MATLAB 5.0 MAT-f|
00000010  69 6c 65 2c 20 50 6c 61  74 66 6f 72 6d 3a 20 75  |ile, Platform: u|
00000020  6e 69 78 2c 20 53 6f 66  74 77 61 72 65 3a 20 52  |nix, Software: R|
00000030  20 76 33 2e 30 2e 32 2c  20 43 72 65 61 74 65 64  | v3.0.2, Created|
00000040  20 6f 6e 3a 20 4d 6f 6e  20 4a 75 6e 20 31 36 20  | on: Mon Jun 16 |
00000050  30 33 3a 30 32 3a 33 31  20 32 30 31 34 20 20 20  |03:02:31 2014   |
00000060  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |                |
00000070  20 20 20 20 20 20 20 20  20 20 20 20 00 01 49 4d  |            ..IM|
00000080  0e 00 00 00 40 00 00 00  06 00 00 00 08 00 00 00  |....@...........|
00000090  06 00 00 00 00 00 00 00  05 00 00 00 04 00 00 00  |................|
000000a0  01 00 00 00 00 00 00 00  01 00 00 00 07 00 00 00  |................|
000000b0  6c 61 74 65 6e 63 79 00  09 00 00 00 08 00 00 00  |latency.........|
000000c0  00 00 00 00 00 00 14 40                           |.......@|
000000c8


As anonymous pointed out the tag for dimension of the data element is in 0x90

05 00 00 00 04 00 00 00

According to the format specification (note on page 1-17) the 04 means there is a single dimension (1x1 not 1x0). This is correctly calculated in line 669 of ls-mat5.cc. However in the following for loop the index_type dims is not fully fill and is left as 1x0.

This is a miss interpretation of the format specification. If the number of dimensions is 1 then the data should be seen as 1x1.

The fix of anonymous works cause ndims is 2 in that case and each value 1, so the resulting index_type dims is 1x1.

Is this correct?

I arrive to this conclusion by looking at the actual creation of the data container in octave


1418            NDArray re (dims);
...
1430                octave_idx_type n = re.numel ();
(gdb) p re
$21 = {<MArray<double>> = {<Array<double>> = {
      _vptr.Array = 0x7ffff7d18150 <vtable for NDArray+16>,
      dimensions = {rep = 0x7c6548}, rep = 0x7c65e0,
      slice_data = 0x7ce750,
      slice_len = 0}, <No data fields>}, <No data fields>}



Juan Pablo Carbajal <juanpi>
Group Member
Mon 16 Jun 2014 01:02:23 AM UTC, comment #10: 

Can you produce a version of the file that was written from Matlab and is readable by Octave?

It would then be easy to do a binary difference between the working and non-working files to find out which field code is being written differently by R.

Rik <rik5>
Group administrator
Mon 16 Jun 2014 12:21:09 AM UTC, comment #9: 

Well, python can also read it and matlab itself can also read it.

I think we better read these files!

To me it seems that read_mat5_tag in libinter/corefun/ls-mat5.cc is not implementing the mat-file format correctly. Although we haven't put the finger on it yet.

The format specification is here
http://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf

page 1-8.

Juan Pablo Carbajal <juanpi>
Group Member
Mon 16 Jun 2014 12:09:10 AM UTC, comment #8: 

According to later entries in the thread, Octave can read the files if they were created with Matlab.  It's only R's version of matlab V5 that seems to be a problem.

Rik <rik5>
Group administrator
Mon 16 Jun 2014 12:05:29 AM UTC, comment #7: 

I am provinding the file created with R for easier debugging.

@Kai, could you provide the file created with matlab? Just save a variable "latency" with a small value in it.


(file #31560)

Juan Pablo Carbajal <juanpi>
Group Member
Sun 15 Jun 2014 11:46:19 PM UTC, comment #6: 

This are the last bytes from the .mat file


hexdump -C sample_clip.mat
...
0000d1c0  0e 00 00 00 40 00 00 00  06 00 00 00 08 00 00 00  |....@...........|
0000d1d0  06 00 00 00 00 00 00 00  05 00 00 00 04 00 00 00  |................|
0000d1e0  01 00 00 00 00 00 00 00  01 00 00 00 07 00 00 00  |................|
0000d1f0  6c 61 74 65 6e 63 79 00  09 00 00 00 08 00 00 00  |latency.........|
0000d200  00 00 00 00 00 00 00 00                           |........|


The tag from position 0xd1d8 is 5,4,1,0

If this is changed to 5,8,1,1 'latency' can be read correctly.

I assume if there's only one value to read, the matrix size is 1x1

Anonymous
Sun 15 Jun 2014 11:18:19 PM UTC, comment #5: 

Indeed, the following lines in R generate the problem

in R:

install.packages("R.matlab")
library(R.matlab)
latency <- 0
writeMat("test.mat",latency=latency)


in Octave (dev)

load test.mat
whos
Variables in the current scope:

   Attr Name         Size                     Bytes  Class
   ==== ====         ====                     =====  =====
        ans          1x20                        20  char
        latency      1x0                          0  double

Total is 20 elements using 20 bytes


The file test.m is incorrectly read in Octave. However python reads it without a problem.

In python:
+veratim+
from scipy.io import loadmat
loadmat("test.mat")
{'__globals__': [],
 '__header__': 'MATLAB 5.0 MAT-file, Platform: unix, Software: R v3.0.2, Created on: Mon Jun 16 01:12:34 2014',
 '__version__': '1.0',
 'latency': array([ 0.])}
-verbatim-


Juan Pablo Carbajal <juanpi>
Group Member
Sun 15 Jun 2014 11:07:15 PM UTC, comment #4: 

I did not created this file.
This are the files form the Kaggle constest.

Maybe the file was created with R?
I will check

Juan Pablo Carbajal <juanpi>
Group Member
Sun 15 Jun 2014 11:05:40 PM UTC, comment #3: 

I can confirm the problem for the M-File you provided. But maybe your M-File is somehow broken. I loaded and saved your file with my MATLAB R2013b and Octave loaded the new one without problems.

The new document header becomes:

MATLAB 5.0 MAT-file, Platform: GLNXA64, Created on: Mon Jun 16 00:59:55 2014

What does "Software R" mean? That's the observed header difference. Do you use any plugins, toolboxes with your MATLAB that overwrites the save function?

Kai Torben Ohlhus <siko1056>
Group Member
Sun 15 Jun 2014 08:53:59 PM UTC, comment #2: 

Trying to re-attach file

(file #31558)

Juan Pablo Carbajal <juanpi>
Group Member
Sun 15 Jun 2014 08:14:36 PM UTC, comment #1: 

Using python module scipy.io, the function loadmat says that the data is has the header

'MATLAB 5.0 MAT-file, Platform: unix, Software: R v3.0.2, Created on: Mon Apr  7 22:05:57 2014'

It seems that octave can't read matfiles v5? I tried option -v4 but it fails. Options -v6 and -v7 read the file incorrectly.

Juan Pablo Carbajal <juanpi>
Group Member
Sun 15 Jun 2014 08:02:55 PM UTC, original submission:  

A matfile saved with a version newer than r2010b of matlab can't be loaded properly in octave.

Try to load the attached file to observe the problem.
The variables "freq" and "latency" should contain a double (399.6098 and 0 respectively) but are loaded empty.

This problem is preventing several users to use Octave for a competition at Kaggle.
http://www.kaggle.com/c/seizure-detection/forums/t/8256/problems-with-freq-and-latency-in-octave-3-8-1/49137#post49137

Juan Pablo Carbajal <juanpi>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #31565:  mat5.diff added by juanpi (627B - text/x-patch)
file #31561:  mat5.diff added by juanpi (1KiB - text/x-patch)
file #31560:  test.mat added by juanpi (200B - application/octet-stream)
file #31558:  sample_clip.mat added by juanpi (53KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by henrikb (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by juanpi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-06-16 rik5 StatusNone Fixed
        Open/ClosedOpen Closed
    2014-06-16 juanpi Attached File- Added mat5.diff, #31565
    2014-06-16 juanpi Attached File- Added mat5.diff, #31561
    2014-06-16 juanpi Attached File- Added test.mat, #31560
    2014-06-15 juanpi Attached File- Added sample_clip.mat, #31558

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code