bugGNU Octave - Bugs: bug #42562, 3.8.1 can't load mat file...

 
 

bug #42562: 3.8.1 can't load mat file (>maltab r2010b)

Submitted by:  Juan Pablo Carbajal <juanpi>
Submitted on:  Sun 15 Jun 2014 08:02:55 PM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Matlab Compatibility
Status: FixedAssigned to: None
Originator Name: juanpiOpen/Closed: Closed
Release: devOperating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon 16 Jun 2014 09:11:14 PM UTC, comment #20:

I see I typed to fast and got row and column vectors reversed in the last post, but the answer still stands. You can use either one for export.

Rik <rik5>
Project Administrator
Mon 16 Jun 2014 09:10:03 PM UTC, comment #19:

Q1 : It's really up to you whether you prefer row (Nx1) or column (1xN) vectors. Octave/Matlab makes the distinction because you might want to form the Matrix/Vector product
which will only work if x is oriented correctly.

Q2 : Correct. We've worked around the 1-D case and it should be part of our 3.8.2 bug fix release in a few weeks so there is no longer any reason for you to hurry a patch into production.

Rik <rik5>
Project Administrator
Mon 16 Jun 2014 08:50:32 PM UTC, comment #18:

Author of R.matlab here:

Thanks for reporting on this out. I take it as R.matlab::writeMat() incorrectly writes R vectors as one-dimensional arrays (i.e. the dimension array subelement contains a single value) whereas the MAT file format requires at least two elements. I must admit that I've missed:

"Dimensions Array Subelement
This subelement specifies the size of each dimension of an n-dimensional array in an n-sized array of 32-bit values (miINT32). All numeric arrays have at least two dimensions. The Dimensions Array subelement is common to all MATLAB array types." (page 1-17 in http://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf)

Q1. Should R vectors (=one dimensional arrays) be written as nx1 or 1xn arrays?

Q2. Should I read this thread as you indeed updated Octave to workaround the one-dimensional case?

/Henrik

PS. It'll probably be a while before I fix this in R.matlab, because I have to careful about cases related to automatic translation/coercing of vectors-to-matrices in writeMat() and readMat().

Henrik Bengtsson <henrikb>
Mon 16 Jun 2014 04:11:59 PM UTC, comment #17:

Rik,

Thank you.

I think the point is that even this "malformed" objects are read in by matlab, as you said. But they weren't loaded by GNU Octave.

I will tell the community of R (which seems to have problems loading empty objects as 0x0 matrices), but I guess their answer will be something like: "matlab and scipy load these files without problems."

Thanks again.

Juan Pablo Carbajal <juanpi>
Project Member
Mon 16 Jun 2014 03:56:26 PM UTC, comment #16:

This is really a problem with R and Python. Matlab objects are always at least 2-D. Even an empty object is implemented as a 2-D 0x0 object. Hence, when the file is read in by Matlab and written back out again it generates a proper scalar object which is 2-D 1x1.

We can help out our open source friends though since the code change is minimal. I made the change on the stable branch in this changeset (http://hg.savannah.gnu.org/hgweb/octave/rev/94d791a03737). It will be part of the 3.8.2 bug fix release.

Rik <rik5>
Project Administrator
Mon 16 Jun 2014 11:49:02 AM UTC, comment #15:

I implemented the suggestion from anonymous.

The attached patch should solve the problem.

Please check and close the bug.

Thanks

(file #31565)

Juan Pablo Carbajal <juanpi>
Project Member
Mon 16 Jun 2014 08:44:55 AM UTC, comment #14:

Juan Pablo,

a possible shorter version of your patch.

Anonymous
Mon 16 Jun 2014 03:02:05 AM UTC, comment #13:

To find out ndims directly from a dim_vector it is usually sufficient to take the length of the dim_vector. This will not tell you whether one of the dimensions is zero though so a 1x0 object still has 2 dimensions.

Rik <rik5>
Project Administrator
Mon 16 Jun 2014 02:13:45 AM UTC, comment #12:

The attached change set fixes the problem, but is not elegant.

I have tested it with row vectors, columns, matrices and NDarrays created in R.

Once the dim_vector is resized (line 670 ls-mat5.cc), the loop should use the information stored inside it (since when resized it is check that if dims < 2 then dim == 2), but I couldn't find the proper way of doing it.

How can one get ndims from dim_vector directly?

Thanks

(file #31561)

Juan Pablo Carbajal <juanpi>
Project Member
Mon 16 Jun 2014 01:36:19 AM UTC, comment #11:

OK, I think I found the problem.

I created a file as explained below using R.

As anonymous pointed out the tag for dimension of the data element is in 0x90
According to the format specification (note on page 1-17) the 04 means there is a single dimension (1x1 not 1x0). This is correctly calculated in line 669 of ls-mat5.cc. However in the following for loop the index_type dims is not fully fill and is left as 1x0.

This is a miss interpretation of the format specification. If the number of dimensions is 1 then the data should be seen as 1x1.

The fix of anonymous works cause ndims is 2 in that case and each value 1, so the resulting index_type dims is 1x1.

Is this correct?

I arrive to this conclusion by looking at the actual creation of the data container in octave

Juan Pablo Carbajal <juanpi>
Project Member
Mon 16 Jun 2014 01:02:23 AM UTC, comment #10:

Can you produce a version of the file that was written from Matlab and is readable by Octave?

It would then be easy to do a binary difference between the working and non-working files to find out which field code is being written differently by R.

Rik <rik5>
Project Administrator
Mon 16 Jun 2014 12:21:09 AM UTC, comment #9:

Well, python can also read it and matlab itself can also read it.

I think we better read these files!

To me it seems that read_mat5_tag in libinter/corefun/ls-mat5.cc is not implementing the mat-file format correctly. Although we haven't put the finger on it yet.

The format specification is here
http://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf

page 1-8.

Juan Pablo Carbajal <juanpi>
Project Member
Mon 16 Jun 2014 12:09:10 AM UTC, comment #8:

According to later entries in the thread, Octave can read the files if they were created with Matlab. It's only R's version of matlab V5 that seems to be a problem.

Rik <rik5>
Project Administrator
Mon 16 Jun 2014 12:05:29 AM UTC, comment #7:

I am provinding the file created with R for easier debugging.

@Kai, could you provide the file created with matlab? Just save a variable "latency" with a small value in it.

(file #31560)

Juan Pablo Carbajal <juanpi>
Project Member
Sun 15 Jun 2014 11:46:19 PM UTC, comment #6:

This are the last bytes from the .mat file

The tag from position 0xd1d8 is 5,4,1,0

If this is changed to 5,8,1,1 'latency' can be read correctly.

I assume if there's only one value to read, the matrix size is 1x1

Anonymous
Sun 15 Jun 2014 11:18:19 PM UTC, comment #5:

Indeed, the following lines in R generate the problem

in R:

in Octave (dev)

The file test.m is incorrectly read in Octave. However python reads it without a problem.

In python:
+veratim+
from scipy.io import loadmat
loadmat("test.mat")
{'__globals__': [],
'__header__': 'MATLAB 5.0 MAT-file, Platform: unix, Software: R v3.0.2, Created on: Mon Jun 16 01:12:34 2014',
'__version__': '1.0',
'latency': array([ 0.])}
-verbatim-

Juan Pablo Carbajal <juanpi>
Project Member
Sun 15 Jun 2014 11:07:15 PM UTC, comment #4:

I did not created this file.
This are the files form the Kaggle constest.

Maybe the file was created with R?
I will check

Juan Pablo Carbajal <juanpi>
Project Member
Sun 15 Jun 2014 11:05:40 PM UTC, comment #3:

I can confirm the problem for the M-File you provided. But maybe your M-File is somehow broken. I loaded and saved your file with my MATLAB R2013b and Octave loaded the new one without problems.

The new document header becomes:

MATLAB 5.0 MAT-file, Platform: GLNXA64, Created on: Mon Jun 16 00:59:55 2014

What does "Software R" mean? That's the observed header difference. Do you use any plugins, toolboxes with your MATLAB that overwrites the save function?

Kai Torben Ohlhus <siko1056>
Project Member
Sun 15 Jun 2014 08:53:59 PM UTC, comment #2:

Trying to re-attach file

(file #31558)

Juan Pablo Carbajal <juanpi>
Project Member
Sun 15 Jun 2014 08:14:36 PM UTC, comment #1:

Using python module scipy.io, the function loadmat says that the data is has the header

'MATLAB 5.0 MAT-file, Platform: unix, Software: R v3.0.2, Created on: Mon Apr 7 22:05:57 2014'

It seems that octave can't read matfiles v5? I tried option -v4 but it fails. Options -v6 and -v7 read the file incorrectly.

Juan Pablo Carbajal <juanpi>
Project Member
Sun 15 Jun 2014 08:02:55 PM UTC, original submission:

A matfile saved with a version newer than r2010b of matlab can't be loaded properly in octave.

Try to load the attached file to observe the problem.
The variables "freq" and "latency" should contain a double (399.6098 and 0 respectively) but are loaded empty.

This problem is preventing several users to use Octave for a competition at Kaggle.
http://www.kaggle.com/c/seizure-detection/forums/t/8256/problems-with-freq-and-latency-in-octave-3-8-1/49137#post49137

Juan Pablo Carbajal <juanpi>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #31565:  mat5.diff added by juanpi (627B - text/x-patch)
file #31561:  mat5.diff added by juanpi (1KiB - text/x-patch)
file #31560:  test.mat added by juanpi (200B - application/octet-stream)
file #31558:  sample_clip.mat added by juanpi (53KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by henrikb (Posted a comment)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by siko1056 (Posted a comment)
  • -unavailable- added by juanpi (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 16 Jun 2014 03:56:26 PM UTCrik5StatusNone=>Fixed
      Open/ClosedOpen=>Closed
    Mon 16 Jun 2014 11:49:02 AM UTCjuanpiAttached File-=>Added mat5.diff, #31565
    Mon 16 Jun 2014 02:13:45 AM UTCjuanpiAttached File-=>Added mat5.diff, #31561
    Mon 16 Jun 2014 12:05:29 AM UTCjuanpiAttached File-=>Added test.mat, #31560
    Sun 15 Jun 2014 08:53:59 PM UTCjuanpiAttached File-=>Added sample_clip.mat, #31558

    Back to the top


    Powered by Savane 3.1-cleanup1