Mon 14 Nov 2011 08:10:21 AM UTC, comment #9:
Hi,
I am trying to load a fairly large .mat file created by Matlab and I'm getting a lot of errors like those shown in the original post here.
using octave 3.4.0 and java-1.2.8
eg.
error: load: invalid element type = 0
zero is very common 65535 also and raft of less frequent values.
Do these numbers actually represent something related to "data element type" or are they spurious values output because octave is incorrectly parsing the data?
ie does this mean I have a dozen formats that are not supported or probably just one ?
If it's just one I may have a go at fixing it or getting the data in an acceptable format.
Shouldn't Octave be trapping this a bit better?
|
Wed 12 May 2010 10:43:34 AM UTC, comment #8:
Great David, thanks a lot!
|
Wed 12 May 2010 12:21:46 AM UTC, comment #7:
There was a minor typo in the last changeset. You'll also need the changeset
http://hg.savannah.gnu.org/hgweb/octave/rev/3eafa521550b
D.
|
Wed 12 May 2010 12:11:24 AM UTC, comment #6:
As I suspected the length of the sparse data element of a mat-file was miscalculated when it was written in the following two cases
1) When the value of NNZ/NZMAX was 1. In this case the length was overestimated by 4 bytes
2) When the number of columns in the sparse matrix was zero. In this case the length of the data element was overestimated by 4 bytes..
and the bug was only evident when the sparse data element wasn't the last element in the file. These cases are relative rare in practice which is why this issue hadn't been spotted yet.
Also only the NNZ elements of the index vector ridx were being written to the file, and in fact now that NNZ and NZMAX are not equal in Octave the NZMAX elements of ridx should have been written. This however was a separate bug.
The changeset
http://hg.savannah.gnu.org/hgweb/octave/rev/0e98fb2068fc
addresses this bug. Thanks for the report
D.
|
Mon 10 May 2010 04:22:06 PM UTC, comment #5:
I checked Octave's behavior on a 32-bit machine and it also won't save sparse matrices correctly. Thus, my supposition about a difference between 32-bit and 64-bit ints was incorrect.
|
Mon 10 May 2010 12:35:48 PM UTC, comment #4:
The matlab file format is ugly in that the size of ech block of data is stored at the start of the block. However to calculate the size of the block you basically have to write the file and figure out the size post-hoc. This leaves two choices. Firstly you write the block once to a temporary location, calculate its size from there and then write the size to the correct location. Or secondly, you write a blank to the block size, write the data to the file, then use fseek to come back and correct the block size. Octave chooses the second option for performance reasons.
However, if the size of the block calculation is incorrect, then the seeks will be misperformed and the data of the following variables will be corruptted. I strongly suspect this is what is happening. I'm however surprised that I never noticed this issue.
I'll take this one on later in the week
D.
|
Sat 08 May 2010 07:06:53 PM UTC, comment #3:
Thanks Rik for investigating this issue further and narrowing it down to the sparse matrix. Regarding the computing platform, yes indeed, I'm on a x86_64 Linux machine.
|
Sat 08 May 2010 02:20:47 PM UTC, comment #2:
Apparently Octave has trouble saving sparse matrices and has for quite a while.
I spent some time narrowing the problem down. The following code is sufficient to provoke an error.
a = sparse(1,1);
a(1,1) = 1;
b = 2;
save -mat test.mat
clear
load test.mat
The error is reproducible with the current development sources as well as an ancient 3.0.5 version I tried.
During the load the error message is:
error: load: invalid element type = 1016
I suspect that binary structures are not being properly aligned during the save process. For instance, if the variables are reversed so that the sparse matrix is called b and the double variable is a then everything works. This puts the sparse matrix at the end of the file where it can't collide with any following variables. Similarly, if the sparse matrix is empty the test passes.
These results were obtained on an x86_64 machine. The original reporter didn't note whether they were using a 64 bit machine but it could be something as simple as using an int value and expecting that to be a defined size rather than using named sizes such as int32_t.
|
Thu 06 May 2010 08:16:39 AM UTC, comment #1:
As noted in my intial bug report, Matlab is able to load the data structure generated in Octave, while Octave can not read it anymore. However, looking at the data structure loaded in Matlab I noticed that the data in the struct array is being shifted around. First entry:
matchdata(1)
ans =
matchloc1: [219 313]
matchloc2: [267 87]
nbmatches: 1
votecenter: [81 278]
voteloc1: [231 307]
voteloc2: [279 81]
votemat: [884x1047 double]
Entry 200:
matchdata(200)
ans =
matchloc1: [884x1047 double]
matchloc2: [2x2 double]
nbmatches: [2x2 double]
votecenter: 2
voteloc1: [250 203]
voteloc2: [2x2 double]
votemat: [2x2 double]
I tested further and found when storing in Octave's native binary format (-binary), this bug does not occur. However, if data is stored in either the v6 or v7 Matlab format, the stored data format seems to be incorrect.
I attach a simple a file (testvotedata.m) that allows to generate faulty data with the option v6 (outtype 2) and v7 (outtype 1) and correct data with -binary (outtype 3).
(file #20464)
|
Wed 05 May 2010 12:05:44 PM UTC, original submission:
Octave can not load the attached data file (generated in Octave via save('-v7',mdname,'matchdata');
). Matlab (R2009a) on the other hand is able to do so.
load voteresults_01.mat in Matlab works. After loading we find the following data structure:
matchdata =
1x1256 struct array with fields:
matchloc1
matchloc2
nbmatches
votecenter
voteloc1
voteloc2
votemat
Note that votemat are sparse matrices.
In Octave (v 3.0.5 and v 3.2.3) I get the following behavior:
>> load voteresults_01.mat
error: load: invalid element type = 96
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 8
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 0
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 8
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 2
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 0
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 48
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 32768
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 0
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 0
error: load: trouble reading binary file `voteresults_01.mat'
etc
|