Thu 16 May 2013 07:48:53 PM UTC, comment #16:
Thanks, I pushed your changeset so savannah.
Yes, it would be great to have some test files for the load/save code. Preferably they should not be too large since we will be distributing them with every copy of the Octave sources.
|
Mon 06 May 2013 07:10:53 PM UTC, comment #15:
The commit message of the attachment should now be within the guidelines. I'll see if I write a test later on.
(file #28034)
|
Mon 06 May 2013 01:58:00 PM UTC, comment #14:
As for testing, the current way to test I/O is to write to a temporary file and then check that that file is read. You might have to use fwrite to write this temporary file. Here is an example:
http://hg.savannah.gnu.org/hgweb/octave/file/38fef1e833ea/libinterp/corefcn/dlmread.cc#l481
This test should ideally go right after the DEFUN here:
http://hg.savannah.gnu.org/hgweb/octave/file/38fef1e833ea/libinterp/interpfcn/load-save.cc#l545
|
Mon 06 May 2013 01:50:34 PM UTC, comment #13:
Thanks, your patch looks good.
Would it be too much of a burden to amend it to include a commit message according to our guidelines?
http://wiki.octave.org/Commit_message_guidelines
Do "hg log -v" to see an example of other commit messages. If you're standing on your current patch, (if not, you can go to your current patch with "hg up -red31dac4e36"), you can do "hg commit --amend" to edit your commit message there accordingly.
If you find this too troublesome, let us know, and I can edit your commit message myself.
|
Sun 05 May 2013 07:55:38 PM UTC, comment #12:
I wrote a patch which basically communicates back whether the data is actually written in small data element format or not when reading a tag. That information is then used to handle padding. I.e., the patch does not assume that all small data elements are written as small data elements. I didn't change the write part so there is now READ_PAD (which takes the info of whether small element format is used) and PAD (the original).
This works correctly with both results.mat, results2.mat and Octave save + loads for matrices, cells containing stuff, char-arrays and sparse matrices. I didn't test objects, functions nor workspaces or different types (which int and float/double) of storage - just how Octave decided to store my tries. Is there a set of .mat -files which have been used in testing?
The mercurial changeset is included. Is there a correct way to get it forward?
(file #28025)
|
Fri 03 May 2013 09:43:05 PM UTC, comment #11:
Good point: small data elements (i.e. data included in tag) are the norm as written by Matlab. That compatability should not be broken.
The MAT-file format actually declares small data elements as optional. Octave just doesn't support them as normal data elements and does not even detect the situation and therefore reads in bogus data.
The in-lab code can be changed. I made a quick test and (1) four letter name gave incorrect read, (2) five letter name gave correct read, (3) writing in small data element format gave correct read. To me this is enough evidence.
I'll see if I can make Octave at least to report the standard format as unsupported, if not read it.
The 'results2.mat' is attached.
Thanks for help so far and for the repository tip. Checking it out at the moment.
(file #28012)
|
Fri 03 May 2013 07:44:11 PM UTC, comment #10:
I think you're on the right track about the short variable names. However, the 4-byte padding for short names is correct, at least in some of my v5 mat files that were written by Matlab.
Perhaps Matlab is flexible in how it reads the type element following the name, so it is permissive enough to read in your badly formatted file.
Can your in-laboratory program be modified to pad short variable names to only 4 bytes instead of 8? Or to ensure all variable names are longer than 4 characters?
Can you attach the file you get if you load this dataset into Matlab and then save it in v5 format?
|
Fri 03 May 2013 07:25:20 PM UTC, comment #9:
If you want to get a local copy and build and test with Octave, you'll have to install Mercurial and clone it:
hg clone http://hg.savannah.gnu.org/hgweb/octave/
If you just want a tarball of the current dev version, you can get it here:
http://hg.savannah.gnu.org/hgweb/octave/archive/tip.tar.gz
This is currently missing the gnulib subrepo that you can get here:
http://hg.savannah.gnu.org/hgweb/octave/gnulib-hg/archive/tip.tar.gz
|
Fri 03 May 2013 07:02:18 PM UTC, comment #8:
The program is an in-laboratory code, which is not distributed to any other place. In such, the trouble is local. I think I am the first one who has tried to use the output in Octave.
However, I noticed that xxxxxxx and yyyyyy variables had no trouble loading correctly all the time. The trouble is with the single letter names.
I went through the ls-mat5.cc and I think the problem is line
696 (reading variable name):
is.seekg (tmp_pos + static_cast<std::streamoff> (PAD (len)));
PAD is defined as (len > 0 && len <= 4) ? 4 : ((l+7)/8)8; For the 1 char variables this gives padding of 4 and not proper 7. The padding should here be just ((l+7)/8)8 without the iffing (and probably in a few other places where small data fragments might occur).
This also explains why the parser sees the "type" as 0 (padding) instead of 9 (and the 9 will become data length).
Could either of you check this out? I don't have source code of Octave nor idea how to get a version.
|
Fri 03 May 2013 05:17:24 PM UTC, comment #7:
I looked at this file a bit too. I think the problem may be that the "type" codes for the matrices in the file are set to 0 instead of miDOUBLE. This does not seem like a valid format, at least according to the parser in Octave. The case statements in ls-mat5.cc simply fall through and don't read any data into the uninitialized matrices, which is probably why they are appearing as random values.
I inserted something like
somewhere around here:
http://hg.savannah.gnu.org/hgweb/octave/file/6698d0c42fbf/libinterp/interp-core/ls-mat5.cc#l1395
and get consistent values every time.
|
Fri 03 May 2013 04:07:05 PM UTC, comment #6:
I am inspecting the file and I would like to know which program generated it, because it doesn't look completely like a valid file. I cannot diagnose it in detail yet, but I suspect that this file has the wrong size for one of its parameters. I suspect this is exposing both a bug in the program that generated this file and in Octave, plus undocumented behaviour in Matlab.
Note that Matlab has some quirks that are not in the published spec. I have run into some of them when debugging mat files. It is entirely possible that the program that generated this file is exploiting some of these hidden quirks in Matlab, which I expect we will now have to reproduce in Octave.
|
Fri 03 May 2013 02:03:43 PM UTC, comment #5:
It is supposed to be a valid MAT -file, but the best I can say is that Matlab reads it fine and as expected. In any case Octave should not give different results from run to run since the original file is the same. That is, Octave should fail deterministically if it fails to read the file.
|
Fri 03 May 2013 01:52:09 PM UTC, comment #4:
Yes, I loaded results.mat to Matlab (8.0.0.783) and saved it as results2.mat. Loading results2.mat in Octave gave the expected results (i.e. all doubles as -2.2900e+226).
results.mat is from a program, which writes its output in Level 5 MAT -format. Not from Matlab or Octave.
I reported this because Octave seems to read the variables correctly, but actually reads in random data. The variable names and sizes are read correctly.
|
Fri 03 May 2013 01:51:02 PM UTC, comment #3:
Assuming this is a valid mat file, I can confirm what you are reporting in the current development release as well.
|
Fri 03 May 2013 01:32:26 PM UTC, comment #2:
When you say saving in Matlab and reloading in Octave solves the problem, do you mean that if you load results.mat and save as results2.mat from Matlab, and then load results2.mat in Octave there is no error? Where did results.mat originate from? How was it saved differently?
|
Thu 02 May 2013 10:05:08 PM UTC, comment #1:
Update: test this with 3.6.2, still stochastic data from the load.
-------------------
octave:1> version
ans = 3.6.2
octave:2> clear; run2 = load('results.mat'); run2.q
ans =
6.9174e-310
1.9439e-316
1.9849e-316
5.7095e+228
1.9850e-316
1.9849e-316
1.9849e-316
4.9407e-324
1.9850e-316
1.9848e-316
1.9848e-316
1.4559e+218
1.9850e-316
1.9849e-316
1.9847e-316
4.9407e-324
1.9850e-316
1.9847e-316
1.9847e-316
5.7095e+228
1.9850e-316
1.9847e-316
1.9847e-316
7.7278e+228
1.9850e-316
1.9847e-316
1.9846e-316
1.8767e-152
1.9850e-316
1.9846e-316
1.9846e-316
4.9407e-324
1.3439e-321
octave:3> clear; run2 = load('results.mat'); run2.q
ans =
6.9174e-310
2.0806e-316
1.9913e-316
4.9407e-324
1.9914e-316
1.9913e-316
1.9913e-316
4.9407e-324
1.9914e-316
1.9913e-316
1.9913e-316
6.0135e-154
1.9914e-316
1.9913e-316
1.9912e-316
5.2628e+199
1.9914e-316
1.9911e-316
1.9911e-316
4.9407e-324
1.9914e-316
1.9912e-316
1.9912e-316
6.9173e-310
1.9914e-316
1.9912e-316
1.9912e-316
4.9407e-324
1.9914e-316
1.9912e-316
1.9912e-316
6.9173e-310
1.3439e-321
octave:4> whos -file 'results.mat' run2.q
Variables in the file results.mat:
Attr Name Size Bytes Class
==== ==== ==== ===== =====
q 33x1 264 double
w 33x1 264 double
xxxxxxx 1x12 12 char
yyyyyy 34x1 272 double
z 33x33 8712 double
Total is 1201 elements using 9524 bytes
---------------
|
Thu 02 May 2013 05:11:57 PM UTC, original submission:
Load function gives out stochastic data for a few element MAT -file.
Used file (results.mat) attached: all included doubles are 0xEEEEEEEE. Matlab 8.0.0.783 reads this correctly. Saving in Matlab and reloading in Octave solves the acute problem, but does not really solve the underlying problem in Octave.
Here is the backlog:
----------------------------------------------
octave:1> version
ans = 3.2.4
octave:2> clear; run2 = load('results.mat'); run2.q
ans =
6.9225e-310
6.9225e-310
2.1220e-314
6.9225e-310
2.3715e-322
2.3715e-322
6.9567e-317
6.9566e-317
6.8847e-317
1.1932e+190
4.7430e-322
3.1620e-322
6.9333e-317
6.9322e-317
6.9953e-317
6.9737e-317
9.0236e-317
9.0261e-317
9.0261e-317
1.0326e-321
6.8898e-317
9.0182e-317
6.8861e-317
4.9407e-324
0.0000e+00
2.4209e-322
6.9553e-317
7.0220e-317
9.1979e-317
6.9225e-310
4.7430e-322
3.1620e-322
9.0182e-317
octave:3> clear; run2 = load('results.mat'); run2.q
ans =
6.9225e-310
6.9225e-310
1.5810e-322
2.3715e-322
9.2235e-317
5.4347e-323
2.1220e-314
6.2590e-85
3.7683e-317
1.3488e-321
6.9225e-310
6.9995e-317
8.1152e-317
8.3198e-317
8.3198e-317
6.3734e-322
8.2575e-317
6.9919e-317
0.0000e+00
0.0000e+00
8.3072e-317
7.7742e-317
7.7742e-317
3.2114e-322
9.2222e-317
6.9543e-317
0.0000e+00
0.0000e+00
8.3547e-317
8.3429e-317
1.5810e-321
3.1620e-322
6.8920e-317
octave:4> whos -file 'results.mat'
Variables in the file results.mat:
Attr Name Size Bytes Class
==== ==== ==== ===== =====
q 33x1 264 double
w 33x1 264 double
xxxxxxx 1x12 12 char
yyyyyy 34x1 272 double
z 33x33 8712 double
Total is 1201 elements using 9524 bytes
----------------------------------------------
|