bugGNU Octave - Bugs: bug #38870, load reads stochastic data from...

 
 

bug #38870: load reads stochastic data from MAT -file

Submitted by:  None
Submitted on:  Thu 02 May 2013 05:11:57 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: FixedAssigned to: None
Originator Name: Risto VanhanenOriginator Email: -unavailable-
Open/Closed: ClosedRelease: dev
Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Thu 16 May 2013 07:48:53 PM UTC, comment #16:

Thanks, I pushed your changeset so savannah.

Yes, it would be great to have some test files for the load/save code. Preferably they should not be too large since we will be distributing them with every copy of the Octave sources.

John W. Eaton <jwe>
Project Administrator
Mon 06 May 2013 07:10:53 PM UTC, comment #15:

The commit message of the attachment should now be within the guidelines. I'll see if I write a test later on.

(file #28034)

Anonymous
Mon 06 May 2013 01:58:00 PM UTC, comment #14:

As for testing, the current way to test I/O is to write to a temporary file and then check that that file is read. You might have to use fwrite to write this temporary file. Here is an example:

http://hg.savannah.gnu.org/hgweb/octave/file/38fef1e833ea/libinterp/corefcn/dlmread.cc#l481

This test should ideally go right after the DEFUN here:

http://hg.savannah.gnu.org/hgweb/octave/file/38fef1e833ea/libinterp/interpfcn/load-save.cc#l545

Jordi Gutiérrez Hermoso <jordigh>
Project Administrator
Mon 06 May 2013 01:50:34 PM UTC, comment #13:

Thanks, your patch looks good.

Would it be too much of a burden to amend it to include a commit message according to our guidelines?

http://wiki.octave.org/Commit_message_guidelines

Do "hg log -v" to see an example of other commit messages. If you're standing on your current patch, (if not, you can go to your current patch with "hg up -red31dac4e36"), you can do "hg commit --amend" to edit your commit message there accordingly.

If you find this too troublesome, let us know, and I can edit your commit message myself.

Jordi Gutiérrez Hermoso <jordigh>
Project Administrator
Sun 05 May 2013 07:55:38 PM UTC, comment #12:

I wrote a patch which basically communicates back whether the data is actually written in small data element format or not when reading a tag. That information is then used to handle padding. I.e., the patch does not assume that all small data elements are written as small data elements. I didn't change the write part so there is now READ_PAD (which takes the info of whether small element format is used) and PAD (the original).

This works correctly with both results.mat, results2.mat and Octave save + loads for matrices, cells containing stuff, char-arrays and sparse matrices. I didn't test objects, functions nor workspaces or different types (which int and float/double) of storage - just how Octave decided to store my tries. Is there a set of .mat -files which have been used in testing?

The mercurial changeset is included. Is there a correct way to get it forward?

(file #28025)

Anonymous
Fri 03 May 2013 09:43:05 PM UTC, comment #11:

Good point: small data elements (i.e. data included in tag) are the norm as written by Matlab. That compatability should not be broken.

The MAT-file format actually declares small data elements as optional. Octave just doesn't support them as normal data elements and does not even detect the situation and therefore reads in bogus data.

The in-lab code can be changed. I made a quick test and (1) four letter name gave incorrect read, (2) five letter name gave correct read, (3) writing in small data element format gave correct read. To me this is enough evidence.

I'll see if I can make Octave at least to report the standard format as unsupported, if not read it.

The 'results2.mat' is attached.

Thanks for help so far and for the repository tip. Checking it out at the moment.

(file #28012)

Anonymous
Fri 03 May 2013 07:44:11 PM UTC, comment #10:

I think you're on the right track about the short variable names. However, the 4-byte padding for short names is correct, at least in some of my v5 mat files that were written by Matlab.

Perhaps Matlab is flexible in how it reads the type element following the name, so it is permissive enough to read in your badly formatted file.

Can your in-laboratory program be modified to pad short variable names to only 4 bytes instead of 8? Or to ensure all variable names are longer than 4 characters?

Can you attach the file you get if you load this dataset into Matlab and then save it in v5 format?

Mike Miller <mtmiller>
Project Member
Fri 03 May 2013 07:25:20 PM UTC, comment #9:

If you want to get a local copy and build and test with Octave, you'll have to install Mercurial and clone it:

hg clone http://hg.savannah.gnu.org/hgweb/octave/

If you just want a tarball of the current dev version, you can get it here:

http://hg.savannah.gnu.org/hgweb/octave/archive/tip.tar.gz

This is currently missing the gnulib subrepo that you can get here:

http://hg.savannah.gnu.org/hgweb/octave/gnulib-hg/archive/tip.tar.gz

Jordi Gutiérrez Hermoso <jordigh>
Project Administrator
Fri 03 May 2013 07:02:18 PM UTC, comment #8:

The program is an in-laboratory code, which is not distributed to any other place. In such, the trouble is local. I think I am the first one who has tried to use the output in Octave.

However, I noticed that xxxxxxx and yyyyyy variables had no trouble loading correctly all the time. The trouble is with the single letter names.

I went through the ls-mat5.cc and I think the problem is line
696 (reading variable name):

is.seekg (tmp_pos + static_cast<std::streamoff> (PAD (len)));

PAD is defined as (len > 0 && len <= 4) ? 4 : ((l+7)/8)8; For the 1 char variables this gives padding of 4 and not proper 7. The padding should here be just ((l+7)/8)8 without the iffing (and probably in a few other places where small data fragments might occur).

This also explains why the parser sees the "type" as 0 (padding) instead of 9 (and the 9 will become data length).

Could either of you check this out? I don't have source code of Octave nor idea how to get a version.

Anonymous
Fri 03 May 2013 05:17:24 PM UTC, comment #7:

I looked at this file a bit too. I think the problem may be that the "type" codes for the matrices in the file are set to 0 instead of miDOUBLE. This does not seem like a valid format, at least according to the parser in Octave. The case statements in ls-mat5.cc simply fall through and don't read any data into the uninitialized matrices, which is probably why they are appearing as random values.

I inserted something like

somewhere around here:

http://hg.savannah.gnu.org/hgweb/octave/file/6698d0c42fbf/libinterp/interp-core/ls-mat5.cc#l1395

and get consistent values every time.

Mike Miller <mtmiller>
Project Member
Fri 03 May 2013 04:07:05 PM UTC, comment #6:

I am inspecting the file and I would like to know which program generated it, because it doesn't look completely like a valid file. I cannot diagnose it in detail yet, but I suspect that this file has the wrong size for one of its parameters. I suspect this is exposing both a bug in the program that generated this file and in Octave, plus undocumented behaviour in Matlab.

Note that Matlab has some quirks that are not in the published spec. I have run into some of them when debugging mat files. It is entirely possible that the program that generated this file is exploiting some of these hidden quirks in Matlab, which I expect we will now have to reproduce in Octave.

Jordi Gutiérrez Hermoso <jordigh>
Project Administrator
Fri 03 May 2013 02:03:43 PM UTC, comment #5:

It is supposed to be a valid MAT -file, but the best I can say is that Matlab reads it fine and as expected. In any case Octave should not give different results from run to run since the original file is the same. That is, Octave should fail deterministically if it fails to read the file.

Anonymous
Fri 03 May 2013 01:52:09 PM UTC, comment #4:

Yes, I loaded results.mat to Matlab (8.0.0.783) and saved it as results2.mat. Loading results2.mat in Octave gave the expected results (i.e. all doubles as -2.2900e+226).

results.mat is from a program, which writes its output in Level 5 MAT -format. Not from Matlab or Octave.

I reported this because Octave seems to read the variables correctly, but actually reads in random data. The variable names and sizes are read correctly.

Anonymous
Fri 03 May 2013 01:51:02 PM UTC, comment #3:

Assuming this is a valid mat file, I can confirm what you are reporting in the current development release as well.

Mike Miller <mtmiller>
Project Member
Fri 03 May 2013 01:32:26 PM UTC, comment #2:

When you say saving in Matlab and reloading in Octave solves the problem, do you mean that if you load results.mat and save as results2.mat from Matlab, and then load results2.mat in Octave there is no error? Where did results.mat originate from? How was it saved differently?

Mike Miller <mtmiller>
Project Member
Thu 02 May 2013 10:05:08 PM UTC, comment #1:

Update: test this with 3.6.2, still stochastic data from the load.

-------------------
octave:1> version
ans = 3.6.2
octave:2> clear; run2 = load('results.mat'); run2.q
ans =

6.9174e-310
1.9439e-316
1.9849e-316
5.7095e+228
1.9850e-316
1.9849e-316
1.9849e-316
4.9407e-324
1.9850e-316
1.9848e-316
1.9848e-316
1.4559e+218
1.9850e-316
1.9849e-316
1.9847e-316
4.9407e-324
1.9850e-316
1.9847e-316
1.9847e-316
5.7095e+228
1.9850e-316
1.9847e-316
1.9847e-316
7.7278e+228
1.9850e-316
1.9847e-316
1.9846e-316
1.8767e-152
1.9850e-316
1.9846e-316
1.9846e-316
4.9407e-324
1.3439e-321

octave:3> clear; run2 = load('results.mat'); run2.q
ans =

6.9174e-310
2.0806e-316
1.9913e-316
4.9407e-324
1.9914e-316
1.9913e-316
1.9913e-316
4.9407e-324
1.9914e-316
1.9913e-316
1.9913e-316
6.0135e-154
1.9914e-316
1.9913e-316
1.9912e-316
5.2628e+199
1.9914e-316
1.9911e-316
1.9911e-316
4.9407e-324
1.9914e-316
1.9912e-316
1.9912e-316
6.9173e-310
1.9914e-316
1.9912e-316
1.9912e-316
4.9407e-324
1.9914e-316
1.9912e-316
1.9912e-316
6.9173e-310
1.3439e-321

octave:4> whos -file 'results.mat' run2.q
Variables in the file results.mat:

Attr Name Size Bytes Class
==== ==== ==== ===== =====
q 33x1 264 double
w 33x1 264 double
xxxxxxx 1x12 12 char
yyyyyy 34x1 272 double
z 33x33 8712 double

Total is 1201 elements using 9524 bytes
---------------

Anonymous
Thu 02 May 2013 05:11:57 PM UTC, original submission:

Load function gives out stochastic data for a few element MAT -file.

Used file (results.mat) attached: all included doubles are 0xEEEEEEEE. Matlab 8.0.0.783 reads this correctly. Saving in Matlab and reloading in Octave solves the acute problem, but does not really solve the underlying problem in Octave.

Here is the backlog:

----------------------------------------------
octave:1> version
ans = 3.2.4
octave:2> clear; run2 = load('results.mat'); run2.q
ans =

6.9225e-310
6.9225e-310
2.1220e-314
6.9225e-310
2.3715e-322
2.3715e-322
6.9567e-317
6.9566e-317
6.8847e-317
1.1932e+190
4.7430e-322
3.1620e-322
6.9333e-317
6.9322e-317
6.9953e-317
6.9737e-317
9.0236e-317
9.0261e-317
9.0261e-317
1.0326e-321
6.8898e-317
9.0182e-317
6.8861e-317
4.9407e-324
0.0000e+00
2.4209e-322
6.9553e-317
7.0220e-317
9.1979e-317
6.9225e-310
4.7430e-322
3.1620e-322
9.0182e-317

octave:3> clear; run2 = load('results.mat'); run2.q
ans =

6.9225e-310
6.9225e-310
1.5810e-322
2.3715e-322
9.2235e-317
5.4347e-323
2.1220e-314
6.2590e-85
3.7683e-317
1.3488e-321
6.9225e-310
6.9995e-317
8.1152e-317
8.3198e-317
8.3198e-317
6.3734e-322
8.2575e-317
6.9919e-317
0.0000e+00
0.0000e+00
8.3072e-317
7.7742e-317
7.7742e-317
3.2114e-322
9.2222e-317
6.9543e-317
0.0000e+00
0.0000e+00
8.3547e-317
8.3429e-317
1.5810e-321
3.1620e-322
6.8920e-317

octave:4> whos -file 'results.mat'
Variables in the file results.mat:

Attr Name Size Bytes Class
==== ==== ==== ===== =====
q 33x1 264 double
w 33x1 264 double
xxxxxxx 1x12 12 char
yyyyyy 34x1 272 double
z 33x33 8712 double

Total is 1201 elements using 9524 bytes
----------------------------------------------

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #28034:  bug38870 added by None (11kB - application/octet-stream)
file #28025:  bug38870 added by None (11kB - application/octet-stream)
file #28012:  results2.mat added by None (430B - application/octet-stream)
file #27995:  results.mat added by None (10kB - application/x-extension-mat - Test data.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by jordigh (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 9 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 16 May 2013 07:48:53 PM UTCjweStatusPatch Submitted=>Fixed
      Open/ClosedOpen=>Closed
    Mon 06 May 2013 07:10:53 PM UTCNoneAttached File-=>Added bug38870, #28034
    Mon 06 May 2013 01:50:34 PM UTCjordighStatusNeed Info=>Patch Submitted
    Sun 05 May 2013 07:55:38 PM UTCNoneAttached File-=>Added bug38870, #28025
    Fri 03 May 2013 09:43:05 PM UTCNoneAttached File-=>Added results2.mat, #28012
    Fri 03 May 2013 01:51:02 PM UTCmtmillerRelease3.2.4=>dev
    Fri 03 May 2013 01:32:26 PM UTCmtmillerStatusNone=>Need Info
    Thu 02 May 2013 05:11:57 PM UTCNoneAttached File-=>Added results.mat, #27995

    Back to the top


    Powered by Savane 3.1-cleanup