bugGNU Octave - Bugs: bug #38870, load reads stochastic data from...

 
 

bug #38870: load reads stochastic data from MAT -file

Submitter:  None
Submitted:  Thu 02 May 2013 05:11:57 PM UTC
   
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Fixed Assigned to:  None
Originator Name:  Risto Vanhanen Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * GNU/Linux Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 16 May 2013 07:48:53 PM UTC, comment #16: 

Thanks, I pushed your changeset so savannah.

Yes, it would be great to have some test files for the load/save code.  Preferably they should not be too large since we will be distributing them with every copy of the Octave sources.

John W. Eaton <jwe>
Group administrator
Mon 06 May 2013 07:10:53 PM UTC, comment #15: 

The commit message of the attachment should now be within the guidelines. I'll see if I write a test later on.

(file #28034)

Anonymous
Mon 06 May 2013 01:58:00 PM UTC, comment #14: 

As for testing, the current way to test I/O is to write to a temporary file and then check that that file is read. You might have to use fwrite to write this temporary file. Here is an example:

http://hg.savannah.gnu.org/hgweb/octave/file/38fef1e833ea/libinterp/corefcn/dlmread.cc#l481

This test should ideally go right after the DEFUN here:

http://hg.savannah.gnu.org/hgweb/octave/file/38fef1e833ea/libinterp/interpfcn/load-save.cc#l545

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Mon 06 May 2013 01:50:34 PM UTC, comment #13: 

Thanks, your patch looks good.

Would it be too much of a burden to amend it to include a commit message according to our guidelines?

http://wiki.octave.org/Commit_message_guidelines

Do "hg log -v" to see an example of other commit messages. If you're standing on your current patch, (if not, you can go to your current patch with "hg up -red31dac4e36"), you can do "hg commit --amend" to edit your commit message there accordingly.

If you find this too troublesome, let us know, and I can edit your commit message myself.

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Sun 05 May 2013 07:55:38 PM UTC, comment #12: 

I wrote a patch which basically communicates back whether the data is actually written in small data element format or not when reading a tag. That information is then used to handle padding. I.e., the patch does not assume that all small data elements are written as small data elements. I didn't change the write part so there is now READ_PAD (which takes the info of whether small element format is used) and PAD (the original).

This works correctly with both results.mat, results2.mat and Octave save + loads for matrices, cells containing stuff, char-arrays and sparse matrices. I didn't test objects, functions nor workspaces or different types (which int and float/double) of storage - just how Octave decided to store my tries. Is there a set of .mat -files which have been used in testing?

The mercurial changeset is included. Is there a correct way to get it forward?

(file #28025)

Anonymous
Fri 03 May 2013 09:43:05 PM UTC, comment #11: 

Good point: small data elements (i.e. data included in tag) are the norm as written by Matlab. That compatability should not be broken.

The MAT-file format actually declares small data elements as optional. Octave just doesn't support them as normal data elements and does not even detect the situation and therefore reads in bogus data.

The in-lab code can be changed. I made a quick test and (1) four letter name gave incorrect read, (2) five letter name gave correct read, (3) writing in small data element format gave correct read. To me this is enough evidence.

I'll see if I can make Octave at least to report the standard format as unsupported, if not read it.

The 'results2.mat' is attached.

Thanks for help so far and for the repository tip. Checking it out at the moment.

(file #28012)

Anonymous
Fri 03 May 2013 07:44:11 PM UTC, comment #10: 

I think you're on the right track about the short variable names. However, the 4-byte padding for short names is correct, at least in some of my v5 mat files that were written by Matlab.

Perhaps Matlab is flexible in how it reads the type element following the name, so it is permissive enough to read in your badly formatted file.

Can your in-laboratory program be modified to pad short variable names to only 4 bytes instead of 8? Or to ensure all variable names are longer than 4 characters?

Can you attach the file you get if you load this dataset into Matlab and then save it in v5 format?

Mike Miller <mtmiller>
Group Member
Fri 03 May 2013 07:25:20 PM UTC, comment #9: 

If you want to get a local copy and build and test with Octave, you'll have to install Mercurial and clone it:

hg clone http://hg.savannah.gnu.org/hgweb/octave/

If you just want a tarball of the current dev version, you can get it here:

http://hg.savannah.gnu.org/hgweb/octave/archive/tip.tar.gz

This is currently missing the gnulib subrepo that you can get here:

http://hg.savannah.gnu.org/hgweb/octave/gnulib-hg/archive/tip.tar.gz

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Fri 03 May 2013 07:02:18 PM UTC, comment #8: 

The program is an in-laboratory code, which is not distributed to any other place. In such, the trouble is local. I think I am the first one who has tried to use the output in Octave.

However, I noticed that xxxxxxx and yyyyyy variables had no trouble loading correctly all the time. The trouble is with the single letter names.

I went through the ls-mat5.cc and I think the problem is line
696 (reading variable name):

is.seekg (tmp_pos + static_cast<std::streamoff> (PAD (len)));

PAD is defined as (len > 0 && len <= 4) ? 4 : ((l+7)/8)*8; For the 1 char variables this gives padding of 4 and not proper 7. The padding should here be just ((l+7)/8)*8 without the iffing (and probably in a few other places where small data fragments might occur).

This also explains why the parser sees the "type" as 0 (padding) instead of 9 (and the 9 will become data length).

Could either of you check this out? I don't have source code of Octave nor idea how to get a version.

Anonymous
Fri 03 May 2013 05:17:24 PM UTC, comment #7: 

I looked at this file a bit too. I think the problem may be that the "type" codes for the matrices in the file are set to 0 instead of miDOUBLE. This does not seem like a valid format, at least according to the parser in Octave. The case statements in ls-mat5.cc simply fall through and don't read any data into the uninitialized matrices, which is probably why they are appearing as random values.

I inserted something like


if (!type) type = miDOUBLE;


somewhere around here:

http://hg.savannah.gnu.org/hgweb/octave/file/6698d0c42fbf/libinterp/interp-core/ls-mat5.cc#l1395

and get consistent values every time.

Mike Miller <mtmiller>
Group Member
Fri 03 May 2013 04:07:05 PM UTC, comment #6: 

I am inspecting the file and I would like to know which program generated it, because it doesn't look completely like a valid file. I cannot diagnose it in detail yet, but I suspect that this file has the wrong size for one of its parameters. I suspect this is exposing both a bug in the program that generated this file and in Octave, plus undocumented behaviour in Matlab.

Note that Matlab has some quirks that are not in the published spec. I have run into some of them when debugging mat files. It is entirely possible that the program that generated this file is exploiting some of these hidden quirks in Matlab, which I expect we will now have to reproduce in Octave.

Jordi Gutiérrez Hermoso <jordigh>
Group Member
Fri 03 May 2013 02:03:43 PM UTC, comment #5: 

It is supposed to be a valid MAT -file, but the best I can say is that Matlab reads it fine and as expected. In any case Octave should not give different results from run to run since the original file is the same. That is, Octave should fail deterministically if it fails to read the file.

Anonymous
Fri 03 May 2013 01:52:09 PM UTC, comment #4: 

Yes, I loaded results.mat to Matlab (8.0.0.783) and saved it as results2.mat. Loading results2.mat in Octave gave the expected results (i.e. all doubles as -2.2900e+226).

results.mat is from a program, which writes its output in Level 5 MAT -format. Not from Matlab or Octave.

I reported this because Octave seems to read the variables correctly, but actually reads in random data. The variable names and sizes are read correctly.

Anonymous
Fri 03 May 2013 01:51:02 PM UTC, comment #3: 

Assuming this is a valid mat file, I can confirm what you are reporting in the current development release as well.

Mike Miller <mtmiller>
Group Member
Fri 03 May 2013 01:32:26 PM UTC, comment #2: 

When you say saving in Matlab and reloading in Octave solves the problem, do you mean that if you load results.mat and save as results2.mat from Matlab, and then load results2.mat in Octave there is no error? Where did results.mat originate from? How was it saved differently?

Mike Miller <mtmiller>
Group Member
Thu 02 May 2013 10:05:08 PM UTC, comment #1: 

Update: test this with 3.6.2, still stochastic data from the load.

-------------------
octave:1> version
ans = 3.6.2
octave:2> clear; run2 = load('results.mat'); run2.q
ans =

   6.9174e-310
   1.9439e-316
   1.9849e-316
   5.7095e+228
   1.9850e-316
   1.9849e-316
   1.9849e-316
   4.9407e-324
   1.9850e-316
   1.9848e-316
   1.9848e-316
   1.4559e+218
   1.9850e-316
   1.9849e-316
   1.9847e-316
   4.9407e-324
   1.9850e-316
   1.9847e-316
   1.9847e-316
   5.7095e+228
   1.9850e-316
   1.9847e-316
   1.9847e-316
   7.7278e+228
   1.9850e-316
   1.9847e-316
   1.9846e-316
   1.8767e-152
   1.9850e-316
   1.9846e-316
   1.9846e-316
   4.9407e-324
   1.3439e-321

octave:3> clear; run2 = load('results.mat'); run2.q
ans =

   6.9174e-310
   2.0806e-316
   1.9913e-316
   4.9407e-324
   1.9914e-316
   1.9913e-316
   1.9913e-316
   4.9407e-324
   1.9914e-316
   1.9913e-316
   1.9913e-316
   6.0135e-154
   1.9914e-316
   1.9913e-316
   1.9912e-316
   5.2628e+199
   1.9914e-316
   1.9911e-316
   1.9911e-316
   4.9407e-324
   1.9914e-316
   1.9912e-316
   1.9912e-316
   6.9173e-310
   1.9914e-316
   1.9912e-316
   1.9912e-316
   4.9407e-324
   1.9914e-316
   1.9912e-316
   1.9912e-316
   6.9173e-310
   1.3439e-321

octave:4> whos -file 'results.mat' run2.q
Variables in the file results.mat:

   Attr Name         Size                     Bytes  Class
   ==== ====         ====                     =====  =====
        q           33x1                        264  double
        w           33x1                        264  double
        xxxxxxx      1x12                        12  char
        yyyyyy      34x1                        272  double
        z           33x33                      8712  double

Total is 1201 elements using 9524 bytes
---------------

Anonymous
Thu 02 May 2013 05:11:57 PM UTC, original submission:  

Load function gives out stochastic data for a few element MAT -file.

Used file (results.mat) attached: all included doubles are 0xEEEEEEEE. Matlab 8.0.0.783 reads this correctly. Saving in Matlab and reloading in Octave solves the acute problem, but does not really solve the underlying problem in Octave.

Here is the backlog:

----------------------------------------------
octave:1> version
ans = 3.2.4
octave:2> clear; run2 = load('results.mat'); run2.q
ans =

   6.9225e-310
   6.9225e-310
   2.1220e-314
   6.9225e-310
   2.3715e-322
   2.3715e-322
   6.9567e-317
   6.9566e-317
   6.8847e-317
   1.1932e+190
   4.7430e-322
   3.1620e-322
   6.9333e-317
   6.9322e-317
   6.9953e-317
   6.9737e-317
   9.0236e-317
   9.0261e-317
   9.0261e-317
   1.0326e-321
   6.8898e-317
   9.0182e-317
   6.8861e-317
   4.9407e-324
    0.0000e+00
   2.4209e-322
   6.9553e-317
   7.0220e-317
   9.1979e-317
   6.9225e-310
   4.7430e-322
   3.1620e-322
   9.0182e-317

octave:3> clear; run2 = load('results.mat'); run2.q
ans =

  6.9225e-310
  6.9225e-310
  1.5810e-322
  2.3715e-322
  9.2235e-317
  5.4347e-323
  2.1220e-314
   6.2590e-85
  3.7683e-317
  1.3488e-321
  6.9225e-310
  6.9995e-317
  8.1152e-317
  8.3198e-317
  8.3198e-317
  6.3734e-322
  8.2575e-317
  6.9919e-317
   0.0000e+00
   0.0000e+00
  8.3072e-317
  7.7742e-317
  7.7742e-317
  3.2114e-322
  9.2222e-317
  6.9543e-317
   0.0000e+00
   0.0000e+00
  8.3547e-317
  8.3429e-317
  1.5810e-321
  3.1620e-322
  6.8920e-317

octave:4> whos -file 'results.mat'
Variables in the file results.mat:

  Attr Name         Size                     Bytes  Class
  ==== ====         ====                     =====  =====
       q           33x1                        264  double
       w           33x1                        264  double
       xxxxxxx      1x12                        12  char
       yyyyyy      34x1                        272  double
       z           33x33                      8712  double

Total is 1201 elements using 9524 bytes
----------------------------------------------

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #28034:  bug38870 added by None (11KiB - application/octet-stream)
file #28025:  bug38870 added by None (11KiB - application/octet-stream)
file #28012:  results2.mat added by None (430B - application/octet-stream)
file #27995:  results.mat added by None (10KiB - application/x-extension-mat - Test data.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by jordigh (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-05-16 jwe StatusPatch Submitted Fixed
        Open/ClosedOpen Closed
    2013-05-06 None Attached File- Added bug38870, #28034
    2013-05-06 jordigh StatusNeed Info Patch Submitted
    2013-05-05 None Attached File- Added bug38870, #28025
    2013-05-03 None Attached File- Added results2.mat, #28012
    2013-05-03 mtmiller Release3.2.4 dev
    2013-05-03 mtmiller StatusNone Need Info
    2013-05-02 None Attached File- Added results.mat, #27995

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code