bugGNU Octave - Bugs: bug #29786, Octave does not save sparse...

 
 

bug #29786: Octave does not save sparse matrices correctly in -mat format

Submitted by:  None
Submitted on:  Wed 05 May 2010 12:05:44 PM UTC  
 
Category: NoneSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: FixedAssigned to: David Bateman <dbateman>
Originator Name: TuxOriginator Email: -unavailable-
Open/Closed: ClosedRelease: 3.2.3
Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon 14 Nov 2011 08:10:21 AM UTC, comment #9:

Hi,

I am trying to load a fairly large .mat file created by Matlab and I'm getting a lot of errors like those shown in the original post here.

using octave 3.4.0 and java-1.2.8

eg.
error: load: invalid element type = 0

zero is very common 65535 also and raft of less frequent values.

Do these numbers actually represent something related to "data element type" or are they spurious values output because octave is incorrectly parsing the data?

ie does this mean I have a dozen formats that are not supported or probably just one ?

If it's just one I may have a go at fixing it or getting the data in an acceptable format.

Shouldn't Octave be trapping this a bit better?

Anonymous
Wed 12 May 2010 10:43:34 AM UTC, comment #8:

Great David, thanks a lot!

Beat Fasel <tuxo>
Wed 12 May 2010 12:21:46 AM UTC, comment #7:

There was a minor typo in the last changeset. You'll also need the changeset

http://hg.savannah.gnu.org/hgweb/octave/rev/3eafa521550b

D.

David Bateman <dbateman>
Project MemberIn charge of this item.
Wed 12 May 2010 12:11:24 AM UTC, comment #6:

As I suspected the length of the sparse data element of a mat-file was miscalculated when it was written in the following two cases

1) When the value of NNZ/NZMAX was 1. In this case the length was overestimated by 4 bytes
2) When the number of columns in the sparse matrix was zero. In this case the length of the data element was overestimated by 4 bytes..

and the bug was only evident when the sparse data element wasn't the last element in the file. These cases are relative rare in practice which is why this issue hadn't been spotted yet.

Also only the NNZ elements of the index vector ridx were being written to the file, and in fact now that NNZ and NZMAX are not equal in Octave the NZMAX elements of ridx should have been written. This however was a separate bug.

The changeset

http://hg.savannah.gnu.org/hgweb/octave/rev/0e98fb2068fc

addresses this bug. Thanks for the report

D.

David Bateman <dbateman>
Project MemberIn charge of this item.
Mon 10 May 2010 04:22:06 PM UTC, comment #5:

I checked Octave's behavior on a 32-bit machine and it also won't save sparse matrices correctly. Thus, my supposition about a difference between 32-bit and 64-bit ints was incorrect.

Rik <rik5>
Project Administrator
Mon 10 May 2010 12:35:48 PM UTC, comment #4:

The matlab file format is ugly in that the size of ech block of data is stored at the start of the block. However to calculate the size of the block you basically have to write the file and figure out the size post-hoc. This leaves two choices. Firstly you write the block once to a temporary location, calculate its size from there and then write the size to the correct location. Or secondly, you write a blank to the block size, write the data to the file, then use fseek to come back and correct the block size. Octave chooses the second option for performance reasons.

However, if the size of the block calculation is incorrect, then the seeks will be misperformed and the data of the following variables will be corruptted. I strongly suspect this is what is happening. I'm however surprised that I never noticed this issue.

I'll take this one on later in the week

D.

David Bateman <dbateman>
Project MemberIn charge of this item.
Sat 08 May 2010 07:06:53 PM UTC, comment #3:

Thanks Rik for investigating this issue further and narrowing it down to the sparse matrix. Regarding the computing platform, yes indeed, I'm on a x86_64 Linux machine.

Beat Fasel <tuxo>
Sat 08 May 2010 02:20:47 PM UTC, comment #2:

Apparently Octave has trouble saving sparse matrices and has for quite a while.

I spent some time narrowing the problem down. The following code is sufficient to provoke an error.

a = sparse(1,1);
a(1,1) = 1;
b = 2;
save -mat test.mat
clear
load test.mat

The error is reproducible with the current development sources as well as an ancient 3.0.5 version I tried.

During the load the error message is:
error: load: invalid element type = 1016

I suspect that binary structures are not being properly aligned during the save process. For instance, if the variables are reversed so that the sparse matrix is called b and the double variable is a then everything works. This puts the sparse matrix at the end of the file where it can't collide with any following variables. Similarly, if the sparse matrix is empty the test passes.

These results were obtained on an x86_64 machine. The original reporter didn't note whether they were using a 64 bit machine but it could be something as simple as using an int value and expecting that to be a defined size rather than using named sizes such as int32_t.

Rik <rik5>
Project Administrator
Thu 06 May 2010 08:16:39 AM UTC, comment #1:

As noted in my intial bug report, Matlab is able to load the data structure generated in Octave, while Octave can not read it anymore. However, looking at the data structure loaded in Matlab I noticed that the data in the struct array is being shifted around. First entry:

matchdata(1)

ans =

matchloc1: [219 313]
matchloc2: [267 87]
nbmatches: 1
votecenter: [81 278]
voteloc1: [231 307]
voteloc2: [279 81]
votemat: [884x1047 double]

Entry 200:

matchdata(200)

ans =

matchloc1: [884x1047 double]
matchloc2: [2x2 double]
nbmatches: [2x2 double]
votecenter: 2
voteloc1: [250 203]
voteloc2: [2x2 double]
votemat: [2x2 double]

I tested further and found when storing in Octave's native binary format (-binary), this bug does not occur. However, if data is stored in either the v6 or v7 Matlab format, the stored data format seems to be incorrect.

I attach a simple a file (testvotedata.m) that allows to generate faulty data with the option v6 (outtype 2) and v7 (outtype 1) and correct data with -binary (outtype 3).

(file #20464)

Beat Fasel <tuxo>
Wed 05 May 2010 12:05:44 PM UTC, original submission:

Octave can not load the attached data file (generated in Octave via save('-v7',mdname,'matchdata');
). Matlab (R2009a) on the other hand is able to do so.

load voteresults_01.mat in Matlab works. After loading we find the following data structure:

matchdata =

1x1256 struct array with fields:
matchloc1
matchloc2
nbmatches
votecenter
voteloc1
voteloc2
votemat

Note that votemat are sparse matrices.

In Octave (v 3.0.5 and v 3.2.3) I get the following behavior:

>> load voteresults_01.mat

error: load: invalid element type = 96
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 8
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 0
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 8
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 2
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 0
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 48
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 32768
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 0
error: load: trouble reading binary file `voteresults_01.mat'
error: load: invalid element type = 0
error: load: trouble reading binary file `voteresults_01.mat'

etc

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #20464:  testvotedata.m added by tuxo (1KiB - application/octet-stream)
file #20446:  voteresults_01.mat added by None (122KiB - text/plain - Data file that can not be loaded in Octave)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by dbateman (Posted a comment)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by tuxo (Updated the item)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 8 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 12 May 2010 12:11:24 AM UTCdbatemanStatusIn Progress=>Fixed
      Open/ClosedOpen=>Closed
    Mon 10 May 2010 12:35:48 PM UTCdbatemanStatusConfirmed=>In Progress
      Assigned toNone=>dbateman
    Sat 08 May 2010 02:20:47 PM UTCrik5StatusNone=>Confirmed
      SummaryData file can not be loaded in Octave=>Octave does not save sparse matrices correctly in -mat format
    Thu 06 May 2010 08:16:39 AM UTCtuxoAttached File-=>Added testvotedata.m, #20464
    Wed 05 May 2010 12:05:44 PM UTCNoneAttached File-=>Added voteresults_01.mat, #20446

    Back to the top


    Powered by Savane 3.1-cleanup1