bugGNU Octave - Bugs: bug #38326, save -binary or load -binary fails...

 
 

bug #38326: save -binary or load -binary fails with big matrices

Submitted by:  None
Submitted on:  Thu 14 Feb 2013 07:10:34 AM UTC  
 
Category: InterpreterSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Result
Status: FixedAssigned to: None
Originator Name: Originator Email: -unavailable-
Open/Closed: ClosedRelease: dev
Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 20 Feb 2013 11:08:15 PM UTC, comment #7:

Fixed for array sizes up to 2^31 elements in this changeset (http://hg.savannah.gnu.org/hgweb/octave/rev/821922270b66).

The original reporter will need to build from Mercurial sources in order to access the fix, or wait until the next major release of Octave. Alternatively, integer arrays already support 2^31 elements. If the GWAS data is micro-array intensity data that could be represented with a uint16 data type then that is an available solution without re-compiling.

Rik <rik5>
Project Administrator
Wed 20 Feb 2013 10:42:02 PM UTC, comment #6:

I'll apply the patch. I also checked on ov-base-int.cc and it is using byte_size () which returns a size_t object so save_binary for large int arrays is already functional.

Rik <rik5>
Project Administrator
Wed 20 Feb 2013 10:21:17 PM UTC, comment #5:

OK, your patch seems like a good solution for now. That it doesn't change the file format or the interface to write_doubles is a plus.

John W. Eaton <jwe>
Project Administrator
Wed 20 Feb 2013 10:15:53 PM UTC, comment #4:

I think there is an opportunity for a stop-gap measure which would improve things before actually moving fully to a new format.

Currently the dimensions are stored as int32_t objects, but this isn't so much the problem. The original poster was using dimensions of 20,000 X 4,000 which are both well shy of 2^31. The problem is the write_doubles routine which accepts a length which is the number of elements. The prototype for the function in liboctave/util/data-conv.cc is

And later on there is a call to os.write which does math with the octave_idx_type.

The real fun is that the multiplication by 8 means another 3 bits off the maximum number of elements allowed. There is no need to use the extremely large dimensions of the original poster to demonstrate the problem. The following example much more quickly reveals the problem:

The resulting file is only 45 bytes on disk which is clearly wrong.

As a stop-gap I would use this code in data-conv.cc.

On 32-bit systems you will still be limited to less than 2^31 bytes or 2^28 elements if they are IEEE doubles. But 64-bit systems will have a 64-bit streamsize and they can therefore handle up to 2^31 elements or 2^34 bytes. I verified that this works for me.

I'm attaching a patch that works for me. We might want to touch up ov-base-int.cc as well.

(file #27487)

Rik <rik5>
Project Administrator
Wed 20 Feb 2013 08:27:15 PM UTC, comment #3:

Yeah, we want to fix this so that large files can be opened and read, at least on 64-bit systems where you will be able to store all the data, provided that:

  • each data object is less than 2GB
  • some data objects are larger but contain less than about 2e9 elements.
  • some data object contain more than 2e9 elements and you've compiled with --enable-64 (which is not yet fully functional, so just doing that will not solve your problem).

These changes will require a new version of the Octave binary file format since the current format uses 32-bit integers for writing dimensions, regardless of the actual size of octave_idx_type.

For the Matlab file format, I don't think it is possible to have arrays larger than 2GB unless you use the HDF5-based file format and Octave does not yet support loading files of that type.

On 32-bit systems, it should also be possible to load portions of large files, but you clearly won't be able to load an entire file larger than 2GB into memory all at once. I can't remember whether Octave's load command currently supports loading only some variables from a file.

John W. Eaton <jwe>
Project Administrator
Wed 20 Feb 2013 07:56:13 PM UTC, comment #2:

The issue is that the number of bytes to write out is an octave_idx_type. You can check in your config.h file located in the top-level directory. In general, you probably have

This is generally a signed 32-bit integer. Taking away one bit for the sign leads to a byte-limit on files of 2^31 or 2GB.

One option, particularily if you will have very large matrices, is to compile with '--enable-64' which will make the octave_idx_type 64 bits. I've verified that this works. However, this isn't exactly optimal as this feature is experimental and you would alno This is currently experimental though and could require re-compiling your Fortran libraries (BLAS, LAPACK, ARPACK, etc.) for 64-bit pointers and not just Octave itself.

I'll bring this up on the octave-maintainers list to see if we want to do something short term to fix this.

Rik <rik5>
Project Administrator
Fri 15 Feb 2013 04:17:18 AM UTC, comment #1:

The binary file appears to be truncated at around 2 GB. If compression is used, I get a smaller file on disk but still truncated at the same number of elements.

Also, after clearing and loading the resulting file:

Mike Miller <mtmiller>
Project Administrator
Thu 14 Feb 2013 07:10:34 AM UTC, original submission:

This is also true for -v7
but not for save only but this produces
very huge files.

This size of matrices is common for GWAS genome wide association
studies.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #27487:  patch.LFS added by rik5 (3KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 6 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 20 Feb 2013 11:08:15 PM UTCrik5CategoryNone=>Interpreter
      StatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
      Summarysave -binary or load fails with big matrices=>save -binary or load -binary fails with big matrices
    Wed 20 Feb 2013 10:15:53 PM UTCrik5Attached File-=>Added patch.LFS, #27487
    Fri 15 Feb 2013 04:17:18 AM UTCmtmillerStatusNone=>Confirmed

    Back to the top


    Powered by Savane 3.1-cleanup1