bugGNU Octave - Bugs: bug #49699, Octave crashes when using fread to...

 
 

bug #49699: Octave crashes when using fread to load more than 2 GB from a file

Submitter:  None
Submitted:  Fri 25 Nov 2016 09:43:34 AM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Fixed Assigned to:  None
Originator Name:  David Rendon Originator Email:  -email is unavailable-
Open/Closed:  * Closed Release:  * dev
Operating System:  * Any Fixed Release:  None
Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Wed 04 Jan 2017 08:30:17 PM UTC, comment #6: 

I checked in the following changeset on stable and merged with default:

  http://hg.savannah.gnu.org/hgweb/octave/rev/36df0e0072a5

In the next stable release, if you build without --enable-64, you should still be able to read up to (approximately) 2^31 elements.  On a 64-bit system, that can be more than 2^31 bytes.

With the new way of handling octave_idx_type on the default branch, you will be able to read many more elements because 64-bit indexing will be enabled by default, independent of the size of integers used by the BLAS and other numerical libraries.

This change fixes the problem for me so I'm closing the report.  Re-open or submit a new bug report if there is something I missed.

John W. Eaton <jwe>
Group administrator
Wed 04 Jan 2017 07:04:57 PM UTC, comment #5: 

Rik: I'll look at it.  Thanks for the reminder.

John W. Eaton <jwe>
Group administrator
Wed 04 Jan 2017 06:39:40 PM UTC, comment #4: 

@jwe: The octave_idx_type has been heavily reworked by you.  Is this bug now ready for fixing?

Rik <rik5>
Group administrator
Fri 25 Nov 2016 04:26:44 PM UTC, comment #3: 

I'm reopening this report because if Octave crashes, then it's definitely a bug in Octave.  There are also ways that we can fix this, even without 64-bit indexing.  For example, the attached patch avoids the problem for me, and I'm now wondering whether we should rethink how we handle octave_idx_type.  I'll post something more about that to the maintainers mailing list.


(file #39070)

John W. Eaton <jwe>
Group administrator
Fri 25 Nov 2016 01:17:12 PM UTC, comment #2: 

Thanks, Rik, that explains what to do to fix the problem (though Octave should just produce an error, not crash).

By the way, that wiki article is very misleading. The limitation is not 2 GB of memory, but rather 2^31 elements.

In an installation not compiled with <code>--enable-64</code> this produces an error as expected:

<pre>

>> a = zeros (1024*1024*2048, 1, 'int8');

error: out of memory or dimension too large for Octave's index type
</pre>

If we reduce the volume by just one slice, we do not hit the limit. But if we also change the data type a larger one the memory used by the array can be much larger:

<pre>

>> a = zeros (1024*1024*2047, 1, 'int8');
>> b = zeros (1024*1024*2047, 1, 'double');
>> whos

Variables in the current scope:

   Attr Name            Size                     Bytes  Class
   ==== ====            ====                     =====  =====
        a      2146435072x1                 2146435072  int8
        b      2146435072x1                17171480576  double

Total is 4292870144 elements using 19317915648 bytes
</pre>

Both arrays have a slightly less than 2^31  elements. But while array <code>a</code> is just under 2 GB, <code>b</code> is close to 16 GB.

That is the reason why I'm able to workaround the limitation by reading the file in batches of less than 2 GB: my files and arrays are larger than 2 GB, but the number of elements is less than 2^31 because I use float32 and int16.

Anonymous
Fri 25 Nov 2016 11:53:38 AM UTC, comment #1: 

Unless you have specifically compiled Octave with 64-bit indexing, the maximum array size is 2^31 or 2GB.  See http://wiki.octave.org/Enable_large_arrays:_Build_octave_such_that_it_can_use_arrays_larger_than_2Gb.

Rik <rik5>
Group administrator
Fri 25 Nov 2016 09:43:34 AM UTC, original submission:  

Octave versions: 4.2.0-rc2 (Homebrew), 4.0.3 Mac OS X bundle
OS versions: OS X 10.11.6 El Capitan (Darwin 15.6.0)

Is reproducible:  Yes, always.

Steps to reproduce:

% TheFile is any file larger than 2 GB, e.g. a large CT scan in a NIfTI file.
fid = fopen (TheFile, 'r');
myVar = fread(fid, 536870913, 'float32');

At this point Octave consistently crashes, some times immediately while in some cases it will display this error message for a few seconds:

Assertion failed: (input_buf_size >= 0), function read, file libinterp/corefcn/oct-stream.cc, line 6535.
panic: Abort trap: 6 -- stopping myself...
attempting to save variables to 'octave-workspace'...

(The line number of course depends on the Octave version.)

Other statements that cause the crash:

myVar = fread(fid, 1073741825, '*int16');
myVar = fread(fid, 2147483649, '*char');

They all have in common that the number of bytes requested are more than 2^31.

Workaround: Preallocate myVar and read chunks of less than 2GB  from the file into it.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #39070:  diffs.txt added by jwe (3KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Updated the item)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-01-04 jwe StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2017-01-04 rik5 StatusNone Confirmed
    2016-11-25 jwe Attached File- Added diffs.txt, #39070
        Open/ClosedClosed Open
        Release4.2.0 dev
    2016-11-25 rik5 CategoryOctave Function Interpreter
        Open/ClosedOpen Closed
        Operating SystemMac OS Any

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code