bugGNU Octave - Bugs: bug #49699, Octave crashes when using fread to...

 
 

bug #49699: Octave crashes when using fread to load more than 2 GB from a file

Submitted by:  None
Submitted on:  Fri 25 Nov 2016 09:43:34 AM UTC  
 
Category: InterpreterSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Segfault, Bus Error, etc.
Status: FixedAssigned to: None
Originator Name: David RendonOriginator Email: -unavailable-
Open/Closed: ClosedRelease: dev
Operating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 04 Jan 2017 08:30:17 PM UTC, comment #6:

I checked in the following changeset on stable and merged with default:

http://hg.savannah.gnu.org/hgweb/octave/rev/36df0e0072a5

In the next stable release, if you build without --enable-64, you should still be able to read up to (approximately) 2^31 elements. On a 64-bit system, that can be more than 2^31 bytes.

With the new way of handling octave_idx_type on the default branch, you will be able to read many more elements because 64-bit indexing will be enabled by default, independent of the size of integers used by the BLAS and other numerical libraries.

This change fixes the problem for me so I'm closing the report. Re-open or submit a new bug report if there is something I missed.

John W. Eaton <jwe>
Project Administrator
Wed 04 Jan 2017 07:04:57 PM UTC, comment #5:

Rik: I'll look at it. Thanks for the reminder.

John W. Eaton <jwe>
Project Administrator
Wed 04 Jan 2017 06:39:40 PM UTC, comment #4:

@jwe: The octave_idx_type has been heavily reworked by you. Is this bug now ready for fixing?

Rik <rik5>
Project Administrator
Fri 25 Nov 2016 04:26:44 PM UTC, comment #3:

I'm reopening this report because if Octave crashes, then it's definitely a bug in Octave. There are also ways that we can fix this, even without 64-bit indexing. For example, the attached patch avoids the problem for me, and I'm now wondering whether we should rethink how we handle octave_idx_type. I'll post something more about that to the maintainers mailing list.

(file #39070)

John W. Eaton <jwe>
Project Administrator
Fri 25 Nov 2016 01:17:12 PM UTC, comment #2:

Thanks, Rik, that explains what to do to fix the problem (though Octave should just produce an error, not crash).

By the way, that wiki article is very misleading. The limitation is not 2 GB of memory, but rather 2^31 elements.

In an installation not compiled with <code>--enable-64</code> this produces an error as expected:

<pre>

>> a = zeros (102410242048, 1, 'int8');

error: out of memory or dimension too large for Octave's index type
</pre>

If we reduce the volume by just one slice, we do not hit the limit. But if we also change the data type a larger one the memory used by the array can be much larger:

<pre>

>> a = zeros (102410242047, 1, 'int8');
>> b = zeros (102410242047, 1, 'double');
>> whos

Variables in the current scope:

Attr Name Size Bytes Class
==== ==== ==== ===== =====
a 2146435072x1 2146435072 int8
b 2146435072x1 17171480576 double

Total is 4292870144 elements using 19317915648 bytes
</pre>

Both arrays have a slightly less than 2^31 elements. But while array <code>a</code> is just under 2 GB, <code>b</code> is close to 16 GB.

That is the reason why I'm able to workaround the limitation by reading the file in batches of less than 2 GB: my files and arrays are larger than 2 GB, but the number of elements is less than 2^31 because I use float32 and int16.

Anonymous
Fri 25 Nov 2016 11:53:38 AM UTC, comment #1:

Unless you have specifically compiled Octave with 64-bit indexing, the maximum array size is 2^31 or 2GB. See http://wiki.octave.org/Enable_large_arrays:_Build_octave_such_that_it_can_use_arrays_larger_than_2Gb.

Rik <rik5>
Project Administrator
Fri 25 Nov 2016 09:43:34 AM UTC, original submission:

Octave versions: 4.2.0-rc2 (Homebrew), 4.0.3 Mac OS X bundle
OS versions: OS X 10.11.6 El Capitan (Darwin 15.6.0)

Is reproducible: Yes, always.

Steps to reproduce:

% TheFile is any file larger than 2 GB, e.g. a large CT scan in a NIfTI file.
fid = fopen (TheFile, 'r');
myVar = fread(fid, 536870913, 'float32');

At this point Octave consistently crashes, some times immediately while in some cases it will display this error message for a few seconds:

Assertion failed: (input_buf_size >= 0), function read, file libinterp/corefcn/oct-stream.cc, line 6535.
panic: Abort trap: 6 -- stopping myself...
attempting to save variables to 'octave-workspace'...

(The line number of course depends on the Octave version.)

Other statements that cause the crash:

myVar = fread(fid, 1073741825, '*int16');
myVar = fread(fid, 2147483649, '*char');

They all have in common that the number of bytes requested are more than 2^31.

Workaround: Preallocate myVar and read chunks of less than 2GB from the file into it.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #39070:  diffs.txt added by jwe (3KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jwe (Updated the item)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 9 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 04 Jan 2017 08:30:17 PM UTCjweStatusConfirmed=>Fixed
      Open/ClosedOpen=>Closed
    Wed 04 Jan 2017 06:39:40 PM UTCrik5StatusNone=>Confirmed
    Fri 25 Nov 2016 04:26:44 PM UTCjweAttached File-=>Added diffs.txt, #39070
      Open/ClosedClosed=>Open
      Release4.2.0=>dev
    Fri 25 Nov 2016 11:53:38 AM UTCrik5CategoryOctave Function=>Interpreter
      Open/ClosedOpen=>Closed
      Operating SystemMac OS=>Any

    Back to the top


    Powered by Savane 3.1-cleanup1