bugGNU Octave - Bugs: bug #43495, Crash when reading large TIFF with...

 
 

bug #43495: Crash when reading large TIFF with imread()

Submitter:  Armin Müller <arm_in>
Submitted:  Wed 29 Oct 2014 12:53:25 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Works For Me Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * 3.8.2 Operating System:  * Microsoft Windows
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 04 Nov 2014 04:15:39 PM UTC, comment #15: 

Slightly OT but related tot RAM usage:

At work we run the network installation of Matlab in a "bubble", a sort of virtual environment.
That gives considerably better results as to RAM usage (but don't get me started on CPU performance :-) ):

>> ver
------------------------------------------------------------------------------------------------
MATLAB Version: 7.14.0.739 (R2012a)
MATLAB License Number: ******
Operating System: Microsoft Windows 7 Version 6.1 (Build 7601: Service Pack 1)
Java Version: Java 1.6.0_17-b04 with Sun Microsystems Inc. Java HotSpot(TM) Client VM mixed mode
------------------------------------------------------------------------------------------------
MATLAB                                                Version 7.14       (R2012a)

<snip>

>> memory
Maximum possible array:            1151 MB (1.207e+09 bytes) *
Memory available for all arrays:   1423 MB (1.492e+09 bytes) **
Memory used by MATLAB:              357 MB (3.744e+08 bytes)
Physical Memory (RAM):             4096 MB (4.295e+09 bytes)

*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.
>>


(compare to the results in comment #13)

Maybe that can be made into a useful hint (I cannot actually confirm it here) to get max. array sizes: i.e., to run Octave in a virtual machine in a guest environment stripped down to the bare minimum required to run user programs.

When I get time for it I'll put something about RAM usage and OOM errors on the wiki.

Philip Nienhuis <philipnienhuis>
Group Member
Fri 31 Oct 2014 02:22:38 PM UTC, comment #14: 

Your build of GraphisMagick may be the limitation here. The way you built it can change your results. On my case, it's built with quantum 16 so it reads it as uint16 image (and probably RGB as well). Octave receives it and changes it back into what it really is, a logical matrix. In your case, it may be reading it as an uint8 RGB image.

Ideally, you'd implement the missing Tiff class which would simply be a wrapper to libtiff. This allows for a finer control while reading the image file. This requires classdef will be released with Octave 4.2.

Alternatively, you can:

1) try using the "PixelRegion" of imread() to read it in blocks and make your image out of its pieces. Use iminfo to get the image size, and then fill it in blocks.

2) implement a tiff reader for your files. I've done it before to read very special Tiffs. It's not that hard and there's plenty of examples on the internet written as m files (writing a general tiff is hard, but writing for a very specific subset can be really easy).

Carnë Draug <carandraug>
Group Member
Fri 31 Oct 2014 12:43:46 PM UTC, comment #13: 

@Armin:
"..far less than I would expect"....  Well I have some less rosy info for you then ;-)

To illustrate what I wrote at the bottom of comment #6, here's some output from Matlab r2014b prerelease on my WinXP 32bit box w. 4 GB RAM:

Just after starting up WinXP, first ML command:

>> memory
Maximum possible array:    354 MB (3.715e+08 bytes) *
Memory available for all arrays:   1358 MB (1.424e+09 bytes) **
Memory used by MATLAB:    403 MB (4.227e+08 bytes)
Physical Memory (RAM):   3324 MB (3.485e+09 bytes)

*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.
>>


and after some work:

>> memory
Maximum possible array:    277 MB (2.901e+08 bytes) *
Memory available for all arrays:   1285 MB (1.347e+09 bytes) **
Memory used by MATLAB:    456 MB (4.786e+08 bytes)
Physical Memory (RAM):   3324 MB (3.485e+09 bytes)

*  Limited by contiguous virtual address space available.
** Limited by virtual address space available.
>>


For comparison, Octave 4.1.0+ CLI allowed me to squeeze a double array of up to 8341x8341 into memory, max. 556 MB (or MiB?).

(To be honest, these results disappoint me, I had expected max. array sizes would be 600-700 MB. But: no.)

In conclusion I think you're even quite lucky to get 0.73 GB arrays into Octave on a 32-bit system .... (but on the bright side: Octave compares quite good to the competition...)

Philip Nienhuis <philipnienhuis>
Group Member
Fri 31 Oct 2014 08:33:53 AM UTC, comment #12: 

I've tried the "large indexing" version of Octave. Everything runs fine. That's the good point.

Unfortunately, I've also tried to fathom the maximum image size allowed in this version. Typing

x = ones(50000, 50000);    % ~18.6 GB

did lock up my system completely. No warning, no error, just endlessly swapping memory on the HD, not even the task manager did respond. I had to do a "hard" reboot.

As written below, 14k x 14k "int32" = 0.73 GB is the maximum I can get on the 32 bit system. Alternatively 9k x 9k "double" = 0.60 GB. The exact value varies, it seems that it is heavily depending on the preconditions if the memory is absolutely "clear" and Octave started newly, or if it is "used". But both values are far less than I would expect.

Armin Müller <arm_in>
Fri 31 Oct 2014 12:19:09 AM UTC, comment #11: 

@Arun: Try the large index version of Octave and report back if it makes a difference.

The problem from Octave's perspective is that we do not know the return size of the image ahead of time.  The whole point of using the external GraphicsMagick package is to take care of all the annoying image formats.  Sometimes, for example, an RGB JPEG image will be returned as a grayscale image (1/3 the memory) if the R, G, and B planes all have the same value.

So, my guess is that GraphicsMagick itself is trying to load the file, and choosing a format that is too large initially, which leads to a segfault.  On a successful load the end class is logical, or just 1B per pixel, but it may be loading it at TrueColor depth of 4B per pixel first before slimming it down.

As Armin tested,


x = ones (15153, 20127, "logical")
WORKS, storage required is 15153*20127*1 = ~305 MB

x = ones (15153, 20127)
FAILS, storage required is 15153*20127*8 = 2.4 GB


This sure looks like an indication that it is exceeding a 32-bit range.  Note that for int32 the top-bit is often used as a valid bit so the real range is just to 2^31 which is 2.1GB.

I suppose a way to be certain is to try


x = ones (15153, 20127, "int32");


This would require only 1.2GB which should not cause any problems.

Rik <rik5>
Group administrator
Thu 30 Oct 2014 10:18:16 PM UTC, comment #10: 

@Carnë / comment #6:

With a freshly built 3.9.0+ (--enable-64) for MingW, using Markus' patch (bug #43505), I get similar results as you:

>> tic; img = imread ("Erik_1_6000dpi.tiff'); toc
Elapsed time = 6.85139 seconds.


but then:

tic; imshow (img); toc
Elapsed time = 4.97828 seconds.

and although Octave may be ready, only the outline of the graphics window is drawn and Octave doesn't return to the prompt, instead the hard disk started crunching. After half an hour I gave up and killed Octave.
Windows' Task Manager indicated ~6.7 GB of the available 8 GB is used. Even with "stolen RAM" (video) and drivers address space I think there's enough RAM available.

Similar goes for "spy (img)"

Philip Nienhuis <philipnienhuis>
Group Member
Thu 30 Oct 2014 07:16:32 PM UTC, comment #9: 

Well I spoke too soon - to be able to test Octave on Linux with --enable-64 the entire dependency chain has to be rebuilt (using mxe), see bug #43319 comment #19.
That'll take too much time ATM. Maybe maybe later this weekend.


Philip Nienhuis <philipnienhuis>
Group Member
Thu 30 Oct 2014 05:42:21 PM UTC, comment #8: 

@Carnë / comment #6:

Maybe. Mine are just plain mxe-octave builds.
So what could be wrong? all dependencies have been built with the same tool chain.
What I did see is that "spy (img)" and "imshow (img)" were very fast in 3.9.0+ but only showed one star in the center; on 3.8.2 both took ages and in the end a dark rectangle appeared; zooming in there lead to a blank picture. (AFAIK the actual picture should be a electronics component board.)
So, not good. But before entering a bug report I'll investigate further.

Later on I'll try on Linux, but presently an mxe build is running there.

Philip Nienhuis <philipnienhuis>
Group Member
Thu 30 Oct 2014 12:32:27 PM UTC, comment #7: 

There is also a "large indexing" version available on http://mxeoctave.osuv.de/ - I should better try this one.

Armin Müller <arm_in>
Thu 30 Oct 2014 11:58:38 AM UTC, comment #6: 

If it's taking 19 minutes on your system, there's something wrong with your installation:


$ octave
octave-cli-3.8.2:1> tic; a1 = imread ("./Erik_1_2400dpi.tiff"); toc
warning: your version of GraphicsMagick limits images to 16 bits per pixel
Elapsed time is 1.08101 seconds.
octave-cli-3.8.2:2> tic; a2 = imread ("./Erik_1_4800dpi.tiff"); toc
Elapsed time is 4.68481 seconds.
octave-cli-3.8.2:3> tic; a3 = imread ("./Erik_1_6000dpi.tiff"); toc
Elapsed time is 7.49605 seconds.
octave-cli-3.8.2:4> whos
Variables in the current scope:

   Attr Name        Size                     Bytes  Class
   ==== ====        ====                     =====  =====
        a1       6062x8051                48805162  logical
        a2      12123x16102              195204546  logical
        a3      15153x20127              304984431  logical

Total is 548994139 elements using 548994139 bytes

octave-cli-3.8.2:5> octave_config_info ("USE_64_BIT_IDX_T")
ans = no


Carnë Draug <carandraug>
Group Member
Thu 30 Oct 2014 11:37:34 AM UTC, comment #5: 

Sure, I agree that Octave shouldn't crash.
But AFAIU out-of-memory errors are extremely hard to catch. In my own experience, processes that hit this kind of error are often doomed to crash sooner or later.
But maybe the core Octave devs know a trick.

As to "works for me" - the solution -a 64-bit indexing Octave- does work and that is what counted in my perception.
Unfortunately for you that solution is yet out of reach for you.
Hopefully a 64-bit & 64-bit indexing Octave for Windows will be available soon.
Using mxe-octave, it can be built OOTB these days by about anyone.

BTW, in the majority of cases memory fragmentation is far more important then plain amounts of installed RAM. Even on my 8 GB RAM laptop I've hit OOM errors when reading extremely large files even when it looked like there was sufficient free RAM.


Philip Nienhuis <philipnienhuis>
Group Member
Thu 30 Oct 2014 10:11:38 AM UTC, comment #4: 

Answering Rik' question:


>> x = ones (15153, 20127, "logical");
>> whos
Variables in the current scope:

   Attr Name        Size                     Bytes  Class
   ==== ====        ====                     =====  =====
        x       15153x20127              304984431  logical

Total is 304984431 elements using 304984431 bytes


In reference to Philip:

It is true that my Octave installation is compiled as 32bit. So the 16GB memory are useless for that purpose.


>> octave_config_info ("USE_64_BIT_IDX_T")
ans = no


But anyway, either there is enough memory and the script should run to the end. Or it should complain about insufficient memory and return to the command line. I find it unsatisfying to interpret a crash as "Works For Me". Probably, the issue is that 15k x 20k "logical" fit into memory very well. But matrix size is limited to approx. 9k x 9k "double" or 14k x 14k "int32". IMHO, there is some check within imread.m / _imread_.m / _magic_read_ that does not respect different memory requirements for different types.


>> x = ones (15153, 20127);
error: out of memory or dimension too large for Octave's index type


Armin Müller <arm_in>
Wed 29 Oct 2014 10:58:25 PM UTC, comment #3: 

I just tried to load the 6000 dpi variant in Octave-3.8.2 and Octave-3.9.0+ (both 64 bits w. 64-bit indexing) for Windows on a Core i5 laptop with 8 GB RAM.
It took Octave 19 minutes to load the image, with Octave's RAM usage slowly increasing from 2.5 to 4+ GB with a short initial peak of 5 GB. In the end, Octave's RAM footprint was only 350 KB and "whos" says that img is 304984431 bytes (class logical).

As this works fine here but with quite a bit of RAM usage (>4 GB) I strongly suspect that the Octave binary you used hasn't been built with --enable-64 flag, only with the --enable-windows-64 flag.
Try:
octave_config_info ("USE_64_BIT_IDX_T")
If the answer is "yes" Octave has been built with 64-bit indexing. If "no", you're out of luck.

As to the time required to read the image, I can only guess that imread is maybe not a very efficient routine for large files; >15 minutes just to accommodate a bitmap (your TIFF has a bit depth of 1) looks excessive. Perhaps Octave's imread (or rather the GraphicsMagic library) doesn't reserve the required RAM in one time but extends RAM usage slice by slice until the entire image has been processed (just my guess).

Closing report.

Philip Nienhuis <philipnienhuis>
Group Member
Wed 29 Oct 2014 04:22:26 PM UTC, comment #2: 

This also works for me on Ubuntu 12.04, 64-bit, 8 GB RAM, GrahpicsMagick 1.3.12, quantum depth 8.

I suspect that this is something strange with Windows and the GraphicsMagick version that you used.  Note that there isn't any official release from the Octave maintainers of an MXE build.  The website http://mxeoctave.osuv.de/ has a build but it is not run by us.

Just to round out one piece of testing, can you create a matrix as large as the image you are trying to read in?  Does this command complete successfully in Octave?


x = ones (15153, 20127, "logical");
whos


If so, then the problem lies with GraphicsMagick or Windows and the build of Octave you have.



Rik <rik5>
Group administrator
Wed 29 Oct 2014 01:25:51 PM UTC, comment #1: 

I can't replicate this issue in Debian (Octave built by myself using GraphicsMagick 1.3.20 built with quantum depth 16).


$ octave
octave-cli-3.8.2:1> a = imread ("./Erik_1_2400dpi.tiff" );
warning: your version of GraphicsMagick limits images to 16 bits per pixel
octave-cli-3.8.2:2> a = imread ("./Erik_1_4800dpi.tiff" );
octave-cli-3.8.2:3> a = imread ("./Erik_1_6000dpi.tiff" );
octave-cli-3.8.2:4>


Carnë Draug <carandraug>
Group Member
Wed 29 Oct 2014 12:53:25 PM UTC, original submission:  

I get some strange behaviour when reading large TIFF images using imread(). Using official MXE Octave 3.8.2-3 on Windows 7 64bit, 16GB of RAM.

Example:

1) Create large monochrome TIFF images. The example here is using Sourceforge "gerb2tiff" on the Windows command line.


% Windows Command Line

gerb2tiff.exe examplesErik_1.gbr --dpi=2400 -o examplesErik_1_2400dpi.tiff
gerb2tiff.exe examplesErik_1.gbr --dpi=4800 -o examplesErik_1_4800dpi.tiff
gerb2tiff.exe examplesErik_1.gbr --dpi=6000 -o examplesErik_1_6000dpi.tiff


I've attached the 3 example images.

2) Load large TIFF images in Octave


img=imread('Erik_1_2400dpi.tiff');        %  6062 x  8051
img=imread('Erik_1_4800dpi.tiff');        % 12124 x 16102
img=imread('Erik_1_6000dpi.tiff');        % 15153 x 20127


3) Results in Octave

2400dpi: OK


warning: your version of GraphicsMagick limits images to 8 bits per pixel


4800dpi: Either OK or warning, may crash later due to memory issues


error: imread: invalid image file: out of memory or dimension too large for Octave's index type
error: called from:
error:   C:Program Files (x86)OctaveOctave-3.8.2shareoctave3.8.2mimageprivate__imread__.m at line 181, column 7
error:   C:Program Files (x86)OctaveOctave-3.8.2shareoctave3.8.2mimageprivateimageIO.m at line 66, column 26
error:   C:Program Files (x86)OctaveOctave-3.8.2shareoctave3.8.2mimageimread.m at line 107, column 30


9600dpi: Crash immediately

Armin Müller <arm_in>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #32335:  Erik_1_6000dpi.tiff added by arm_in (611KiB - image/tiff)
file #32334:  Erik_1_4800dpi.tiff added by arm_in (443KiB - image/tiff)
file #32333:  Erik_1_2400dpi.tiff added by arm_in (187KiB - image/tiff)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by arm_in (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-10-29 philipnienhuis Open/ClosedOpen Closed
    2014-10-29 rik5 StatusNone Works For Me
    2014-10-29 arm_in Attached File- Added Erik_1_2400dpi.tiff, #32333
        Attached File- Added Erik_1_4800dpi.tiff, #32334
        Attached File- Added Erik_1_6000dpi.tiff, #32335

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code