bugGNU Octave - Bugs: bug #43495, Crash when reading large TIFF with...

 
 

bug #43495: Crash when reading large TIFF with imread()

Submitted by:  Armin Müller <arm_in>
Submitted on:  Wed 29 Oct 2014 12:53:25 PM UTC  
 
Category: Octave FunctionSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Segfault, Bus Error, etc.
Status: Works For MeAssigned to: None
Originator Name: Open/Closed: Closed
Release: 3.8.2Operating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 04 Nov 2014 04:15:39 PM UTC, comment #15:

Slightly OT but related tot RAM usage:

At work we run the network installation of Matlab in a "bubble", a sort of virtual environment.
That gives considerably better results as to RAM usage (but don't get me started on CPU performance :-) ):

(compare to the results in comment #13)

Maybe that can be made into a useful hint (I cannot actually confirm it here) to get max. array sizes: i.e., to run Octave in a virtual machine in a guest environment stripped down to the bare minimum required to run user programs.

When I get time for it I'll put something about RAM usage and OOM errors on the wiki.

Philip Nienhuis <philipnienhuis>
Project Member
Fri 31 Oct 2014 02:22:38 PM UTC, comment #14:

Your build of GraphisMagick may be the limitation here. The way you built it can change your results. On my case, it's built with quantum 16 so it reads it as uint16 image (and probably RGB as well). Octave receives it and changes it back into what it really is, a logical matrix. In your case, it may be reading it as an uint8 RGB image.

Ideally, you'd implement the missing Tiff class which would simply be a wrapper to libtiff. This allows for a finer control while reading the image file. This requires classdef will be released with Octave 4.2.

Alternatively, you can:

1) try using the "PixelRegion" of imread() to read it in blocks and make your image out of its pieces. Use iminfo to get the image size, and then fill it in blocks.

2) implement a tiff reader for your files. I've done it before to read very special Tiffs. It's not that hard and there's plenty of examples on the internet written as m files (writing a general tiff is hard, but writing for a very specific subset can be really easy).

Carnë Draug <carandraug>
Project Member
Fri 31 Oct 2014 12:43:46 PM UTC, comment #13:

@Armin:
"..far less than I would expect".... Well I have some less rosy info for you then ;-)

To illustrate what I wrote at the bottom of comment #6, here's some output from Matlab r2014b prerelease on my WinXP 32bit box w. 4 GB RAM:

Just after starting up WinXP, first ML command:

and after some work:

For comparison, Octave 4.1.0+ CLI allowed me to squeeze a double array of up to 8341x8341 into memory, max. 556 MB (or MiB?).

(To be honest, these results disappoint me, I had expected max. array sizes would be 600-700 MB. But: no.)

In conclusion I think you're even quite lucky to get 0.73 GB arrays into Octave on a 32-bit system .... (but on the bright side: Octave compares quite good to the competition...)

Philip Nienhuis <philipnienhuis>
Project Member
Fri 31 Oct 2014 08:33:53 AM UTC, comment #12:

I've tried the "large indexing" version of Octave. Everything runs fine. That's the good point.

Unfortunately, I've also tried to fathom the maximum image size allowed in this version. Typing
did lock up my system completely. No warning, no error, just endlessly swapping memory on the HD, not even the task manager did respond. I had to do a "hard" reboot.

As written below, 14k x 14k "int32" = 0.73 GB is the maximum I can get on the 32 bit system. Alternatively 9k x 9k "double" = 0.60 GB. The exact value varies, it seems that it is heavily depending on the preconditions if the memory is absolutely "clear" and Octave started newly, or if it is "used". But both values are far less than I would expect.

Armin Müller <arm_in>
Fri 31 Oct 2014 12:19:09 AM UTC, comment #11:

@Arun: Try the large index version of Octave and report back if it makes a difference.

The problem from Octave's perspective is that we do not know the return size of the image ahead of time. The whole point of using the external GraphicsMagick package is to take care of all the annoying image formats. Sometimes, for example, an RGB JPEG image will be returned as a grayscale image (1/3 the memory) if the R, G, and B planes all have the same value.

So, my guess is that GraphicsMagick itself is trying to load the file, and choosing a format that is too large initially, which leads to a segfault. On a successful load the end class is logical, or just 1B per pixel, but it may be loading it at TrueColor depth of 4B per pixel first before slimming it down.

As Armin tested,

This sure looks like an indication that it is exceeding a 32-bit range. Note that for int32 the top-bit is often used as a valid bit so the real range is just to 2^31 which is 2.1GB.

I suppose a way to be certain is to try

This would require only 1.2GB which should not cause any problems.

Rik <rik5>
Project Administrator
Thu 30 Oct 2014 10:18:16 PM UTC, comment #10:

@Carnë / comment #6:

With a freshly built 3.9.0+ (--enable-64) for MingW, using Markus' patch (bug #43505), I get similar results as you:

but then:
and although Octave may be ready, only the outline of the graphics window is drawn and Octave doesn't return to the prompt, instead the hard disk started crunching. After half an hour I gave up and killed Octave.
Windows' Task Manager indicated ~6.7 GB of the available 8 GB is used. Even with "stolen RAM" (video) and drivers address space I think there's enough RAM available.

Similar goes for "spy (img)"

Philip Nienhuis <philipnienhuis>
Project Member
Thu 30 Oct 2014 07:16:32 PM UTC, comment #9:

Well I spoke too soon - to be able to test Octave on Linux with --enable-64 the entire dependency chain has to be rebuilt (using mxe), see bug #43319 comment #19.
That'll take too much time ATM. Maybe maybe later this weekend.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 30 Oct 2014 05:42:21 PM UTC, comment #8:

@Carnë / comment #6:

Maybe. Mine are just plain mxe-octave builds.
So what could be wrong? all dependencies have been built with the same tool chain.
What I did see is that "spy (img)" and "imshow (img)" were very fast in 3.9.0+ but only showed one star in the center; on 3.8.2 both took ages and in the end a dark rectangle appeared; zooming in there lead to a blank picture. (AFAIK the actual picture should be a electronics component board.)
So, not good. But before entering a bug report I'll investigate further.

Later on I'll try on Linux, but presently an mxe build is running there.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 30 Oct 2014 12:32:27 PM UTC, comment #7:

There is also a "large indexing" version available on http://mxeoctave.osuv.de/ - I should better try this one.

Armin Müller <arm_in>
Thu 30 Oct 2014 11:58:38 AM UTC, comment #6:

If it's taking 19 minutes on your system, there's something wrong with your installation:

Carnë Draug <carandraug>
Project Member
Thu 30 Oct 2014 11:37:34 AM UTC, comment #5:

Sure, I agree that Octave shouldn't crash.
But AFAIU out-of-memory errors are extremely hard to catch. In my own experience, processes that hit this kind of error are often doomed to crash sooner or later.
But maybe the core Octave devs know a trick.

As to "works for me" - the solution -a 64-bit indexing Octave- does work and that is what counted in my perception.
Unfortunately for you that solution is yet out of reach for you.
Hopefully a 64-bit & 64-bit indexing Octave for Windows will be available soon.
Using mxe-octave, it can be built OOTB these days by about anyone.

BTW, in the majority of cases memory fragmentation is far more important then plain amounts of installed RAM. Even on my 8 GB RAM laptop I've hit OOM errors when reading extremely large files even when it looked like there was sufficient free RAM.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 30 Oct 2014 10:11:38 AM UTC, comment #4:

Answering Rik' question:

In reference to Philip:

It is true that my Octave installation is compiled as 32bit. So the 16GB memory are useless for that purpose.

But anyway, either there is enough memory and the script should run to the end. Or it should complain about insufficient memory and return to the command line. I find it unsatisfying to interpret a crash as "Works For Me". Probably, the issue is that 15k x 20k "logical" fit into memory very well. But matrix size is limited to approx. 9k x 9k "double" or 14k x 14k "int32". IMHO, there is some check within imread.m / _imread_.m / _magic_read_ that does not respect different memory requirements for different types.

Armin Müller <arm_in>
Wed 29 Oct 2014 10:58:25 PM UTC, comment #3:

I just tried to load the 6000 dpi variant in Octave-3.8.2 and Octave-3.9.0+ (both 64 bits w. 64-bit indexing) for Windows on a Core i5 laptop with 8 GB RAM.
It took Octave 19 minutes to load the image, with Octave's RAM usage slowly increasing from 2.5 to 4+ GB with a short initial peak of 5 GB. In the end, Octave's RAM footprint was only 350 KB and "whos" says that img is 304984431 bytes (class logical).

As this works fine here but with quite a bit of RAM usage (>4 GB) I strongly suspect that the Octave binary you used hasn't been built with --enable-64 flag, only with the --enable-windows-64 flag.
Try:
octave_config_info ("USE_64_BIT_IDX_T")
If the answer is "yes" Octave has been built with 64-bit indexing. If "no", you're out of luck.

As to the time required to read the image, I can only guess that imread is maybe not a very efficient routine for large files; >15 minutes just to accommodate a bitmap (your TIFF has a bit depth of 1) looks excessive. Perhaps Octave's imread (or rather the GraphicsMagic library) doesn't reserve the required RAM in one time but extends RAM usage slice by slice until the entire image has been processed (just my guess).

Closing report.

Philip Nienhuis <philipnienhuis>
Project Member
Wed 29 Oct 2014 04:22:26 PM UTC, comment #2:

This also works for me on Ubuntu 12.04, 64-bit, 8 GB RAM, GrahpicsMagick 1.3.12, quantum depth 8.

I suspect that this is something strange with Windows and the GraphicsMagick version that you used. Note that there isn't any official release from the Octave maintainers of an MXE build. The website http://mxeoctave.osuv.de/ has a build but it is not run by us.

Just to round out one piece of testing, can you create a matrix as large as the image you are trying to read in? Does this command complete successfully in Octave?

If so, then the problem lies with GraphicsMagick or Windows and the build of Octave you have.

Rik <rik5>
Project Administrator
Wed 29 Oct 2014 01:25:51 PM UTC, comment #1:

I can't replicate this issue in Debian (Octave built by myself using GraphicsMagick 1.3.20 built with quantum depth 16).

Carnë Draug <carandraug>
Project Member
Wed 29 Oct 2014 12:53:25 PM UTC, original submission:

I get some strange behaviour when reading large TIFF images using imread(). Using official MXE Octave 3.8.2-3 on Windows 7 64bit, 16GB of RAM.

Example:

1) Create large monochrome TIFF images. The example here is using Sourceforge "gerb2tiff" on the Windows command line.

I've attached the 3 example images.

2) Load large TIFF images in Octave

3) Results in Octave

2400dpi: OK

4800dpi: Either OK or warning, may crash later due to memory issues

9600dpi: Crash immediately

Armin Müller <arm_in>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #32333:  Erik_1_2400dpi.tiff added by arm_in (187KiB - image/tiff)
file #32334:  Erik_1_4800dpi.tiff added by arm_in (443KiB - image/tiff)
file #32335:  Erik_1_6000dpi.tiff added by arm_in (611KiB - image/tiff)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by arm_in (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 29 Oct 2014 10:58:25 PM UTCphilipnienhuisOpen/ClosedOpen=>Closed
    Wed 29 Oct 2014 04:22:26 PM UTCrik5StatusNone=>Works For Me
    Wed 29 Oct 2014 12:53:25 PM UTCarm_inAttached File-=>Added Erik_1_2400dpi.tiff, #32333
      Attached File-=>Added Erik_1_4800dpi.tiff, #32334
      Attached File-=>Added Erik_1_6000dpi.tiff, #32335

    Back to the top


    Powered by Savane 3.1-cleanup1