bugGNU Octave - Bugs: bug #47707, hist takes up a lot of memory

 
 

bug #47707: hist takes up a lot of memory

Submitted by:  Carnë Draug <carandraug>
Submitted on:  Fri 15 Apr 2016 06:05:52 PM UTC  
 
Category: PerformanceSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Performance
Status: FixedAssigned to: None
Originator Name: Open/Closed: Closed
Release: devOperating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 16 Apr 2016 06:22:33 PM UTC, comment #3:

After investigation, there is no issue with saturation (or at least not anymore -- maybe there was in 2007). There was an issue when adding the bin edges (cutoff variable) to the array which caused the edges to be rounded. Fix was to floor the cut off points ourselves.

Fixed with http://hg.savannah.gnu.org/hgweb/octave/rev/225d88a97104 which adds several tests for class integer in the second argument.

Of special note that Matlab does not even support input of class integers.

Carnë Draug <carandraug>
Project Member
Sat 16 Apr 2016 02:07:16 PM UTC, comment #2:

> The script doesn't say what issues it is trying to avoid. Do you know what they are? I'd have thought that working with integers would be the way to go.


hg blame points to e8d953d03f6a . This precedes meaningful commit messages, but adds the following note on the ChangeLog file.

  • plot/hist.m: Avoid saturation when the x values are in an inetger type.

There are no tests in hist for integer input so I will write them and see if this conversion to double is really necessary or if it can be avoided in some cases.

> Alternatively, do you need a histogram of your whole 4-D image, or is a histogram over one or two dimensions? A compromise may be to convert only one row/column/page/... at a time to double.


No, I want a single histogram for the whole image (this is to compute global threshold value). I have worked around it by using histc like so:

> However, I thought that "hist" was being phased out by Matlab in favour of "histogram". (I haven't checked the differences.) Perhaps we should just write an efficient implementation of that and leave hist.


Doesn't mean that the function hist will be removed. I think it's been around for far too long and widely used to just be dropped like that. If I can also make it better, good.

---

Another issue that I didn't mention at the start is that somewhere in the code, this actually crashed Octave. Sometimes it would error about not having enough memory (fair enough), but roughly half of the times the kernel would just kill the process due to out-of-memory.

Carnë Draug <carandraug>
Project Member
Sat 16 Apr 2016 05:32:41 AM UTC, comment #1:

The script doesn't say what issues it is trying to avoid. Do you know what they are? I'd have thought that working with integers would be the way to go.

Alternatively, do you need a histogram of your whole 4-D image, or is a histogram over one or two dimensions? A compromise may be to convert only one row/column/page/... at a time to double.

However, I thought that "hist" was being phased out by Matlab in favour of "histogram". (I haven't checked the differences.) Perhaps we should just write an efficient implementation of that and leave hist.

Lachlan Andrew <lachlan>
Project Member
Fri 15 Apr 2016 06:05:52 PM UTC, original submission:

The function hist converts the input vector or matrix to class double to avoid any issues with integers

http://hg.savannah.gnu.org/hgweb/octave/file/1dda942a2514/scripts/plot/draw/hist.m

If the input is uint8 (very common in images), that's an 8x increase in memory usage. In my current case, I have a 480x480x15x28 image (uint16) which is roughly 190Mb but Octave needs 4GB of memory to make an histogram of the image.

Carnë Draug <carandraug>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by lachlan (Posted a comment)
  • -unavailable- added by carandraug (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 16 Apr 2016 06:22:33 PM UTCcarandraugStatusNone=>Fixed
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1