taskGNU Astronomy Utilities - Tasks: task #13759, Decrease memory usage

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #13759: Decrease memory usage

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Sat 03 Oct 2015 06:14:20 AM UTC
   
 
Should Start On:  Fri 02 Oct 2015 03:00:00 PM UTC Should be Finished on:  Fri 02 Oct 2015 03:00:00 PM UTC
Category:  MakeCatalog Priority:  5 - Normal
Item Group:  Enhancement Status:  Done
Privacy:  Public Assigned to:  makhlaghi
Percent Complete:  100% Open/Closed:  Closed
Effort:  0.00

Fri 06 Apr 2018 12:07:16 AM UTC, comment #3: 

This task has been completed as part of a larger process and pushed to the main repo. MakeCatalog can now read the Sky and Sky standard deviation datasets as a single-element-per-tile array. It will also only load datasets that are necessary for the requested columns.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 23 Mar 2016 05:37:18 AM UTC, comment #2: 

The first pass and second pass functions until now also did some post-processing. For example dividing the brightness weighted positions by the total brightness. This is now done while calculating the final columns.

So now the first pass and second pass functions only actually pass through the pixels and don't do any processing. This will later allow us to easily parallelize the process, for example, only reading part of the image, getting all the raw information we need and then freeing that space in parallel. This can greatly decrease the memory usage and (maybe, possibly not!) increase speed. In some cases, memory is more important than speed.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 19 Feb 2016 07:45:49 AM UTC, comment #1: 

task #13900 was defined as a more general solution/infrastructure for any Gnuastro utility that has similar memory problems on large images.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 03 Oct 2015 06:14:20 AM UTC, original submission:  

MakeCatalog currently needs to load a lot of data before starting. In the case of large images, this can easily be felt.

One big step is to make NoiseChisel store the Sky and Sky Standard deviation values on the mesh grid (one pixel per grid) as it does internally. And configure MakeCatalog to read that kind of Sky and Sky standard deviation too.

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-04-06 makhlaghi StatusNone Done
        Percent Complete0% 100%
        Assigned toNone makhlaghi
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code