taskGNU Astronomy Utilities - Tasks: task #13759, Decrease memory usage

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #13759: Decrease memory usage

Submitted by:  Mohammad Akhlaghi <makhlaghi>
Submitted on:  Sat 03 Oct 2015 06:14:20 AM UTC  
 
Should Start On:  Fri 02 Oct 2015 03:00:00 PM UTC Should be Finished on:  Fri 02 Oct 2015 03:00:00 PM UTC
Category:  MakeCatalog Priority:  5 - Normal
Item Group:  Enhancement Status:  Done
Privacy:  Public Percent Complete:  100%
Assigned to:  Mohammad Akhlaghi <makhlaghi> Open/Closed:  Closed
Effort:  0.00

Fri 06 Apr 2018 12:07:16 AM UTC, comment #3:

This task has been completed as part of a larger process and pushed to the main repo. MakeCatalog can now read the Sky and Sky standard deviation datasets as a single-element-per-tile array. It will also only load datasets that are necessary for the requested columns.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Wed 23 Mar 2016 05:37:18 AM UTC, comment #2:

The first pass and second pass functions until now also did some post-processing. For example dividing the brightness weighted positions by the total brightness. This is now done while calculating the final columns.

So now the first pass and second pass functions only actually pass through the pixels and don't do any processing. This will later allow us to easily parallelize the process, for example, only reading part of the image, getting all the raw information we need and then freeing that space in parallel. This can greatly decrease the memory usage and (maybe, possibly not!) increase speed. In some cases, memory is more important than speed.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Fri 19 Feb 2016 07:45:49 AM UTC, comment #1:

task #13900 was defined as a more general solution/infrastructure for any Gnuastro utility that has similar memory problems on large images.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Sat 03 Oct 2015 06:14:20 AM UTC, original submission:

MakeCatalog currently needs to load a lot of data before starting. In the case of large images, this can easily be felt.

One big step is to make NoiseChisel store the Sky and Sky Standard deviation values on the mesh grid (one pixel per grid) as it does internally. And configure MakeCatalog to read that kind of Sky and Sky standard deviation too.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-04-06 makhlaghi StatusNone => Done
        Percent Complete0% => 100%
        Assigned toNone => makhlaghi
        Open/ClosedOpen => Closed

    Back to the top


    Powered by Savane 3.3