taskGNU Astronomy Utilities - Tasks: task #14131, Utility to match catalogs of...


You are not allowed to post comments on this tracker with your current authentication level.

task #14131: Utility to match catalogs of objects and find warpings

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Thu 25 Aug 2016 08:13:53 AM UTC
Should Start On:  Wed 24 Aug 2016 03:00:00 PM UTC Should be Finished on:  Wed 24 Aug 2016 03:00:00 PM UTC
Category:  New program Priority:  5 - Normal
Item Group:  None Status:  Postponed
Privacy:  Public Percent Complete:  50%
Assigned to:  None Open/Closed:  Open
Effort:  0.00

( Jump to the original submission )

Thu 19 Mar 2020 06:37:55 PM UTC, comment #8: 

I just posted a general set of suggestions for the proposal on the main GSoC issue. Hopefully that should be good for some tips on how to organize your proposal.

In general, at this stage you don't need to go into the fine detail of the coding of astrometry.net, astroalign, or any other library/program you want to look into (details like "which particular files to be used and what features are to be implemented").

The same applies to the FITS and WCS standards.

The mere fact that you have read their papers/manuals, and have a general understanding of the problem and their code is enough to write an excellent proposal ;-).

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Thu 19 Mar 2020 12:52:32 PM UTC, comment #7: 

How should I prepare a draft proposal? I started to go through the code of astronomy.net but it is so vast to be evaluated within the given time.

For the wcs distortions, there are also many keywords to go through and I currently am a bit preoccupied with registration.

Can you give me specifications of the what to be put down in the project proposal like a basic layout of the project(like which particular files to be used and what features are to be implemented) and a bit specific idea of what should I write as a solution for the tasks in the proposal? It's a bit much but I'm preparing the draft now and will fill in the more detailed specification for review once a firm idea of what is to be done is specified:-)

Sachin Kumar Singh <sks_15>
Project Member
Thu 19 Mar 2020 12:22:14 AM UTC, comment #6: 

That was a great review Sachin and a good question ;-).

The main reason we want to implement this in Gnuastro is that astrometry.net's solution is only useful within astrometry.net: it is written in a monolithic style, not as high/low-level libraries that can be used in other contexts too.

So your GSoC project won't just be a re-write of astrometry.net's code, it will be more like a re-organization of it into a highly-modular (and well-documented) library structure within Gnuastro. But also, its not going to be merely astrometry.net, there is also astroalign and some other tools you also mentioned that also do good things like not needing a reference/final catalog. There are also somethings that astrometry.net doesn't implement, leading many to "correct" it's output with SCAMP.

We need to have a robust modular library structure to do all of these things, while also having a high-level program that is easy to use and configure.

But all of this can be just the "image registration" part of your GSoC proposal. To complete it, you also need tools to do non-linear warpings and apply the WCS/registration on to the pixel grid which is necessary to later compare the images with each other.

All of these are currently done in multiple software that are often not compatible (they use different distortion standards in the WCS), all needing many dependencies. So this GSoC project to enable this within Gnuastro will be useful to many!

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Wed 18 Mar 2020 09:53:55 PM UTC, comment #5: 

Sorry comment#25 task #13658 was meant to be here.

Sachin Kumar Singh <sks_15>
Project Member
Sun 15 Mar 2020 11:16:26 PM UTC, comment #4: 

Indeed, this is arguably one of the top things I had in mind when I wrote the GSoC general ideas! It is a critical step to add to Gnuastro for reaching the version 1.0 vision.

There is also another good resource/reference you can use: astrometry.net, its main paper is freely available (as PDF if you like) on the Astronomical Journal. Its source code can also be a good reference.

`astrometry.net' goes a step further than `astroalign' and even assigns World Coordinate System metadata to the image because it compares with an external catalog, not another image.

In fact, given a good reference catalog, it is also possible to include distortions (both due to the optical and sky projection effects) in the WCS headers is well. That will then enable you to merge this task with task #13565 (Add non-linear warpings) in your proposal.

If you define the various elements in a very low-level/modular way (like the rest of Gnuastro), you can easily merge these two tasks and fill this largest remaining gap for the release of Gnuastro version 1.0 ;-).

In fact this is the main problem with those tools mentioned above: they have done a good job, but in a very high-level language, with MANY dependencies, and in a non-modular way. So their hard work can't be used by others easily! However, if we define it as a modular C library in Gnuastro, wrappers can easily be written to use it in many other languages and programs.

For example, ultimately this particular task is a problem of matching catalogs (possibly generated from images), not images! But since the high-level interface of these programs accepts images, we don't have control over how the catalog is extracted from it.

In short, I think this will be a great project to do, and can be very useful to a very large community ;-).

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sun 15 Mar 2020 10:43:57 PM UTC, comment #3: 

I read the paper and also saw its Python implementation. This is a big feature to implement and I can add this to my proposal.

For this, I need to implement all the basic python modules used in the python implementation in C and also make the required data structures(kd-tree for example) for different algorithms involved.

More detailed implementation discussion is required. Any other thoughts?

Sachin Kumar Singh <sks_15>
Project Member
Tue 10 Mar 2020 02:31:27 PM UTC, comment #2: 

arXiv:1909.02946 provides an algorithm to find the necessary warp which may be interesting.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sat 02 Dec 2017 07:49:45 PM UTC, comment #1: 

The new Match program, will now match two catalogs based on positions.

The most recent tarball (which now includes Match from now on) can always be checked at http://akhlaghi.org/gnuastro-latest.tar.lz

Its matching is actually done through a library function that can later be used to find the proper warping between two images to register them.

Later, we can also implement a feature that given two images (instead of tables), it finds the warping matrix that must be applied to the second input so it comes to the same grid as the first. The actual warping can later be done by Warp.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Thu 25 Aug 2016 08:13:53 AM UTC, original submission:  

Due to issues like the cosmic rays and also to increase the resolution and cover contiguous wide regions, astronomical images are often taken in short exposures. Each short exposure is usually "dithered" (slightly moved) compared to the previous exposure for better coverage and resolution.

Therefore the separate exposures do not match or align with each other. The hardware information (like the telescope pointing) are often not enough compared to the very high pixel resolution. One way to do the alignment is this:

1. Make catalogs of sufficiently bright sources that are imaged in the separate exposures accurately (ideally stars which are the sharpest objects, thus making their center best defined).

2. Match those catalogs (find the corresponding objects in separate images), by finding the warping matrix which can minimize the distance between the matches. GSL has good minimization algorithms.

3. Warping the pixels in each image with the relevant warping matrix.

In Gnuastro, MakeCatalog and ImageWarp can finely do the first and third steps. But we don't currently have a utility for the second step.

I have been working on a small C program to match two catalogs within a given radius, but currently it only identifies which objects of which catalog are closest to which object in the other. It can deal with multiple objects being present within the search radius of both catalogs.

So the utility for the second step above would have to progressively apply warpings to the coordinates of one catalog and also select different maximum search distances (which would ideally shrink) such that the RMS distance between the catalogs is minimized. So the free parameters which need to be minimized are the warping matrix and the maximum distance.

I currently don't have time to implement this immediately, but if anyone is interested, I am willing to help (and share that small C program to build upon).

Mohammad Akhlaghi <makhlaghi>
Project Administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by sks_15 (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.




    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-12-02 makhlaghi Percent Complete10% 50%

    Back to the top

    Powered by Savane 3.11