taskGNU Astronomy Utilities - Tasks: task #14639, Select rows that are within a...


You are not allowed to post comments on this tracker with your current authentication level.

task #14639: Select rows that are within a given polygon

Submitted by:  Mohammad Akhlaghi <makhlaghi>
Submitted on:  Sun 24 Sep 2017 11:26:37 PM UTC  
Should Start On:  Sun 24 Sep 2017 10:00:00 PM UTC Should be Finished on:  Sun 24 Sep 2017 10:00:00 PM UTC
Category:  Table Priority:  5 - Normal
Item Group:  Enhancement Status:  Done
Privacy:  Public Percent Complete:  100%
Assigned to:  Sachin Kumar Singh <sks_15> Open/Closed:  Closed
Effort:  0.00

( Jump to the original submission)

Sun 10 May 2020 05:42:19 PM UTC, comment #19: 

Thanks a lot Sachin, I done some further modifications, added an `--outpolygon' option also and pushed it as one commit to Gnuastro's master branch. I marked this task as closed/complete.

Before deleting your original commit, please inspect the `diff' with this final commit on the master branch with your original commit see the modifications I made. They contain some good points which can be educative for you as you get more involved in Gnuastro ;-).

Maybe one interesting thing, which is not unrelated to your GSoC project, is to think about the necessary changes to enable a similar option in a spherical space, not just Euclidean.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sun 10 May 2020 03:27:47 PM UTC, comment #18: 

All requirements are done and pushed here. Docs are also added:)

Sachin Kumar Singh <sks_15>
Project MemberIn charge of this item.
Sun 10 May 2020 01:43:25 AM UTC, comment #17: 

Just a small point on the current implementation: you don't know the type of the two columns, they may be anything: uint8_t, int16_t, or float64. They may also have different types: one of them uint8_t, another float64!

So you should also write a step to write the values of each row into the `point' array (which is float64), based on the different types of each column. You can see several examples of such tests with a `switch' statement on `col->type' in `lib/statistics.c'.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sat 09 May 2020 01:00:50 AM UTC, comment #16: 

I don't understand why you think the ID column is required? If a user wants the ID, they will ask for it with the `--column' option. it is unrelated to the coordinates and if they are in the polygon or not!

Some tables may not have any ID column at all ;-)!

As I mentioned before this option is very similar to the `--range' option (that only takes one input column) and takes the parameters afterwards. For example try this command to find the RA, DEC and SPECZ column values of the rows where the `MAG_F225W' is between 29 and 30.

asttable uvudf_rafelski_2015.fits.gz --range=MAG_F225W,29,30 -cRA,DEC,SPECZ

You see, there is no ID involved, but it works, if you want the ID, you can change the `-c' option value to `RA,DEC,SPECZ,ID' or `RA,DEC,ID,SPECZ' (or any order you want).

Its the same here.

If its not clear, we can talk by video-chat.

Some other side-notes:

  • Please put your commit directly over my commit, after fetching your recent commit, I noticed that the hash of my commit has changed and your commit has come over the modified commit. I don't have the time to go look to see what has been changed in my original commit. So to help us collaborate better/faster, in such scenarios, please be sure that your commit is exactly over my original commit so I can focus exactly on what you have done.
  • About style issues like extra space and comment style, the whole purpose of such standards is to avoid bugs and make the code easier to read. So in fact, the most important time they should be implemented is during the coding, not after it ;-). Please configure your text editor to remove any trailing white-space when the file is being saved and correctly write the comments from the start. Otherwise this different style again doesn't let me focus on your actual work.
Mohammad Akhlaghi <makhlaghi>
Project Administrator
Fri 08 May 2020 10:13:40 PM UTC, comment #15: 

Pushed new changes here. As the ID column was required to be stored in the output, I made use of a third column. Then used a temporary array as the number of rows would be unknown until they are passed through `gal_polygon_is_inside`. Printing the values of column showed the right output.

The only problem is that if I use the `outsize` as the size in allocation, it conflicts with the `mask` in `table_selection` in size. If I use `p->table->dsize` the value is too large to be allocated. The error was there before I decided to use a third column so it is a cause if it. Where is the size going wrong?

Also, this is just a temporary commit fo review. I'll remove it in rebase and hence have not properly followed all the convection. Please ignore any stray whitespace or indentation. I'll fix it before rebasing:)

Sachin Kumar Singh <sks_15>
Project MemberIn charge of this item.
Fri 08 May 2020 02:27:30 AM UTC, comment #14: 

Thanks for starting work on this Sachin, I guess this problem was too hard for this stage, so I done all the major work in this commit on my development-fork of Gnuastro. It is over your branch, so if you add this fork to your remotes and pull it, you'll get the commit and can build upon it.

I have generally explained the things I did in the commit message, but please go through all the changes and try to understand them. The good thing is that the same interface that reads the `--polygon' option in Crop now also reads the polygon option in Table. This is how Gnuastro's library has grown over the years, everytime I find that something is repeated between two programs, I bring it up to the library to be generally useful.

I also modified the `tests/during-dev.sh' script so it uses a real polygon ;-)! With `--inpolygon=1,2', you had set it to try to find a polygon in the two ID columns of the table (`ID' and `COE_ID')!!! They weren't coordinates!

Another minor point: please don't use the `//' style of comments, if you look around in Gnuastro you won't see such comments anywhere! We use the `/* */' style of comments ;-).

So study all the changes in my commit to understand the solution. Being able to easily add options to programs, will make you so powerful with Gnuastro, to add new features, its worth it ;-). If you don't understand any part, don't hesitate to ask ;-).

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Thu 07 May 2020 09:46:52 PM UTC, comment #13: 

I pushed the code here.

But this is far from complete. I created the --inpolygon to store the column number of RA and DEC columns in the `points` array. But I'm still confused about how to go through all the rows of RA and DEC column. I get it that the first for loop in the `table_selection` does this, but it does that for a single column stored in `tmp`. Therefore, how can I specify the column numbers stored in `inpolygon` option in gal_data_t in the `table_selection`?

Also, in option.c there seems to be no method which reads only`,` or `:` separated float values. The CSV string reading option just seems to print the whole polygon vertices array as-is. In the range option, there is the column name specified hence the name and float reading option works, but this doesn't seem to be the case here. So should I change my approach?

Sachin Kumar Singh <sks_15>
Project MemberIn charge of this item.
Tue 21 Apr 2020 08:57:53 PM UTC, comment #12: 

I can give you the exact procedure, but that will take more time than actually doing it myself ;-)! Besides, its a good excuse to get more familiar with the internal Gnuastro program implementation.

As I mentioned before, please follow what the Table program does for the `--range' option, the whole the process for this operation is identical for this option (except the checking function).

If there is something that you don't understand about the implementation of the `--range' option, please feel free to ask ;-).

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Tue 21 Apr 2020 07:43:22 PM UTC, comment #11: 

How can I read the RA and DEC columns values in my function? Also what column should I use for traversal in `table_selection`?

Sachin Kumar Singh <sks_15>
Project MemberIn charge of this item.
Tue 21 Apr 2020 12:50:38 AM UTC, comment #10: 

There are ways to do it in a single option, but that require a lot of customization.

So maybe for a start you can break it into two options, for example in a call like this:

asttable table.fits --inpolyon=X_IMG,Y_IMG --polygon=X1,Y1:X2,Y2:X3,Y3:X4,Y4

You can then use existing options reading tools

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Mon 20 Apr 2020 09:41:05 PM UTC, comment #9: 

Sorry, some important college work came up and had to complete it. But I've started working on this task now. After some brainstorming and a rough implementation, following problems arose:

  • In **structs tableparams**, I included a `gal_data_t *inpolygon` to keep the vertices of the polygon. But how are the required columns (RA and DEC) should be included? If I define 2 new `list_select` in the internal section, it might introduce redundancies and overkills. Should we define it internally then, like without options in the function in table.c?
  • In args.h I introduced a `inpolygon` option but if I parse it as a string there will be problems when using `gal_polygon_is_inside` as it only takes double values. Is there any preexisting function to convert the strings into double arrays? (Solving this problem may solve many of the listed problems)
  • As the `table_selection` function in table.c goes through all the values in the desired columns, it can be easy implementation to just only tableparams p option in our function as we have to use only RA and DEC so giving any choice won't matter. But can we set the columns to RA and DEC in the functions? Using if-else for list_select to check if the column is RA or DEC seems inefficient. What do you think?
Sachin Kumar Singh <sks_15>
Project MemberIn charge of this item.
Thu 16 Apr 2020 01:44:49 AM UTC, comment #8: 

Let me give you a real-world example:

Here is a catalog from the Hubble Space Telescope Ultra-Deep Field (UDF): https://asd.gsfc.nasa.gov/UVUDF/uvudf_rafelski_2015.fits.gz

Its a catalog of sources in the same field that hosts that XDF image that was in the second tutorial. It has 97 columns, you can see them with this command:

asttable uvudf_rafelski_2015.fits.gz -i

Just for the record, here is the image of this field in one of the filters: http://archive.stsci.edu/pub/hlsp/xdf/hlsp_xdf_hst_wfc3ir-60mas_hudf_f160w_v1_sci.fits

When you look at the image, you see that the central polygon is much deeper than the rest of the field (there has been more image to stack, thus we have less noise).

The polygon defining this deep region was given in the tutorial (given to Crop).

This is an EXTREMELY SMALL field (its the deepest image we have of the universe after all!) so spherical corrections can be ignored and you can just assume that RA and DEC are on a flat plane.

Now, the problem is this: I know the polygon vertices of the deep field (from the tutorial above), and I have the table above (in particular its `RA' and `DEC' columns. I want to pull out the IDs (first column in that table) of the objects within the deep field.

Did you go through the implementation of the `--range' option I mentioned before? This will be almost identical. Just instead of looking at the values of one column, you have to look at two, and you have to call your new `gal_polygon_is_inside' function.

Use the scenario above to test your implementation ;-).

Do you think you can do it in the next few days? I am planning to make an alpha release shortly (in preparation for version 0.12), it would be good if this feature can go in it ;-).

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Tue 14 Apr 2020 04:57:23 PM UTC, comment #7: 

So essentially I'm given a column with X and Y coordinates(I think they are represented in spherical coordinates in the fits files as there are MAGNITUDE, RA, DEC columns in HDU's when I did an `asttable table.fits -i`, right?). But for now I assume them to be in rectangular coordinates (X,Y). So we are roughly given data in the  following format:

X1    |  Y1
X2    |  Y2
.     |  .
.     |  .

As assuming the coordinates are represented in spherical form, then X and Y are in same rows.

Now if the data is as above, then to find the region, I simply have to find the positions of uppermost and lowermost coordinates and eliminate all other rows, except the ones within the upper and lower limits. Is that right? Am I missing something?

Sachin Kumar Singh <sks_15>
Project MemberIn charge of this item.
Thu 09 Apr 2020 06:36:50 PM UTC, comment #6: 

Sorry, a few typo corrections in my previous comment:

  • The `--range' feature in Table isn't an arithmetic operator, its an option.
  • The column names in the value to the `--inpolygon' option are `X_IMG' and `Y_IMG'. I mistakenly write `X_COL' and `Y_COL' in the text.
  • Besides following the program's flow, look how similar options in Table or other programs (for example the `--polygon' option in Crop) are implemented.

See you next week ;-).

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Thu 09 Apr 2020 06:32:18 PM UTC, comment #5: 

In my previous comment, I had mentioned this as a column arithmetic operator. But now that I think more about, it is more like the `--range' operator (where we remove some rows based on a certain criteria).

What I am think of now is to call this function something like below, assuming that the desired coordinates are in the `X_COL' and `Y_COL' columns and that the polygon vertices are (X1,Y1), (X2,Y2), (X3,Y3) and (X4,Y4).

asttable table.fits --inpolyon=X_IMG,Y_IMG,X1,Y1:X2,Y2:X3,Y3:X4,Y4

The parsing of the values to the option can be done with the `gal_options_parse_*' functions that deal with strings (defined in `lib/options.c' and declared in `lib/gnuastro-internal/options.h').

Essentially, I would recommend to look into how the `--range' option is implemented in table. Once you understand that, you can start defining this function and work on the implementation of it.

Just please note that I am terribly busy during the next seven days, I will address any Gnuastro issue on Savannah after that. So please feel free to read the code and experiment on it, I am sure you will do it fast ;-). The hardest/newest part for you may be the parsing of the values on the command-line, but I am sure you'll figure it out fast: just follow the program's flow.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Thu 09 Apr 2020 05:57:28 PM UTC, comment #4: 

Which files should I look into for this task?

Sachin Kumar Singh <sks_15>
Project MemberIn charge of this item.
Sun 29 Mar 2020 06:51:27 PM UTC, comment #3: 

Good point! We now have all the low-level tools for this task!

Previously I had mentioned this task as a separate program, but given all the improvements in Gnuastro since 2.5 years ago, I think it now fits nicely within the Table program (as a Column Arithmetic operator.

After all the input will always be a table, the user just has to specify the columns containing the coordinates and the polygon in the same units as the coordinates.

The only complication that I can think of now is that Right Ascension and Declination are defined on the spherical coordinate system of the sky, so if the region is sufficiently large, a Euclidean-space line between the vertices may miss some points. To address this, we have to add a spherical implementation of the `gal_polygon_is_inside' function.

But until then, we can say that it assumes a euclidean space and implement the low-level column arithmetic operator (very easy!). For spherical coordinates, we should define a new operator.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sun 29 Mar 2020 06:03:06 PM UTC, comment #2: 

What other features are left to be implemented on this task?

Sachin Kumar Singh <sks_15>
Project MemberIn charge of this item.
Thu 28 Sep 2017 11:12:22 PM UTC, comment #1: 

The region covered by most surveys can be represented with a convex polygon (see gal_polygon_ordered_corners).

However, some larger surveys (like the GOODS or CANDELS) can have saw-like borders which are represented by a concave polygon.

Therefore to be generic, this task is not independent of task #13658 (Work on concave polygons too). Once `gal_polygon_ordered_corners' also works on concave polygons (task #13658 is complete), the suggested program by this task will also work on any generic survey field.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sun 24 Sep 2017 11:26:37 PM UTC, original submission:  

Astronomical surveys are limited to a certain area of the sky (which can usually be represented nicely as the coordinates of a polygon).

While working on a catalog of objects, some of the objects might fall within the region of the survey we want to study and some might fall outside of it.

It would thus be very useful to have a program in Gnuastro that takes a region and a catalog of objects and identifies which of the objects are inside (or outside) that region.

Just as an initial brainstorm the program interface might look like the following:

$ astregion --region=region.txt catalog.txt

The `region.txt' file could be a standard SAO DS9 region output which is very commonly used. It will work generally on any coordinate system (as long as the region and catalog have the same coordinates, like both are RA-Dec). But we must ensure that it works on RA-Dec coordinates (where the region/polygon cover pass the equator or celestial pole).

Ofcourse, the input region doesn't have to be a polygon, any user interface for any shape can be defined. Following ds9's region file convention should be a good start.

This task was defined during a discussion with Leindert Boogaard.

Mohammad Akhlaghi <makhlaghi>
Project Administrator


No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by makhlaghi
  • -email is unavailable- added by sks_15 (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-05-10 makhlaghi StatusPostponed => Done
        Percent Complete0% => 100%
        Open/ClosedOpen => Closed
        Carbon-Copy- => Added boogaard@strw.leidenuniv.nl
    2020-05-09 makhlaghi SummaryTable arithmetic operator to find rows within given region => Select rows that are within a given polygon
    2020-04-21 makhlaghi Carbon-CopyRemoved -email is unavailable- => -
    2020-04-09 makhlaghi Assigned toNone => sks_15
    2020-03-29 makhlaghi CategoryNew program => Table
        SummaryProgram to identify points inside/outside a region => Table arithmetic operator to find rows within given region
    2017-09-24 makhlaghi Carbon-Copy- => Added boogaard@strw.leidenuniv.nl

    Back to the top

    Powered by Savane 3.5