GNU Astronomy Utilities - Bugs: bug #59904, Large aperture can easily fill...
You are not allowed to post comments on this tracker with your current authentication level.
bug #59904: Large aperture can easily fill memory in sort-based match
Submitter: | Mohammad Akhlaghi <makhlaghi> | ||
Submitted: | Mon 18 Jan 2021 01:08:10 PM UTC | ||
Category: | Match | Severity: | 3 - Normal |
Item Group: | Crash | Status: | Confirmed |
Privacy: | Public | Assigned to: | None |
Open/Closed: | Open |
Mon 18 Jan 2021 01:11:14 PM UTC, comment #1: |
Mohammad Akhlaghi <makhlaghi>![]() |
Mon 18 Jan 2021 01:08:10 PM UTC, original submission:
The sort-based match algorithm which is currently used in the Match program makes a linked list of nearby points (within the given aperture) between the two catalogs to find the best match between them. This is necessary to make sure that the order of the input catalogs doesn't affect the final result (see the comments in the 'match_coordinates_second_in_first' function for more).
However, this has a bad side-effect: when there are many points in both catalogs (for example x100000) and the aperture is large (by mistake usually!), the created lists for each point in each catalog can easily fill the whole system RAM causing Match to crash!
For example, with these two commands we can download the ID, RA and Dec of the same region (randomly selected) in Gaia DR2 and eDR3 (each is 72Mb, with more than 2 million rows):
When we later try to match these two by RA and Dec with the command below, the RAM consumption will exceed 10GB and cause a crash on many systems!
To fix this problem, we should avoid keeping the more distant elements in the list and only keep the top N nearest elements (for example N=10). We just have to use a structure like Gnuastro's "Ordered list of size_t" structure: 'gal_list_osizet_t'.
Until this problem is fixed, to avoid the problem, you should decrease the aperture size to physically meaningful values (in the case of Gaia, something like 0.1 arcsec ('--aperture=0.5/3600'). |
Mohammad Akhlaghi <makhlaghi>![]() |
No files currently attached
Depends on the following items: None found
Items that depend on this one: None found
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Small typo: at the end of the original post, the aperture size written in text (0.1 arcsec) didn't match the example option value (0.5 arcsec).
They should both be the same value (any one of 0.1 or 0.5 is fine, with 0.1 giving a faster result).