GNU Astronomy Utilities - Tasks: task #15317, Concatenate two or more tables by...
You are not allowed to post comments on this tracker with your current authentication level.
task #15317: Concatenate two or more tables by row
Submitter: | Raul Infante-Sainz <infantesainz> | ||
Submitted: | Wed 03 Jul 2019 07:52:04 PM UTC | ||
Should Start On: | Wed 03 Jul 2019 12:00:00 AM UTC | Should be Finished on: | Wed 03 Jul 2019 12:00:00 AM UTC |
Category: | Table | Priority: | 5 - Normal |
Item Group: | New feature | Status: | Done |
Privacy: | Public | Percent Complete: | 100% |
Assigned to: | None | Open/Closed: | Closed |
Effort: | 0.00 |
( Jump to the original submission )
Sun 14 Nov 2021 12:46:04 AM UTC, comment #44: |
Mohammad Akhlaghi <makhlaghi>![]() |
Tue 06 Apr 2021 06:37:48 AM UTC, comment #43: You can use the `tests/during-dev.sh` script to test your modifications and debugging. After making changes, build GNU Astro in a parallel `build` directory using the `developer-build` script using the --debug flag:
There are more options that you can find by `--help` flag but this will get you up and running.
After that, you can use the `during-dev.sh` script to test your commands. Set the parameters in the `SET INPUT PARAMETERS` section and then run the script:
For your segmentation faults, use Valgrind to debug. You can simplify this by prepending the Valgrind commands in `during-dev.sh` in this line near the bottom of `during-dev.sh`:
Then trace the outputs back to the source of the memory leak.
You can download the TexLIve package from here. There is a quick installation instruction that you can quickly get you started. TexLive is a fairly large package but you don't need to install all of it. During the installation prompt, select the `basic` package to download and install. This will get you the minimum set of tools to write documentation easily.
I hope this helps. If you have any queries, please don't hesitate to ask :) |
Sachin Kumar Singh <sks_15>![]() |
Mon 05 Apr 2021 09:19:32 PM UTC, comment #42: I have made a commit to my fork on GitLab, the code is not working fully. It compiles but after running the command with sample files it gives a segmentation fault. I have tried implementing the function, however I'm still not very clear about all the data structures and functions.
Here is my GitLab fork https://gitlab.com/yemaedahrav/gnuastro-dev
I haven't added documentation, while bootstrapping dependencies I was not clear about installation of TexLive packages. Hence while running make, it showed an error at the end
Can you review this commit once
|
Amey Varhade <ameyvariitg> |
Sun 04 Apr 2021 04:43:18 PM UTC, comment #41: Its great that you have gone through the code and have a basic understanding.
In the part you have quoted, we just add a '-N' (where 'N' is a counter) to the new column names, we don't touch the column data/contents in this part.
The option to concatenate rows will be different and arguably much more easier, for example:
You can define a new 'table_catrow" function for concatenation by rows. Here is a rough outline of what it can do inside:
I hope this helps. Later, when you get a good understanding of this, it would also be good if you can help improve the documentation to clarify the points that weren't immediately clear to you: others will have a similar problem. |
Mohammad Akhlaghi <makhlaghi>![]() |
Sun 04 Apr 2021 03:23:13 PM UTC, comment #40: I have gone through the discussion on concatenating tables by columns and have got a basic understanding of the code and file structure.
Here, the tables are stored as lists of gal_data_t structs, how to access rows to add the rows.
|
Amey Varhade <ameyvariitg> |
Sun 22 Mar 2020 03:11:45 PM UTC, comment #39: Thanks! It has been merged and pushed to the main repository.
I also followed it up by a commit, polishing it a little and also enabling multiple calls to the `--catcolumn' option to add multiple tables. Please have a look at the commit message and changes to hopefully have smoother merges in the future.
I changed the task name to only focus on the rows now. If you get the chance to also implement that feature, we can close this task when its merged ;-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Sun 22 Mar 2020 02:40:18 AM UTC, comment #38: I rebased it.
|
madhav_bansal <madhav_bansal>![]() |
Sat 21 Mar 2020 11:49:46 PM UTC, comment #37: Madhav, given that your copyright forms are complete, I am ready to merge, but I noticed that your commit was in parallel to another commit that I just merged into master.
To have a cleaner history, can you please pull the master branch to the most recent commit, the rebase your commit over it? Let me know when its done and I'll do the merge ;-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Fri 20 Mar 2020 05:43:59 PM UTC, comment #36: Yes, that is correct, in many scenarios a file has multiple HDUs/extensions and the user may even want to concatenate two tables in one file for example.
OK! Everything is ready :-D. I'll merge it into the master branch as soon as the copyright things are done. Just post a comment here when its done. If you think its taking long, you can send them another email and remind them that you want your work merged before the GSoC deadline ;-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Fri 20 Mar 2020 03:33:02 PM UTC, comment #35: done.
|
madhav_bansal <madhav_bansal>![]() |
Fri 20 Mar 2020 02:49:31 PM UTC, comment #34: The options of `lib/gnuastro-internal/commonopts.h' are common to all programs, so you shouldn't add it there! You should have added it to the `args.h' of the Table program ;-). Can you please correct this? |
Mohammad Akhlaghi <makhlaghi>![]() |
Fri 20 Mar 2020 06:35:20 AM UTC, comment #33: I have inserted the option in (lib/gnuastro-internal/commonopts.h)and also check the documentation for --catcolhdu is it correct?
|
madhav_bansal <madhav_bansal>![]() |
Wed 18 Mar 2020 10:22:21 PM UTC, comment #32: Thanks, I gave it a try and looked at the code, it is almost ready to be merged. But just now something occurred to me: when calling `gal_table_read', you are giving it the correct file name, but not the correct HDU.
The `cp->hdu' is actually the HDU of the input file, not the file to be concatenated. You should define a new option, maybe called `--catcolhdu' (defined similar to the `--hdu' option (defined in `lib/gnuastro-internal/commonopts.h'). and feed its string instead of `cp->hdu'. This will enable users to have their desired concatenation table in any extension they want. Also don't forget to document this new option also.
Finally, while you are making corrections to this code, please break down the error message such that each line doesn't exceed 75 characters (the rest of the function is good, don't let the lines of the error message get any longer per line). |
Mohammad Akhlaghi <makhlaghi>![]() |
Wed 18 Mar 2020 04:15:06 PM UTC, comment #31: please give a final check
|
madhav_bansal <madhav_bansal>![]() |
Wed 18 Mar 2020 03:09:52 PM UTC, comment #30: Thanks Madhav, the explanation is more clear now. You can commit and rebase it now (so all your work comes into one commit after the most recent master branch commit) then let me know. I'll then give it one last try and as soon as your copyright forms are complete, I'll merge it into the master branch :-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Wed 18 Mar 2020 11:36:07 AM UTC, comment #29: I think this explains the option correctly
I have mailed again, for the copyright confirmation today.
|
madhav_bansal <madhav_bansal>![]() |
Tue 17 Mar 2020 01:29:51 AM UTC, comment #28: I actually checked out to your branch this time and ran it. Things look good now :-). Only a small problem: I see that the added documentation again has lines ending with white-space.
Generally, its good to make this a rule of thumb: before pushing your commits, run `git log -p -1' and look at what you have committed. Following the discussion before professional editors have modes to remove such things (another is an indentation with TAB, which is very bad except for Makefiles) every time the file is saved. For example I have activated it in Emacs for myself, so I never have to worry about it ;-).
Besides the ending with whitespace, the description of the option can be improved: 1) its English, 2) note in other parts of `gnuastro.texi' that we have the convention of one-sentence-per-line. This greatly helps in managing changes in this long documentation with Git!
Once you apply these final changes I am ready to merge, but I just need the Copyright confirmation first. Can you please send them another email and confirm if they are pursuing your case?
About the many things you can add to Table, there is a list already a list of table-specific features in the tasks (especially if you select the "Category" to be table).
But these are just the top of the ice burg! The main tool people currently use for Table processing in astronomy is TOPCAT. Besides its visualization features, it also has many useful processing features that we can also add to table. The problem with TOPCAT is that its written in Java, and thus not usable by anyone else! In Gnuastro useful features can be added to the library and used in many other contexts, besides the simple and easy to use Table program ofcourse.
Another interesting program that heavily uses tables is Match, with its own set of features that may need adding. Gnuastro's MakeCatalog is also the interface between images and tables, with MANY useful operators/measurements to add. |
Mohammad Akhlaghi <makhlaghi>![]() |
Tue 17 Mar 2020 01:09:38 AM UTC, comment #27: By the way, you can check the documenation with this command:
The generated PDF file (if TeX doesn't crash!) is saved in `doc/gnuastro.pdf'. |
Mohammad Akhlaghi <makhlaghi>![]() |
Tue 17 Mar 2020 12:56:18 AM UTC, comment #26: I'll look at it in a few minutes, but most editors have a feature to automatically adjust plain-text widths, for example if you use Emacs, you can simply press Alt-Q and the paragraph text will align. Vim also has a similar feature, I don't know about other editors. But generally try to use a professional text editor and master it to greatly save your time and energy in coding ;-).
I recommend GNU Emacs, but won't get into the Editor war: Vim is also good! Its just important to master one and benefit from all its powerful features. |
Mohammad Akhlaghi <makhlaghi>![]() |
Tue 17 Mar 2020 12:51:15 AM UTC, comment #25: sorry, for making the same mistake again and again
|
madhav_bansal <madhav_bansal>![]() |
Tue 17 Mar 2020 12:40:23 AM UTC, comment #24: I have updated the documentation also, but not sure if did it right. Is there any way to check how it looks as output.
|
madhav_bansal <madhav_bansal>![]() |
Tue 17 Mar 2020 12:04:37 AM UTC, comment #23: Good, but still not enough! Please pay attention to all the other commits/codes/comments in the code. You want your code to fully blend into the program's source.
It is a little frustrating for me to see a previously raised point is being repeated multiple times. Please pay more attention if you are serious about this ;-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Mon 16 Mar 2020 11:31:54 PM UTC, comment #22: I have made commit message little sorter and corrected the other two points also.
|
madhav_bansal <madhav_bansal>![]() |
Mon 16 Mar 2020 11:17:45 PM UTC, comment #21: Another (possibly) important point: I see that in the commit info, your name is "madhav-bansal". We use this name to generate the `AUTHORS' file, and also print it in the second page of the final PDF. If you are happy with "madhav-bansal" that is fine, but I just thought of bringing it up in case you want your standard name string to be put in those places ;-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Mon 16 Mar 2020 11:14:05 PM UTC, comment #20: A few other small touches:
|
Mohammad Akhlaghi <makhlaghi>![]() |
Mon 16 Mar 2020 11:01:40 PM UTC, comment #19: well done all the things in the table_catcolumn function itself. please give feedback.https://github.com/madhav-bansal/gnuastro/commits/table |
madhav_bansal <madhav_bansal>![]() |
Mon 16 Mar 2020 10:28:45 PM UTC, comment #18: Of course, if a task is VERY INTERESTING for you and you feel you an do it better, you can also write your GSoC proposed project based on a task that someone else has already expressed interest on. But be careful! The competition (and thus possibility of failure) will increase ;-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Mon 16 Mar 2020 10:07:21 PM UTC, comment #17: Great! I look forward to the corrections ;-).
Its good to also add the documentation in the same commit so it can be complete.
For the GSoC proposal, once your first commit is merged into the main history of Gnuastro, I think you can make a good case that you know the source code and the workflow. But you also need to choose a useful/important project to propose to work on in Gnuastro. There are some that are suggested in the GSoC suggestion page, and of course the general task list is very complete. The Table program itself can be greatly improved and has a lot of potential, but there are many other image processing tasks.
I guess its really up to you and what you are interested in, so have a general look and if something is not taken by others, you can post a comment that you will be working on it and write your proposal. |
Mohammad Akhlaghi <makhlaghi>![]() |
Mon 16 Mar 2020 09:56:51 PM UTC, comment #16: Thank you.
|
madhav_bansal <madhav_bansal>![]() |
Mon 16 Mar 2020 09:49:47 PM UTC, comment #15: Thanks, its getting much better ;-). I can do all of these myself (and it would take much less time for me than writing these comments), but I think it is much better to get you to a good start now. This way, your future projects in Gnuastro will be much more smoothly merged :-D.
Here are some points that occurred to me:
|
Mohammad Akhlaghi <makhlaghi>![]() |
Mon 16 Mar 2020 09:20:56 PM UTC, comment #14: Now I have elaborated on the comments and the commit message.
|
madhav_bansal <madhav_bansal>![]() |
Mon 16 Mar 2020 07:42:23 PM UTC, comment #13: It looks better, thanks! But some of the points haven't been fully/correctly applied:
Some other points:
|
Mohammad Akhlaghi <makhlaghi>![]() |
Mon 16 Mar 2020 07:09:37 PM UTC, comment #12: I have tried to correct the points directed by you .please check https://github.com/madhav-bansal/gnuastro/tree/table |
madhav_bansal <madhav_bansal>![]() |
Sat 14 Mar 2020 06:18:22 PM UTC, comment #11: No problem ;-). I am just looking at it, here are some points I am wrote as I was inspecting your commit. They are almost all cosmetic, so it won't take much time/effort to implement them. Nevertheless they are important so please implement them and keep them in mind for future commits.
|
Mohammad Akhlaghi <makhlaghi>![]() |
Sat 14 Mar 2020 05:45:25 PM UTC, comment #10: https://github.com/madhav-bansal/gnuastro/tree/tab
|
madhav_bansal <madhav_bansal>![]() |
Sat 14 Mar 2020 05:42:18 PM UTC, comment #9: I am happy to hear that you have a first implementation. But I don't see any link to your forked Gnuastro (as a Git project) to see what you have done. Can you please post the Git repository link? |
Mohammad Akhlaghi <makhlaghi>![]() |
Sat 14 Mar 2020 01:05:41 PM UTC, comment #8: i added the concatenate column option ....please give feedback on it.It is for one table only and i am thinking to use "struct list_select " to store tables for merging many tables.please give feedback about it |
madhav_bansal <madhav_bansal>![]() |
Fri 13 Mar 2020 12:10:50 PM UTC, comment #7: I encourage you to read the Developing chapter of the book. In particular the Program source and Mandatory source code files section and subsection. There, you'll see an explanation of all the basic program source files.
If you also have a look at the Gnuastro library section, you will see how you can find the definition of the macros and library functions. For example, you will see that `GAL_TYPE_STRLL' is defined in the Library data types section.
Adding new options is easy once you get use to it, but I haven't had too much time to document it yet unfortunately. In principle if you follow the functions from `main.c' you will see how the program enters `ui.c', and how it starts parsing the options.
But here is a summary of steps to add new options (I'll put this into the book later):
I hope I haven't missed anything ;-). |
Mohammad Akhlaghi <makhlaghi>![]() |
Fri 13 Mar 2020 10:40:06 AM UTC, comment #6: I actually I am having a problem on understanding the args.h
|
madhav_bansal <madhav_bansal>![]() |
Fri 13 Mar 2020 09:51:03 AM UTC, comment #5: I am having a problem on understanding how the command line arguments are parsed in the gnuastro. I think it is done in ui.c file.
|
madhav_bansal <madhav_bansal>![]() |
Fri 13 Mar 2020 12:52:24 AM UTC, comment #4: Thanks Madhav, This seems like a good implementation for concatenating columns (where the number of rows doesn't change). I am happy to see you have such a good understanding of Gnuastro's core data structure ;-).
When concatenating rows (such that the number of columns doesn't change), you can use the column metadata of the first input.
After the functions are implemented, this operation can be done like the command below. When we want to concatenate the columns of `table-add.fits' with `table-base.fits'.
Or this way to concatenate rows:
Some checks also need to be done on the sizes to make sure that it is possible.
Finally, we can implement these two options so they can be called multiple times (and merge multiple tables). For example
|
Mohammad Akhlaghi <makhlaghi>![]() |
Thu 12 Mar 2020 08:59:52 AM UTC, comment #3: Also, suggest what to do with the other meta-data while concatenating the tables along the row |
madhav_bansal <madhav_bansal>![]() |
Wed 11 Mar 2020 06:37:58 AM UTC, comment #2: does it seems right to you for adding columns to existing rows
I have just connected the last column of the first table to the first column of second table
|
madhav_bansal <madhav_bansal>![]() |
Fri 14 Feb 2020 01:53:00 PM UTC, comment #1: This is an interesting idea, but we can make it more general: the `tcat' option in STILTS, adds rows to existing columns. Besides that, we can also have an option to add columns to existing rows. |
Mohammad Akhlaghi <makhlaghi>![]() |
Wed 03 Jul 2019 07:52:04 PM UTC, original submission:
It would be worth to have an option in `Table' to be able to concatenate two or more tables. Something similar to what `tcat' of STILTS (http://www.star.bris.ac.uk/~mbt/stilts/sun256/tcat.html) does. |
Raul Infante-Sainz <infantesainz>![]() |
Depends on the following items: None found
Items that depend on this one: None found
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Follow 5 latest changes.
Date | Changed by | Updated Field | Previous Value | => | Replaced by |
---|---|---|---|---|---|
2021-11-14 | makhlaghi | Status | None | ![]() |
Done |
Percent Complete | 0% | ![]() |
100% | ||
Open/Closed | Open | ![]() |
Closed | ||
2021-04-05 | ameyvariitg | Attached File | - | ![]() |
Added Screenshot@from@2021-04-04@10-45-35.png, #51197 |
2020-03-22 | makhlaghi | Summary | Concatenate two or more tables | ![]() |
Concatenate two or more tables by row |
This task (adding rows from many tables) has been implemented in Commit b747d83e64, so I am closing the task.