The whole purpose of these discussions is that it should not up to me ;-), we want to make the best collective descision.

Once it is implemented, anyone can enable and/or disable this feature (as well as all the other features) in all Gnuastro, for the whole system, for a program, for a user, for a directory, or even only for one run of any program. I don't know of any software in astronomy which allow this level of configurability. So I am not against it ;-).

I have also been thinking about this feature for a long time, so I was really happy when you suggested it. I just don't have the mental capacity now to implement this new feauture immediately. All my concentration is on the much more fundamental `gal_data_t' (task #14245). Fixing bugs is easy, but implementing a whole new feature elegantly/robustly takes concentration. So I suggested that I would happily help anyone else who can do it before me ;-).

Thanks again for bringing up this issue ;-)...

Initially you mentioned that you "favour the latter option of the two". So I tried to elaborate more on it. The latter option was to use the Header program and not internally in each program.

I agree, existing software do this. But that doesn't mean this this the best approach. Infact, as you can see in Gnuastro, we have tried to make everything as maximally configurable as possible, this principle would thus also apply to this task: the user should be able to choose if they want this extra information or not.

Having the principal of maximal configurability in mind, in the previous post I also suggested that this feature be implemented as a library function. So, once implemented, applying it within all the programs will be really easy (with one common option and one line in the code). Infact with version 0.3, there will also be a `gnuastro.conf' configuration file for default values to common options (see task #14300). So once its implemented, enabling this feature in all Gnuastro programs will be as easy as changing a `0' to `1' in the main `gnuastro.conf' (which will be installed).

Let's leave the discssion on if it should be `0' (off) or `1' (on) by default for once it is implemented. I suggest that in order to encourage people to produce more reproducible/scientific/efficient results (task #14319), by default it can be off by default.

For the time being I am too busy with implementing task #14245 in all the programs. Infact `gal_data_t' (from that task) has also significantly simplified header keyword reading/writing. So applying this task shouldn't be too hard. Most (if not all) of the necessary building blocks for this task are in place. This is an important task, so if anyone is interested in implementing this feature, please post a comment here and start, I would be happy to help ;-)...

Thanks, I agree, the second/latter method (where the "Header" program would be in charge of propagating headers from one or multiple files into another file) is a much better method, and is in-line with the modularity principle.

Ultimately, each program should only produce the raw header keywords that are directly related to it's processing and the minimum required FITS/WCS keywords (written by CFITSIO and WCSLIB, not taken directly from the input).

Science-related keywords, are only relevant/useful at higher-level than any single program. Whether they should be added or not is a choice by the science-case the data is meant to be used for. They are also commonly only necessary in the final data product of a research process, not in all the intermediate files.

But ultimately, even if it is directly available only in the Header program, this feature will be implemented as a library function, so people can also implement it directly in their own separate programs, or if there is enough demand, it can be added as a common option to all programs (the first option, but will be disabled by default).

The Header program works at a lower level than CFITSIO or WCSLIB. For example, you can even delete a mandatory FITS header with it: Header will do the job, but your FITS file will be unreadable! So as far as the Header program is concerned, it only sees keywords that exist in the input and those that are only available in the reference/other file+extension.

The name-clashes you mentioned are indeed important. In the implementation, atleast we can account for those in the FITS standard manually (for example the case for RADECSYS and RADESYS that you mentioned in bug #50073).

So your suggestion to add already existing keywords as `COMMENT' keywords is indeed very interesting. But just at the top of my head, I feel this might be too much: if a header value has changed, adding it has the potential to confuse people more than inform them. If the user feels it is very important to have the old values, they can define their own conventions for old values and use Header to add them individually (easily done in a script) to add them in what-ever way they like. The Header program has many ways to write/update header keywords.

All the keywords from another file+extension can be added under a title like "      \ keywords from XXXXX.fits (hdu: YYY) " (the same way WCS keywords are separated from program-specific and also version keywords). This way, headers from multiple files can even be merged into the final output.

I think this classification of the header keywords would also fix the problem you mentioned on the value-added keywords. As you suggested, repeated keyword names and their value can be put in a COMMENT keyword, so the information isn't lost and there is also no conflict with (possibly) existing keywords of the same name.

Unfortunately the FITS standard puts a limit of 8 characters on the keyword names, so reserving two characters for `_n' suffix would make them even more cryptic than they already are and hard to read/understand/use. If the problem can be addressed with the classification of headers by the file+extension they originated from, that might be a better solution.

Reproducibility information (how the raw data was processed to produce an output) is indeed very important, but that information is much better/usefully preserved in scripts/Makefiles rather than FITS headers. I have been thinking about this for some time, so I just defined task #14319 as a starting point for that discussion. Implementing something like what is proposed there will remove the need to keep many extra header keywords. However, it won't remove the need to propagate some keywords in the final output, so it is independent of this task.

But please share you thoughts on that task too ;-)...

This is another very useful suggestion.

I can confirm that all Gnuastro programs that output FITS files are like this. They do not include any of the input keywords.

The main reason is that the most important header keywords (mandated by FITS) change, for example in both ImageCrop and ImageWarp the size and WCS information change. Searching through the keywords to see which ones have changed and which ones haven't can be a great burden in terms of efficiency/speed.

Another issue is that in some surveys the main information is in the first extension and the second (and etc) extentions just have the data with the bare minimun keywords (for example one extension with the image, the next for masked pixels, and the third for exposure/weight.

So I do support the idea that we add a common option (for all Gnuastro programs). When this option is given the programs can check for differing keywords and include all keywords from the input that have not changed.

Alternatively, a more general way would be to add a feature to the Header program to add all keywords that don't exist in the input file+extension, from another file+extension. I think this second option would be more general than doing it internally in each program.

Please share your thoughts/suggestions. This is an inportant issue and we want to find the most elegant/robust solution to it.