taskGNU Astronomy Utilities - Tasks: task #14315, Propagate all FITS header keywords...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #14315: Propagate all FITS header keywords into output FITS files

Submitter:  Lee Kelvin <leeskelvin>
Submitted:  Fri 20 Jan 2017 06:00:34 PM UTC
   
 
Should Start On:  Fri 20 Jan 2017 12:00:00 AM UTC Should be Finished on:  Fri 20 Jan 2017 12:00:00 AM UTC
Category:  All Gnuastro Priority:  5 - Normal
Item Group:  Enhancement Status:  None
Privacy:  Public Assigned to:  None
Percent Complete:  0% Open/Closed:  Open
Effort:  0.00

Jump to the original submission

Mon 23 Jan 2017 07:23:47 PM UTC, comment #7: 

The whole purpose of these discussions is that it should not up to me ;-), we want to make the best collective descision.

Once it is implemented, anyone can enable and/or disable this feature (as well as all the other features) in all Gnuastro, for the whole system, for a program, for a user, for a directory, or even only for one run of any program. I don't know of any software in astronomy which allow this level of configurability. So I am not against it ;-).

I have also been thinking about this feature for a long time, so I was really happy when you suggested it. I just don't have the mental capacity now to implement this new feauture immediately. All my concentration is on the much more fundamental `gal_data_t' (task #14245). Fixing bugs is easy, but implementing a whole new feature elegantly/robustly takes concentration. So I suggested that I would happily help anyone else who can do it before me ;-).

Thanks again for bringing up this issue ;-)...

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 23 Jan 2017 06:45:05 PM UTC, comment #6: 


> Initially you mentioned that you "favour the latter option of the two".


Ah, in which case I misunderstood. Apologies for that. I thought option 1 was to manually check a predefined list of protected CFITSIO/WCSLIB keywords whereas option 2 was to essentially do the inverse of that, only checking for name clashes. I'm not familiar with the inner workings of the GNUAstro code, so I did not fully appreciate your explanation.

> I agree, existing software do this. But that doesn't mean this this the best approach.


Yes, this is a very valid point. I would just caution however that I'm certain organisations such as NASA have thought hard about the end user experience for their software, and designed it accordingly. However, the decision on what to do here is of course up to you.

Regards,
Lee Kelvin


Lee Kelvin <leeskelvin>
Mon 23 Jan 2017 06:14:44 PM UTC, comment #5: 

Initially you mentioned that you "favour the latter option of the two". So I tried to elaborate more on it. The latter option was to use the Header program and not internally in each program.

I agree, existing software do this. But that doesn't mean this this the best approach. Infact, as you can see in Gnuastro, we have tried to make everything as maximally configurable as possible, this principle would thus also apply to this task: the user should be able to choose if they want this extra information or not.

Having the principal of maximal configurability in mind, in the previous post I also suggested that this feature be implemented as a library function. So, once implemented, applying it within all the programs will be really easy (with one common option and one line in the code). Infact with version 0.3, there will also be a `gnuastro.conf' configuration file for default values to common options (see task #14300). So once its implemented, enabling this feature in all Gnuastro programs will be as easy as changing a `0' to `1' in the main `gnuastro.conf' (which will be installed).

Let's leave the discssion on if it should be `0' (off) or `1' (on) by default for once it is implemented. I suggest that in order to encourage people to produce more reproducible/scientific/efficient results (task #14319), by default it can be off by default.

For the time being I am too busy with implementing task #14245 in all the programs. Infact `gal_data_t' (from that task) has also significantly simplified header keyword reading/writing. So applying this task shouldn't be too hard. Most (if not all) of the necessary building blocks for this task are in place. This is an important task, so if anyone is interested in implementing this feature, please post a comment here and start, I would be happy to help ;-)...

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 23 Jan 2017 05:27:25 PM UTC, comment #4: 

Hi Mohammad,

A few thoughts on your comments.

> Ultimately, each program should only produce the raw header keywords that are directly related to it's processing and the minimum required FITS/WCS keywords (written by CFITSIO and WCSLIB, not taken directly from the input).


Whilst I appreciate your desire to keep FITS files simple and as small as possible, from an end-user perspective I anticipate that this approach will cause much confusion. I would strongly advocate for all header keywords to be propagated forwards instead as default.

Many other software packages propagate all FITS header keywords in this manner. For example, the NASA HEASARC program 'fitscopy' may be used on the initial 'input.fits' file I provided at the start of this thread to crop a small 1x1 pixel region from the original 3x3 image using the command:


fitscopy input.fits[1:1,1:1] output.fits


In this command, the relevant CFITSIO header keywords are changed (NAXIS1/NAXIS2), as expected, and the scientific header keyword TEST is automatically propagated. It is up to the end-user to remove that header keyword after-the-fact should they see fit. STScI's 'imcopy' also operates in the same manner. They both operate this way because it is easier to remove surplus information than it is to add missing information.

I think operating in this fashion is the natively expected behaviour, however, I wanted to check this in a different field. I took a photograph on my phone, uploaded the JPEG image to my computer, and opened it using the image processing software GIMP. I cropped a small region, and exported the output JPEG. On opening the cropped image, all of the extra metadata (aperture, exposure, focal length, flash, etc...) had all made it through to the new output image.

> So your suggestion to add already existing keywords as `COMMENT' keywords is indeed very interesting.


Just to clarify what I mean here, I do not mean to add existing keywords as COMMENT keywords. The keywords should be kept intact wherever possible, to aid in pipeline processing. A COMMENT tag can be used to delineate between old header keywords and new ones. For example, a file prior to image warping may have a header which looks like this:


SIMPLE  = T
BITPIX  = 16
NAXIS   = 2
NAXIS1  = 3
NAXIS2  = 3
EXTEND  = T
...
TEST    = 'VALUE   '


and afterwards may have a header which looks like this:


SIMPLE  = T
BITPIX  = 16
NAXIS   = 2
NAXIS1  = 1
NAXIS2  = 1
EXTEND  = T
...
TEST    = 'VALUE   '
COMMENT
COMMENT Entries below from GNUAstro ImageWarp
COMMENT
INF_1   = ...
...


Once a standard for propagating through existing header information has been agreed upon, then would be a good time to tackle the issue of keyword name clashes more closely.

Regards,
Lee Kelvin

Lee Kelvin <leeskelvin>
Mon 23 Jan 2017 03:49:10 PM UTC, comment #3: 

Thanks, I agree, the second/latter method (where the "Header" program would be in charge of propagating headers from one or multiple files into another file) is a much better method, and is in-line with the modularity principle.

Ultimately, each program should only produce the raw header keywords that are directly related to it's processing and the minimum required FITS/WCS keywords (written by CFITSIO and WCSLIB, not taken directly from the input).

Science-related keywords, are only relevant/useful at higher-level than any single program. Whether they should be added or not is a choice by the science-case the data is meant to be used for. They are also commonly only necessary in the final data product of a research process, not in all the intermediate files.

But ultimately, even if it is directly available only in the Header program, this feature will be implemented as a library function, so people can also implement it directly in their own separate programs, or if there is enough demand, it can be added as a common option to all programs (the first option, but will be disabled by default).

The Header program works at a lower level than CFITSIO or WCSLIB. For example, you can even delete a mandatory FITS header with it: Header will do the job, but your FITS file will be unreadable! So as far as the Header program is concerned, it only sees keywords that exist in the input and those that are only available in the reference/other file+extension.

The name-clashes you mentioned are indeed important. In the implementation, atleast we can account for those in the FITS standard manually (for example the case for RADECSYS and RADESYS that you mentioned in bug #50073).

So your suggestion to add already existing keywords as `COMMENT' keywords is indeed very interesting. But just at the top of my head, I feel this might be too much: if a header value has changed, adding it has the potential to confuse people more than inform them. If the user feels it is very important to have the old values, they can define their own conventions for old values and use Header to add them individually (easily done in a script) to add them in what-ever way they like. The Header program has many ways to write/update header keywords.

All the keywords from another file+extension can be added under a title like "      \ keywords from XXXXX.fits (hdu: YYY) " (the same way WCS keywords are separated from program-specific and also version keywords). This way, headers from multiple files can even be merged into the final output.

I think this classification of the header keywords would also fix the problem you mentioned on the value-added keywords. As you suggested, repeated keyword names and their value can be put in a COMMENT keyword, so the information isn't lost and there is also no conflict with (possibly) existing keywords of the same name.

Unfortunately the FITS standard puts a limit of 8 characters on the keyword names, so reserving two characters for `_n' suffix would make them even more cryptic than they already are and hard to read/understand/use. If the problem can be addressed with the classification of headers by the file+extension they originated from, that might be a better solution.

Reproducibility information (how the raw data was processed to produce an output) is indeed very important, but that information is much better/usefully preserved in scripts/Makefiles rather than FITS headers. I have been thinking about this for some time, so I just defined task #14319 as a starting point for that discussion. Implementing something like what is proposed there will remove the need to keep many extra header keywords. However, it won't remove the need to propagate some keywords in the final output, so it is independent of this task.

But please share you thoughts on that task too ;-)...

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 23 Jan 2017 10:27:42 AM UTC, comment #2: 

I personally favour the latter option of the two you present, adding all keywords which do not already exist in the header. In my own efforts writing FITS read/write software for the R astro package, we arrived at a similar conclusion.

The safest way ahead would be to allow WCSLIB to construct a basic FITS header output, and then propagate any additional header keywords from the original FITS file which do not already have a match in the proto-output file.

Many software packages do this as default, so I would have this option enabled by default too. Personally, and knowing the value of FITS header keywords for science, I would not make this an option. Rather, I would force this behaviour always.

FITS header keywords not supplied by WCSLIB but rather propagated through from the original header could be left beneath a COMMENT tag with the comment along the lines of 'Propagated FITS keywords below', or similar.

However, some care must be taken to avoid FITS header keyword name clashes. It is inevitable that new WCS header keywords will clash with the old, previous WCS header keywords. This is as expected, and defaulting to the new value when a clash exists is the desired option. The biggest source of potential clash comes from GNU Astro value-added header keywords. It is not inconceivable that value-added GNU Astro header keywords (such as 'INF_1' after using ImageWarp) may already exist in the original FITS file, e.g., from a previous invocation of ImageWarp. In a scenario such as this, which header keyword gets the preference? My instinct is to lean towards the original, but it would be nice to have the new information also. To achieve this, a number could be appended to output GNUAstro value-added keywords to avoid keyword name clashes?

Perhaps the output FITS header should be constructed like this:

  • WCS keywords: always take new keywords over old
  • GNUAstro keywords: append _n to all value-added GNUAstro keyword outputs to avoid clashes (also has the bonus of revealing how many times a GNUAstro operation has been performed on the file)
  • Propagated keywords: always take all old keywords which do not have a match in any of the two classes above.


Keen to hear your thoughts,

Regards,
Lee Kelvin

Lee Kelvin <leeskelvin>
Fri 20 Jan 2017 06:27:53 PM UTC, comment #1: 

This is another very useful suggestion.

I can confirm that all Gnuastro programs that output FITS files are like this. They do not include any of the input keywords.

The main reason is that the most important header keywords (mandated by FITS) change, for example in both ImageCrop and ImageWarp the size and WCS information change. Searching through the keywords to see which ones have changed and which ones haven't can be a great burden in terms of efficiency/speed.

Another issue is that in some surveys the main information is in the first extension and the second (and etc) extentions just have the data with the bare minimun keywords (for example one extension with the image, the next for masked pixels, and the third for exposure/weight.

So I do support the idea that we add a common option (for all Gnuastro programs). When this option is given the programs can check for differing keywords and include all keywords from the input that have not changed.

Alternatively, a more general way would be to add a feature to the Header program to add all keywords that don't exist in the input file+extension, from another file+extension. I think this second option would be more general than doing it internally in each program.

Please share your thoughts/suggestions. This is an inportant issue and we want to find the most elegant/robust solution to it.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 20 Jan 2017 06:00:34 PM UTC, original submission:  

When performing an image rotation using ImageWarp, or an image crop using ImageCrop, all user-supplied FITS header keywords should be propagated through to the resultant output file.

Many contemporary astronomy surveys embed a lot of valuable information in the headers of FITS images, such as date of observation, wavelength, telescope, airmass, etc. If is often desirable to crop or manipulate large survey FITS files to produce smaller postage stamps around objects of interest. However, when doing this with ImageCrop or ImageWarp, all of this extra information is lost.

By way of example, I attach a very simple FITS file ('input.fits') to this post. It is a 3x3 image with the bare minimum FITS header. I have added to the header the new user-supplied keyword 'TEST'. I then run the following two commands:


$ astimgcrop --section=1:1,1:1 input.fits
$ astimgwarp --rotate=90 input.fits


The first command creates an even smaller 1x1 pixel cutout from the 3x3 image, whilst the second simply rotates the image through 90 degrees. This produces the two output files 'input_crop.fits' and 'input_rotated.fits'.

As can be seen, the header keyword 'TEST' has been removed from both output files.

I am unsure if this behaviour impacts more than one GNU Astro software packages, as these are the two I have tested most extensively so far. However, as this behaviour impacts more than one package, I have marked it as 'All Gnuastro' for the time being.

Regards,
Lee Kelvin

Lee Kelvin <leeskelvin>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #39521:  input.fits added by leeskelvin (6KiB - image/fits)
file #39522:  input_crop.fits added by leeskelvin (6KiB - image/fits)
file #39523:  input_rotated.fits added by leeskelvin (8KiB - image/fits)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by leeskelvin (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-01-20 leeskelvin Attached File- Added input.fits, #39521
        Attached File- Added input_crop.fits, #39522
        Attached File- Added input_rotated.fits, #39523

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code