bugGNU Astronomy Utilities - Bugs: bug #63815, Manual should mention that only...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #63815: Manual should mention that only essential FITS keywords are in output

Submitter:  Jean-Luc Margot <jlm>
Submitted:  Sat 18 Feb 2023 04:27:46 AM UTC
   
 
Category:  Warp Severity:  3 - Normal
Item Group:  Input/Output Status:  In Progress
Privacy:  Public Assigned to:  makhlaghi
Open/Closed:  Open

Sun 19 Feb 2023 10:33:01 AM UTC, comment #2: 

I just noticed that in the previous comment, I gave an outdated URL for the descriprion of the keyword related options of the 'astfits' program! Sorry for the mistake.

This is the correct URL: https://www.gnu.org/software/gnuastro/manual/html_node/Keyword-inspection-and-manipulation.html

Generally, for the documentation it is best to rely on your local copy that corresponts to the version you are using: https://www.gnu.org/software/gnuastro/manual/html_node/Accessing-documentation.html

You can also directly go into the documentation of each program with/section like the examples nelow:


$ info astfits
$ info gnuastro keyword


Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 18 Feb 2023 08:54:16 PM UTC, comment #1: 

Thanks For the comment,

All Gnuastro programs only keep essential keywords in their output, it is not just Warp. But there is a simple solution to this issue that is described below.

Let me first explain the reason for this behavior: we usually need more than one program to complete a task and moving around all the non-essential keywords in between intermediate data products is just an extra overhead and can cause conflicts.

This overhead can be significant because in the FITS standard, the keywords are kept in blocks of 2880 bytes at the start of the file (which translates to 36 lines of 80 bytes: the length of each keyword). When you initially define the output array in CFITSIO there is only one of these blocks for keywords. As you add more keywords such that a new block will be necessary, CFITSIO is forced to shift the whole data array also (because these blocks should be at the start of the file). This shift can slow-down the write operation on larger images.

Furthermore, searching for keywords that are not affected by the operation of each program will also add a lot of overhead. Especially as the number of keywords increase to very large number of keywords (I have seen files with thousands of keywords).

Preserving all extra keywords can also potentially add many bugs/annoyances for:

  • Users who may have keywords that are meaningful for any program (making them worry about loosing information).
  • Developers, who will have to keep track of the keywords that are created during the operation of a program (sometimes they are not known a-priori: they depend on the processing). 


All this complexity is really against the Unix philosophy; which has a term explicitly saying: "Don't clutter output with extraneous information".

Since the Job of the Warp program is to only Warp the pixel grid, any keyword that is not directly related to the pixel grid is "extraneous". 

The solution however is simple (again following the Unix philosopy: "Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new 'features'."):

Gnuastro has the 'astfits' program which you can use (through its '--copykeys' option) to copy keywords from one file to your final output (possibly after many operations through any number of programs).

Here is an example following your original example (just replace 'KEY1', 'KEY2', 'KEY3' with the keywords you want in the end).


$ astwarp --hdu=1 --rotate=0 -o warped.fits original.fits

$ astfits original.fits --copykeys=KEY1,KEY2,KEY3 \
          --output=warped.fits --outhdu=1


You can even add a "title" key entry for the new keywords you want to preserve in the final output (of many operations) through the '--write' option of 'astfits' like below.


$ astwarp --hdu=1 --rotate=0 -o warped.fits original.fits

$ astfits warped.fits --write=/,"Pipeline keywords"

$ astfits original.fits --copykeys=KEY1,KEY2,KEY3 \
          --output=warped.fits --outhdu=1


You can curate the keywords in your final output from many separate files and in any order and any grouping (with different titles) to be more clear for the final recipients of your pipeline's data products.

For more on the keyword editing features of the 'astfits' program, see https://www.gnu.org/software/gnuastro/manual/html_node/Keyword-manipulation.html

I hope this helps clarify that this issue is not a bug, but a feature of Gnuastro ;-).

However, to avoid confusion for users that are not use to this behavior I am keeping this bug open to add all the discussion here inside the manual to be clear for the users.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 18 Feb 2023 04:27:46 AM UTC, original submission:  

Take a fits file 'original.fits'.  Warp it:

astwarp --hdu=1 --rotate=0 -o warped.fits original.fits

(I read the documentation and did not find an option to preserve the header keys not affected by the warp operation)

Compare headers of original and warped images:

imhead original.fits > original.txt
imhead warped.fits > warped.txt

Note how the header has been reduced from 471 lines to 152 lines
wc original.txt warped.txt
  471  original.txt
  152  warped.txt

Note how some essential keys/values are missing from the warped file

grep MJD original.txt warped.txt
original.txt:MJD_OBS =       55360.51456292 / [days] Obs. midpoint Modified Julian Day
warped.txt:MJDREF  =                  0.0 / [d] MJD of fiducial time

grep UTC warped.txt returns 0 values, whereas the original file had 5 UTC values

Thank you.

Jean-Luc Margot <jlm>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by jlm (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-02-18 makhlaghi StatusNone In Progress
        Assigned toNone makhlaghi
        SummaryWarp removes essential keys from header Manual should mention that only essential FITS keywords are in output

    Back to the top

    Powered by Savane 3.12.
    Corresponding source code