taskGNU Astronomy Utilities - Tasks: task #13767, Support FITS compressed with fpack

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #13767: Support FITS compressed with fpack

Submitter:  Mosè Giordano <giordano>
Submitted:  Fri 09 Oct 2015 09:39:11 AM UTC
   
 
Should Start On:  Thu 08 Oct 2015 10:00:00 PM UTC Should be Finished on:  Thu 08 Oct 2015 10:00:00 PM UTC
Category:  All Gnuastro Priority:  5 - Normal
Item Group:  Input/Output Status:  None
Privacy:  Public Assigned to:  makhlaghi
Percent Complete:  50% Open/Closed:  Open
Effort:  0.00

Jump to the original submission

Mon 30 May 2016 09:03:42 AM UTC, comment #12: 

I don't think I'm able to change the open/closed status and the progress of a ticket once it's closed

Mosè Giordano <giordano>
Group Member
Mon 30 May 2016 02:00:48 AM UTC, comment #11: 

You are right, thanks! I updated the metadata for this task.

I wonder if you have permissions to update the meta-data (I hope you do ;-)!)? If so, please also update the meta-data before submitting such comments (that show a change in the state of the task/bug is necessary).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 29 May 2016 04:57:51 PM UTC, comment #10: 

I don't think this task is complete: gnuastro programs still don't compress the output file when it has ".fits.fz" extension.

Mosè Giordano <giordano>
Group Member
Fri 20 May 2016 10:55:40 AM UTC, comment #9: 

Sorry, this was my mistake! Adding a -h1 option to the command before fixed the problem and a JPEG image showing the pixels was created!


astconvertt convolve_spatial_warped.fits.fz --output=jpg -h1


I am marking this task as closed.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 20 May 2016 10:50:42 AM UTC, comment #8: 

I explained the installation of fpack and funpack in the manual. However, when I try opening the attached file (in the previous comment) in Gnuastro (for example converting it to JPEG), I get the following error:


astconvertt convolve_spatial_warped.fits.fz --output=jpg

astconvertt: Currently only a 2 dimensional image array is supported. Your array is 0 dimension(s). This might Be due to the fact that the data in images with multiple extensions are sometimes put on the second extension. If this is the case, try changing the hdu (maybe to --hdu=1).


It appears that CFITSIO will not automatically unpack the compressed file. Having a look at the "Image Compression" section of the CFITSIO manual, it is indeed necessary to do some operations before giving the file names to CFITSIO. We should add those operations when the input has a fits.fz suffix so something like the simple command above can work.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 06 Nov 2015 10:26:07 AM UTC, comment #7: 

I'm attaching one of the test file compressed with fpack, but if you have the fpack program on you system (which is probably the case since you have CFITSIO) you can compress a fits file with


fpack file.fits


(file #35396)

Mosè Giordano <giordano>
Group Member
Fri 06 Nov 2015 02:14:20 AM UTC, comment #6: 

Ok, when I get the chance I will have a look at this issue, thanks for all the basic information ;-).

In the mean time, is it possible for you to introduce a sample .fits.fz file for future reference? If any public survey hosts public data, it would be great if you can put a link to its files in this task. Alternatively, if you have publicly released data in this format, you could attach a (small) crop of such a file to this task.

Ofcourse, we are working on outputs now, so as input, I can use a normal FITS image, but I would just like to to have a reference to the standard ways this format is used for the future, at least in my field (deep galaxy imaging surveys), none of the surveys I am familiar with use this format (or maybe they have but I haven't noticed!), so such a reference would be very useful.

For tasks like ImageCrop which fills parts of the output image that overlap with the inputs (in WCS mode), I think it might be easier to first let it save in FITS, then after the job is finished, read in the FITS image and save it as the output fits.fz format and then delete the FITS file. I am guessing that the compression and decompression on a tiled binary table for each input image can (possibly) make it much slower when multiple images are concerned.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 05 Nov 2015 01:12:20 PM UTC, comment #5: 

Sorry, I made a mistake: if I got it correctly "astmknoise" preserves the same data type of the original file, so if it was TLONG, like the file I was testing, the program can't write a compressed file.  But now I tested a double precision file and it works as expected.

Perhaps a "--doubletype" option for "astmknoise" like for "astimgwarp" would be useful here.

Mosè Giordano <giordano>
Group Member
Thu 05 Nov 2015 10:57:54 AM UTC, comment #4: 

I was testing "astimgcrop" and it can't write compressed files because it uses "fits_write_null_img" to write the fits file and this function ("ffpprn" in CFITSIO code) doesn't support writing to compressed files.  Instead, "fits_write_img" does.  Mohammad, could you please give it a look when you have time?  You're more familiar than me with CFITSIO functions and internals of Gnuastro.  No rush ;-)

The syntax to select a compressed output file is described in section 5.6 of CFITSIO manual, the simplest case is


$execname [OPTIONS...] $input --output="output.fits[compress]"


Quotes are required in order to prevent the shell from interpreting the brackets as a regular expression.

Please also note that TLONG and TULONG datatypes are not supported anyway.  So, for example, "astmknoise" uses "fits_write_null_img" to write the files but the datatype is TLONG.  Instead, "astimgwarp" with "--doubletype" option works flawless.

Mosè Giordano <giordano>
Group Member
Thu 05 Nov 2015 03:09:25 AM UTC, comment #3: 

Since the topic of this task is generally to "Support FITS compressed with fpack", not just reading the input, I think it is appropriate to also deal with creating ".fits.fz" output in this task too.

I have no experience with this format, but I don't think that creating an output should be too hard either. If it can be done within CFITSIO, then just the appropriate CFITSIO function has to be called within "arraytofitsimg" (in lib/fitsarrayvv.c) if the output has this format. If it is created by another method, that should also be easy to implement.

Most of the functions and programs use "arraytofitsimg" to write an array into a FITS image.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 04 Nov 2015 06:11:45 PM UTC, comment #2: 

It turned out that there wasn't much to do: "nameisfits" and "nameisfitssuffix" didn't accept a file with "fits.fz" extension, once you add that to the list of acceptable extensions Gnuastro is able to read fpacked fits files.

The problem now is that Gnuastro doesn't compress output files even if they have ".fits.fz" extensions, contrary to what happens with ".fits.gz", but I think this should be addressed in another task.

Mosè Giordano <giordano>
Group Member
Wed 04 Nov 2015 11:22:16 AM UTC, comment #1: 

I'd like to implement this feature as a function to be used across the whole project.  In the vein of having shared libraries, I proposed in task #13765 a new naming convention for this project.

Mosè Giordano <giordano>
Group Member
Fri 09 Oct 2015 09:39:11 AM UTC, original submission:  

Gnuastro is able to read FITS compressed with gzip, but not those compressed with fpack.  CFITSIO should give the tools to do that anyway.

Mosè Giordano <giordano>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #35396:  convolve_spatial_warped.fits.fz added by giordano (25KiB - application/octet-stream - Example of file compressed with fpack)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by giordano (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-05-30 makhlaghi Percent Complete100% 50%
        Open/ClosedClosed Open
    2016-05-20 makhlaghi Percent Complete30% 100%
        Open/ClosedOpen Closed
    2016-05-20 makhlaghi Percent Complete50% 30%
    2015-11-06 giordano Attached File- Added convolve_spatial_warped.fits.fz, #35396
    2015-11-06 makhlaghi Assigned togiordano makhlaghi
    2015-11-05 makhlaghi Percent Complete0% 50%
        Effort0.50 0.00
    2015-11-05 makhlaghi Assigned toNone giordano
        Effort0.00 0.5

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code