taskGNU Astronomy Utilities - Tasks: task #13767, Support FITS compressed with fpack

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #13767: Support FITS compressed with fpack

Submitted by:  Mosè Giordano <giordano>
Submitted on:  Fri 09 Oct 2015 09:39:11 AM UTC  
 
Should Start On: Thu 08 Oct 2015 10:00:00 PM UTCShould be Finished on: Thu 08 Oct 2015 10:00:00 PM UTC
Category: All GnuastroPriority: 5 - Normal
Item Group: Input/OutputStatus: None
Privacy: PublicPercent Complete: 50%
Assigned to: Mohammad Akhlaghi <makhlaghi>Open/Closed: Open
Effort: 0.00

(Jump to the original submission Jump to the original submission)

Mon 30 May 2016 09:03:42 AM UTC, comment #12:

I don't think I'm able to change the open/closed status and the progress of a ticket once it's closed

Mosè Giordano <giordano>
Project Member
Mon 30 May 2016 02:00:48 AM UTC, comment #11:

You are right, thanks! I updated the metadata for this task.

I wonder if you have permissions to update the meta-data (I hope you do ;-)!)? If so, please also update the meta-data before submitting such comments (that show a change in the state of the task/bug is necessary).

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Sun 29 May 2016 04:57:51 PM UTC, comment #10:

I don't think this task is complete: gnuastro programs still don't compress the output file when it has ".fits.fz" extension.

Mosè Giordano <giordano>
Project Member
Fri 20 May 2016 10:55:40 AM UTC, comment #9:

Sorry, this was my mistake! Adding a -h1 option to the command before fixed the problem and a JPEG image showing the pixels was created!

I am marking this task as closed.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Fri 20 May 2016 10:50:42 AM UTC, comment #8:

I explained the installation of fpack and funpack in the manual. However, when I try opening the attached file (in the previous comment) in Gnuastro (for example converting it to JPEG), I get the following error:

It appears that CFITSIO will not automatically unpack the compressed file. Having a look at the "Image Compression" section of the CFITSIO manual, it is indeed necessary to do some operations before giving the file names to CFITSIO. We should add those operations when the input has a fits.fz suffix so something like the simple command above can work.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Fri 06 Nov 2015 10:26:07 AM UTC, comment #7:

I'm attaching one of the test file compressed with fpack, but if you have the fpack program on you system (which is probably the case since you have CFITSIO) you can compress a fits file with

(file #35396)

Mosè Giordano <giordano>
Project Member
Fri 06 Nov 2015 02:14:20 AM UTC, comment #6:

Ok, when I get the chance I will have a look at this issue, thanks for all the basic information ;-).

In the mean time, is it possible for you to introduce a sample .fits.fz file for future reference? If any public survey hosts public data, it would be great if you can put a link to its files in this task. Alternatively, if you have publicly released data in this format, you could attach a (small) crop of such a file to this task.

Ofcourse, we are working on outputs now, so as input, I can use a normal FITS image, but I would just like to to have a reference to the standard ways this format is used for the future, at least in my field (deep galaxy imaging surveys), none of the surveys I am familiar with use this format (or maybe they have but I haven't noticed!), so such a reference would be very useful.

For tasks like ImageCrop which fills parts of the output image that overlap with the inputs (in WCS mode), I think it might be easier to first let it save in FITS, then after the job is finished, read in the FITS image and save it as the output fits.fz format and then delete the FITS file. I am guessing that the compression and decompression on a tiled binary table for each input image can (possibly) make it much slower when multiple images are concerned.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Thu 05 Nov 2015 01:12:20 PM UTC, comment #5:

Sorry, I made a mistake: if I got it correctly "astmknoise" preserves the same data type of the original file, so if it was TLONG, like the file I was testing, the program can't write a compressed file. But now I tested a double precision file and it works as expected.

Perhaps a "--doubletype" option for "astmknoise" like for "astimgwarp" would be useful here.

Mosè Giordano <giordano>
Project Member
Thu 05 Nov 2015 10:57:54 AM UTC, comment #4:

I was testing "astimgcrop" and it can't write compressed files because it uses "fits_write_null_img" to write the fits file and this function ("ffpprn" in CFITSIO code) doesn't support writing to compressed files. Instead, "fits_write_img" does. Mohammad, could you please give it a look when you have time? You're more familiar than me with CFITSIO functions and internals of Gnuastro. No rush ;-)

The syntax to select a compressed output file is described in section 5.6 of CFITSIO manual, the simplest case is

Quotes are required in order to prevent the shell from interpreting the brackets as a regular expression.

Please also note that TLONG and TULONG datatypes are not supported anyway. So, for example, "astmknoise" uses "fits_write_null_img" to write the files but the datatype is TLONG. Instead, "astimgwarp" with "--doubletype" option works flawless.

Mosè Giordano <giordano>
Project Member
Thu 05 Nov 2015 03:09:25 AM UTC, comment #3:

Since the topic of this task is generally to "Support FITS compressed with fpack", not just reading the input, I think it is appropriate to also deal with creating ".fits.fz" output in this task too.

I have no experience with this format, but I don't think that creating an output should be too hard either. If it can be done within CFITSIO, then just the appropriate CFITSIO function has to be called within "arraytofitsimg" (in lib/fitsarrayvv.c) if the output has this format. If it is created by another method, that should also be easy to implement.

Most of the functions and programs use "arraytofitsimg" to write an array into a FITS image.

Mohammad Akhlaghi <makhlaghi>
Project AdministratorIn charge of this item.
Wed 04 Nov 2015 06:11:45 PM UTC, comment #2:

It turned out that there wasn't much to do: "nameisfits" and "nameisfitssuffix" didn't accept a file with "fits.fz" extension, once you add that to the list of acceptable extensions Gnuastro is able to read fpacked fits files.

The problem now is that Gnuastro doesn't compress output files even if they have ".fits.fz" extensions, contrary to what happens with ".fits.gz", but I think this should be addressed in another task.

Mosè Giordano <giordano>
Project Member
Wed 04 Nov 2015 11:22:16 AM UTC, comment #1:

I'd like to implement this feature as a function to be used across the whole project. In the vein of having shared libraries, I proposed in task #13765 a new naming convention for this project.

Mosè Giordano <giordano>
Project Member
Fri 09 Oct 2015 09:39:11 AM UTC, original submission:

Gnuastro is able to read FITS compressed with gzip, but not those compressed with fpack. CFITSIO should give the tools to do that anyway.

Mosè Giordano <giordano>
Project Member

 

Attached Files
file #35396:  convolve_spatial_warped.fits.fz added by giordano (25KiB - application/octet-stream - Example of file compressed with fpack)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by makhlaghi (Posted a comment)
  • -unavailable- added by giordano (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 11 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 30 May 2016 02:00:48 AM UTCmakhlaghiPercent Complete100%=>50%
      Open/ClosedClosed=>Open
    Fri 20 May 2016 10:55:40 AM UTCmakhlaghiPercent Complete30%=>100%
      Open/ClosedOpen=>Closed
    Fri 20 May 2016 10:50:42 AM UTCmakhlaghiPercent Complete50%=>30%
    Fri 06 Nov 2015 10:26:07 AM UTCgiordanoAttached File-=>Added convolve_spatial_warped.fits.fz, #35396
    Fri 06 Nov 2015 02:14:20 AM UTCmakhlaghiAssigned togiordano=>makhlaghi
    Thu 05 Nov 2015 03:10:40 AM UTCmakhlaghiPercent Complete0%=>50%
      Effort0.50=>0.00
    Thu 05 Nov 2015 03:09:25 AM UTCmakhlaghiAssigned toNone=>giordano
      Effort0.00=>0.5

    Back to the top


    Powered by Savane 3.1-cleanup1