taskGNU Astronomy Utilities - Tasks: task #14191, Enable FITS input and output using...


You are not allowed to post comments on this tracker with your current authentication level.

task #14191: Enable FITS input and output using Lzip

Submitted by:  Mohammad Akhlaghi <makhlaghi>
Submitted on:  Wed 26 Oct 2016 11:50:50 AM UTC  
Should Start On:  Tue 25 Oct 2016 03:00:00 PM UTC Should be Finished on:  Tue 25 Oct 2016 03:00:00 PM UTC
Category:  All Gnuastro Priority:  5 - Normal
Item Group:  Enhancement Status:  Postponed
Privacy:  Public Percent Complete:  40%
Assigned to:  None Open/Closed:  Open
Effort:  0.00

Mon 06 Feb 2017 04:18:32 PM UTC, comment #5: 

There are (among others) two very different kinds of data to be considered here. Lets call them "text" and "continuous functions". The main feature of "text" compression algorithms like gzip is not that the size of its compressing element is fixed at 8 bits, but the fact that they search for exact matches in the data. For gzip a miss is as good as a kilometer; 'foo' is as different from 'for' as it is from 'fox'.

OTOH, algorithms designed to compress data of the "continuous function" type, usually take advantage of the fact that differences between consecutive data elements tend to be smaller than the values of the elements, and therefore easier to compress efficiently. This is the case of the rice algorithm.

IMO, the two kinds of algorithms are so fundamentally different that it does not make sense to mix them.

Regarding the use of gzip/lzip to compress binary tables in fits files, the first attempt does not seem promising. The table seems to compress much better if it is in text (ASCII) format than in fits format:

-rw-r--r-- 1   8603322 2017-01-30 10:45 uvudf_rafelski_2015.dat
-rw-r--r-- 1   1877246 2017-01-30 10:45 uvudf_rafelski_2015.dat.gz
-rw-r--r-- 1   1164719 2017-01-30 10:45 uvudf_rafelski_2015.dat.lz
-rw-r--r-- 1   7450560 2017-01-09 08:23 uvudf_rafelski_2015.fits
-rw-r--r-- 1   4625937 2017-01-30 10:43 uvudf_rafelski_2015.fits.gz
-rw-r--r-- 1   3603977 2017-01-09 08:23 uvudf_rafelski_2015.fits.lz

Antonio Diaz Diaz <antonio>
Project Member
Fri 03 Feb 2017 02:06:14 PM UTC, comment #4: 

Thank you very much Antonio for the very nice review. It was very interesting.

I haven't had the chance to read the paper yet, but your great review made me even more enthusiastic to have a look into it.

I should appologize for my ignorance (I am really curious to learn more about compression, but I just haven't had the time yet!). But wouldn't it be an interesting feature for Gzip-like compression algorithms (Lzip in this case) that the number of bits in each compressing element can be set by the user (based on a data-type)? In other words, is the 8-bit size of each independent datum hard-coded in the algorithm, or can it be a variable?

Since this is the major "distinguishing characteristic" of Gzip-like compression, that Pence et al. bring up in that paper, wouldn't it be useful to modify them so they can also accommodate different word-sizes? After-all, multi-byte datasets are not just limited to astronomy, it can be useful in many fields.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Thu 02 Feb 2017 10:33:50 PM UTC, comment #3: 

Mohammad has found an article that explains why using gzip (and by extension bzip2 and lzip) to compress images is in general a bad idea. Gzip, bzip2 and lzip see the data as strings of bytes, not as pixels:

"The most important distinguishing characteristic of GZIP compared to the other compression algorithms used in this study is that GZIP treats each 8-bit byte of the input data stream as an independent datum, whereas the other compression methods operate on the numerical value of the input image pixels as multibyte quantities. This puts GZIP at a distinct disadvantage when compressing astronomical images with 16-bit or 32-bit pixel values because, unlike the Rice and Hcompress algorithms, GZIP cannot use the numerical difference between adjacent pixels as a means of improving the compression."

As a consequence, gzip, bzip2 and lzip may be not very useful to compress data masks because:

"In many cases, choosing the algorithm that produces the very highest compression ratio is of secondary importance because the masks compress so well with any algorithm that the size is insignificant compared to the rest of the associated data set. (...) In practice, it may be simplest to just use the same compression algorithm on the data mask as is used on the associated astronomical image."

Unless we can prove that lzip can compress binary tables or floating point images significantly better than the existing methods, I think we should not propose the addition of lzip to the fits standard because:

"Since the compression ratios produced by Rice and Hcompress are already within 75% to 90% of an ideal algorithm with K = 0, any further improvements to the compression algorithms will produce relatively little gain in the size of the compressed images."

The findings of the article suggest that maybe gzip tiled compression should not have been added to the fits standard either:

"Overall, the Rice compression algorithm strikes the best balance of compression and computational efficiency; it is 2-3 times faster and produces about 1.4 times greater compression than GZIP".

Antonio Diaz Diaz <antonio>
Project Member
Sun 29 Jan 2017 06:49:31 PM UTC, comment #2: 

Hello to all.

I have updated the lzip decompression patch for cfitsio 3.41. I am currently studying the feasibility of implementing 'LZIP_1' and 'LZIP_2' algorithms corresponding to the existing 'GZIP_1' and 'GZIP_2' algorithms. Any advice or comments are welcome.

Best regards,

(file #39608)

Antonio Diaz Diaz <antonio>
Project Member
Thu 17 Nov 2016 05:29:09 PM UTC, comment #1: 

Antonio Diaz Diaz (creator of the Lzip program and library) has kindly offered to help in this task.

He has prepared a patch for CFITSIO 3.39 that will allow it to read `.fits.lz' files the same way that CFITSIO reads `.fits.bz2' files. The patch is attached with this comment. Currently it only uncompresses an `.fits.lz' file, it doesn't compress/create one.

To apply the patch to CFITSIO take the following steps:

tar -xf cfitsio3390.tar.gz
cd cfitsio
lzip -cd PATH/cfitsio3390_lzip.diff.lz | patch -p1

Make sure you have liblz and libbz2 already installed. You can then configure and build CFITSIO as described in the manual, with some minor differences as shown below. If you don't want to confirm/check with bz2 images, ignore the bz2 options.

$ ./configure --prefix=/usr/local --enable-sse2 --enable-reentrant --with-bzip2 --with-lzip
$ make LIBS='-lbz2 -llz'
$ make utils
$ rm cookbook fitscopy imcopy smem speed testprog
$ sudo make install

I am also attaching `sample.fits.lz', which is one random real galaxy image to test on.

After installing CFITSIO, Gnuastro also needs some modifications, I have made a `cfitsio-lzip' branch in my development fork for the Gnuastro modifications (mainly to do with recognizing the file names and checking the libraries).

(file #39006, file #39007)

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Wed 26 Oct 2016 11:50:50 AM UTC, original submission:  

Lzip is a compression format that has both a high compression ratio and archival capabilities (can detect corruption in the data with high completeness).

CFITSIO does already allow .fz (only for FITS images) and .gz internally. So inputs and outputs can easily be done with these two compresssions in Gnuastro.

As a small test on this XDF survey image (originally 106Mb) and using the default output of all the compressors, Gzip's output is 44Mb, Lzip is 41Mb and the `.fz' file is 7Mb.

However, note that the `.fz' file is lossy, so the original values are slightly changed, you can tell with the 0 valued pixels in the original image that are no longer 0 in the `.fz' output. 

For Lzip, we will have to use the Lzip library to internally compress/decompress the files in input and output.

The only problem is Lzip inputs and outputs is that SAO ds9 (and probably other FITS viewers) don't not currently open `.fits.lz' files directly (for easy viewing), while it opens `.fits.gz' and `.fits.fz' easily. We might be able to get in touch with those viewers' developers and ask them to integrate Lzip.

Mohammad Akhlaghi <makhlaghi>
Project Administrator


Attached Files
file #39608:  cfitsio3410_lzip-1.diff.lz added by antonio (3KiB - application/x-tar - Lzip decompression patch for cfitsio 3.41)
file #39006:  cfitsio3390_lzip.diff.lz added by makhlaghi (3KiB - application/x-lzip)
file #39007:  sample.fits.lz added by makhlaghi (25KiB - application/x-lzip)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by antonio (Updated the item)
  • -email is unavailable- added by makhlaghi
  • -email is unavailable- added by makhlaghi (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-01-29 antonio Attached File- => Added cfitsio3410_lzip-1.diff.lz, #39608
    2016-11-17 makhlaghi Attached File- => Added cfitsio3390_lzip.diff.lz, #39006
        Attached File- => Added sample.fits.lz, #39007
        Percent Complete0% => 40%
        Carbon-Copy- => Added antonio@gnu.org

    Back to the top

    Powered by Savane 3.5