taskGNU Astronomy Utilities - Tasks: task #15181, Libcurl for optimal download of...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15181: Libcurl for optimal download of only the necessary bytes

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Thu 14 Feb 2019 11:30:52 AM UTC
   
 
Should Start On:  Thu 14 Feb 2019 12:00:00 AM UTC Should be Finished on:  Thu 14 Feb 2019 12:00:00 AM UTC
Category:  Fits Priority:  5 - Normal
Item Group:  Enhancement Status:  Postponed
Privacy:  Public Assigned to:  None
Percent Complete:  0% Open/Closed:  Open
Effort:  0.00

Wed 04 Oct 2023 04:39:55 PM UTC, comment #3: 

In the last comment, I forgot to give the value to the 'xdfurl' shell variable (from the general Gnuastro tutorial). Here is the full command:


$ xdfurl=http://archive.stsci.edu/pub/hlsp/xdf
$ curl -kRL -r "$((0*80))-$((36*80-1))" --output header.txt $xdfurl/hlsp_xdf_hst_wfc3ir-60mas_hudf_f105w_v1_sci.fits


Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 04 Oct 2023 04:36:39 PM UTC, comment #2: 

For a command-line alternative, Ángel López Sainz suggested the following command that will extract only the first 20 header keywords of the FITS file in the given URL (note that each keyword is 80 characters/bytes long):


curl -kRL -r "$((0*80))-$((20*80-1))" --output header.txt $xdfurl/hlsp_xdf_hst_wfc3ir-60mas_hudf_f105w_v1_sci.fits


Besides the 'fopen.c' solution in the commets below, we can try to investigate we can try to investigate how these command-line options can be called within the 'curl' library.

The basic idea for FITS files is that the header keywords must be in chunks of 2880 bytes (or 36 keywords); the last chunk is the one that contains the 'END' characters at the start of the 80-character line.

So we read chunks of 2880 bytes, and search for the 'END' character at the start of a line. The moment it is found, we stop the download and have the full header keyword structure to feed into CFITSIO (or WCSLIB). Once the necessary pixels to use are found, we can then only read those pixels from the data component of the HDU (which is just after the chunk containing 'END').

Of course, if there are multiple HDUs, from the first HDU we know the byte number of the next HDU and can go there ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 14 Feb 2019 05:07:00 PM UTC, comment #1: 

libcurl seems to be a good tool for this job. For example see their example fopen.c program which simulates C's `fopen', but for URLs.

If we can implement this in the low-level CFITSIO file reading steps in coordination with the CFITSIO developers, it would be great (and usable in any program that uses CFITSIO later). I have raised the issue with the CFITSIO help desk and will see what they suggest.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 14 Feb 2019 11:30:52 AM UTC, original submission:  

CFITSIO allows the usage of network addresses as input FITS files. However, after some tests, I noticed that it simply first downloads the full file, then prepares the required extension/region. So in effect, its is no different from a full download of the full file first, then running CFITSIO on it.

But if we follow the FITS standard, and considering the fact that we can ask download libraries (like cURL), to only download certain byte ranges. In this way, we can optimize the network traffic when only certain extensions/parts of a file on the server are requested.

As servers are hosting larger and larger datasets, optimal usage of network traffic can greatly help in speeding up of processing.

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-10-04 makhlaghi SummaryFits program using network libraries for optimal download Libcurl for optimal download of only the necessary bytes
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code