taskGNU Astronomy Utilities - Tasks: task #15181, Fits program using network...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15181: Fits program using network libraries for optimal download

Submitted by:  Mohammad Akhlaghi <makhlaghi>
Submitted on:  Thu 14 Feb 2019 11:30:52 AM UTC  
 
Should Start On:  Thu 14 Feb 2019 12:00:00 AM UTC Should be Finished on:  Thu 14 Feb 2019 12:00:00 AM UTC
Category:  Fits Priority:  5 - Normal
Item Group:  Enhancement Status:  Postponed
Privacy:  Public Percent Complete:  0%
Assigned to:  None Open/Closed:  Open
Effort:  0.00

Thu 14 Feb 2019 05:07:00 PM UTC, comment #1:

libcurl seems to be a good tool for this job. For example see their example fopen.c program which simulates C's `fopen', but for URLs.

If we can implement this in the low-level CFITSIO file reading steps in coordination with the CFITSIO developers, it would be great (and usable in any program that uses CFITSIO later). I have raised the issue with the CFITSIO help desk and will see what they suggest.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Thu 14 Feb 2019 11:30:52 AM UTC, original submission:

CFITSIO allows the usage of network addresses as input FITS files. However, after some tests, I noticed that it simply first downloads the full file, then prepares the required extension/region. So in effect, its is no different from a full download of the full file first, then running CFITSIO on it.

But if we follow the FITS standard, and considering the fact that we can ask download libraries (like cURL), to only download certain byte ranges. In this way, we can optimize the network traffic when only certain extensions/parts of a file on the server are requested.

As servers are hosting larger and larger datasets, optimal usage of network traffic can greatly help in speeding up of processing.

Mohammad Akhlaghi <makhlaghi>
Project Administrator

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    No changes have been made to this item

    Back to the top


    Powered by Savane 3.4