taskGNU Astronomy Utilities - Tasks: task #14245, Root data structure

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #14245: Root data structure

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Mon 21 Nov 2016 02:50:58 PM UTC
   
 
Should Start On:  Sun 20 Nov 2016 03:00:00 PM UTC Should be Finished on:  Sun 20 Nov 2016 03:00:00 PM UTC
Category:  All Gnuastro Priority:  5 - Normal
Item Group:  Enhancement Status:  Done
Privacy:  Public Assigned to:  makhlaghi
Percent Complete:  100% Open/Closed:  Closed
Effort:  0.00

Mon 24 Apr 2017 03:55:17 PM UTC, comment #5: 

From the reports so far, it was discovered that several old headers still remained in the source files and some necessary ones (for some systems) were missing in others. This has been corrected and pushed to the main repo.

For a tarball version incorporating the corrections, you can use gnuastro-0.2.191-ae7a.tar.gz.

I would be most grateful if you could try compiling and checking this release and let me know of any warnings/errors before the beta-release, thanks in advance ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 24 Apr 2017 03:03:47 AM UTC, comment #4: 

All of Gnuastro has been practically re-written to support this new data structure. The work has been merged with master and is now on the main Gnuastro Git repository so you can pull it. A summary is available in this gnuastro-commits post.

The documentation on individual programs has been updated during the work and I am now busy updating the library's section in the manual. Afterwards the NEWS file will be updated and we will be ready to make an alpha-release in the next few days and the 0.3 release by next week hopefully.

In the meantime, I wanted to see if you could clone, bootstrap, build and check this version to see if everything is working nicely on your systems. If you can't bootstrap, you can use the gnuastro-0.2.189-3339.tar.gz tarball and unpack, build and test it (the link will be removed after the alpha release).

If any warnings/errors come up during the building/testing I would be most grateful if you could let me know so we can address it before the alpha-release.

Important note: make sure to build with `make -j8'. In order to work on any data-type, several of the low-level libraries will take about a minute (or more) to compile, so if you don't compile in parallel it will take very long. On my computer it takes about 3 minutes to finish the build on 8 threads.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 01 Feb 2017 06:30:08 PM UTC, comment #3: 

This is just as an update.

MakeProfiles and CosmicCalculator now also work with the new data structure and all the great new features it has offered (e.g., option management system and table reading/writing, and column selection). So in total 4 programs now work with this system (Arithmetic and Table were done before).

If you want to try it out, you can clone and bootstrap the datastruct branch on my development fork. In case you do, the easiest way is to configure and build is with the ./tmpfs-config-make script. It will have all the `--enable-programname' configure options until all the programs are ported.

With this infra-structure built and tested on these four programs, the rest of the work on porting the other programs should be done soon. If you do get the chance to try them out it would be great.

Its interesting that while the code has become much more readable and clear, from the last commit, the total number of line deletions became larger than the additions in this branch compared to the `master' branch. By the time all programs are ported, this difference will be much more larger. So the code is becoming much more elegant :-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 28 Dec 2016 07:33:25 PM UTC, comment #2: 

From commit 5b61ae332 (Arithmetic functions/macros out of data.h) arithmetic-related operations on `gal_data_t' are now in a separate `gnuastro/arithmetic.h' library header (previously they were in `gnuastro/data.h').

This change also includes the non-installed library files. Hence, the `data-arithmetic-binary.c' link in the previous comment is no longer valid. If you are following that comment, please see arithmetic-binary.c instead.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 28 Dec 2016 03:45:42 PM UTC, comment #1: 

This is just an update.

A large number of functions have already been defined for this data structure, please see data.h. The fits.h functions have also been corrected (and greatly simplified) to return this data structure (for image arrays, header keywords, and I am working on table columns now).

Gnuastro's Arithmetic has been fully migrated to this new data structure, and as it currently stands, it is the only program that works as expected in this branch. The main work-horse behind arithmetic on this data structure is now a library, so library users can also benefit from them. Until now, Arithmetic would convert the input data to double type, do the arithmetic operations, then save the results in the desired type. When the data sets get large (>1Gb for example), this conversion itself consumes a large amount of CPU and memory resources. Thanks to this data structure, we arithmetic library functions will now do all the work on the native type of the data, so no conversion is necessary, and is a great step towards better performance.

So in case you want to compile/test this branch please configure Gnuastro with `--enable-arithmetic --enable-table' (the new FITS functions are now corrected in the other programs yet).

Just one note: to efficiently account for native arithmetic operations on all the combinations of the different data types (especially for the binary operators that need two operands), the only way I could come up with until now was heavy usage of macros, for example see data-arithmetic-binary.c.

These macros greatly help in efficient running of the binary arithmetic operators, but it will greatly lengthen the compilation time. For example, to allow all the types to be natively operated on, on my system with default configure options, the compilation will take about 20 minutes! So I have defined configure time options to specify which types to allow for native binary arithmetic (`--enable-bin-op-XXXX', where `XXXX' is the type name). If the data type is not native, it will be converted to a native type, the arithmetic will be done and the output will be set to the proper type. I have tried to fully describe this under these options in the "Configuring" subsection of the book. With the currently default native types, it takes about 2 minutes to compile by default. But if compilation time is very important for you, you can disable optimization and shared libraries (like in debugging mode) and compilation time will be like before (less than a minute).

I am now working on fully implementing Gnuastro's Table program to use this data structure. Reading of FITS tables has been implemented, and I am now working on the writing and also reading/writing of ASCII tables. Once table columns can also be easily read using this structure, most of the job will be done: since the inputs/outputs of the programs are either table columns or images/arrays.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 21 Nov 2016 02:50:58 PM UTC, original submission:  

In an analysis, data can have multiple dimensions: in a table, each column only has one dimension, in an image we have a 2D dataset and in a datacube, we have 3D data.

Currently in Gnuastro all the different information about a dataset is kept in the main program structures and when passed to a library it is neccesary to pass along all this information. For example the array keeping the data, the size of the array (in 1D or 2D, so far), its WCS information, if it has any blank values and so on. This makes reading the code harder and can also be buggy. It also makes general modifications (for example enabling Gnuastro's operation on 3D datasets) hard.

So I am now working on a general library header and functions for data arrays (`data.h', see it in my fork). The basic idea is that all the major arrays in Gnuastro be converted to this format to make it much more easier to deal with any general array/dataset.

Most of the functions to do with datatypes (for example converting one datatype to another) which are currently defined in `fits.h' are going to be moved in this header.

This is currently a low-level work in progress and will take some time, so please share your thoughts and suggestions if you have any. I am pushing my work on it in my development fork under the datastruct branch.

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-04-24 makhlaghi StatusIn Progress Done
        Percent Complete30% 100%
        Open/ClosedOpen Closed
    2017-02-01 makhlaghi Percent Complete20% 30%
    2016-12-28 makhlaghi Percent Complete10% 20%

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code