mainGNU Astronomy Utilities - Support: sr #110636, asttable is very slow with many...

 
 

sr #110636: asttable is very slow with many columns

Submitter:  None
Submitted:  Thu 07 Apr 2022 11:38:30 PM UTC
   
 
Category:  Table Priority:  5 - Normal
Severity:  3 - Normal Item Group:  * Enhancement
Status:  None Privacy:  Public
Assigned to:  None Originator Email:  -email is unavailable-
Open/Closed:  Open Operating System:  GNU/Linux
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 10 Apr 2022 11:41:04 PM UTC, comment #1: 

Thanks a lot for reporting this.

I was able to reproduce the problem with this manually created table of 10e5 columns:


echo 1 100000 | awk '{for(i=0;i<$2;++i) printf "%d ", $1+i; print ""}' > many-columns.txt

asttable many-columns.txt -i


Increasing the column count to 1e6 did indeed take very long and forced me to manually close it too.

The problem with plain-text tables is that the reading can't be easily parallelized (with FITS tables, since we know the number a-priori, in Gnuastro, we parallelize the column reading).

If you do come up with a good solution for plain-text tables in NEMO, please share it here.

In Gnuastro, we do have a format to give metadata to plain-text tables. But that also won't necessarily help unless all columns are string (their number of characters are known a-priori).

Generally, besides theoretical checking of code, I am curious if there is any real-world scenario that would require so many columns? Usually the workflow can be re-designed (optimized) to avoid so many columns in one file. For example FITS only accepts 999 columns.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 07 Apr 2022 11:38:30 PM UTC, original submission:  

I don't know if this is to be expected, but when I ran

     asttable --information junk.tab

with 10000 columns (and 10 rows), it's 0.25sec,but with 1e5 columns it took 99sec, and with 1e6 columns, it's still running after 17 mins so I'll kill it.

Looking at memory consumption, it's very modest. RES=61M while the table has 10 lines and is 86M in size.

The background here is the re-design of the table system in NEMO, where we have some comparison possible with the gnuastro tables

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.14-f13d.
    Corresponding source code