taskGNU Astronomy Utilities - Tasks: task #16251, Using Make's jobserver for...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #16251: Using Make's jobserver for parallelization

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Sun 04 Sep 2022 12:50:05 PM UTC
   
 
Should Start On:  Sat 03 Sep 2022 11:00:00 PM UTC Should be Finished on:  Sat 03 Sep 2022 11:00:00 PM UTC
Category:  All Gnuastro Priority:  5 - Normal
Item Group:  Enhancement Status:  None
Privacy:  Public Assigned to:  None
Percent Complete:  0% Open/Closed:  Open
Effort:  0.00

Discussion

Sun 04 Sep 2022 12:50:05 PM UTC, original submission:  

One of the most common high-level frameworks where Gnuastro is run is GNU Make since it manages dependencies and can run in Parallel.

When running in Parallel, by default, each program counts the total number of CPU threads and uses all of them. So if a CPU has 8 cores, and a user runs Make with '-j8' (to use all those 8 threads), and each rule has one NoiseChisel call within it (which will use all 8 cores by default), the system will have a queue of 8x8=64 jobs! It is possible to manage this by setting all Gnuastro programs to use one thread in a 'gnuastro.conf' configuration file, but that is not usually the most effective way either (there might be times that more than one thread is available).

For such situations, Make has the "jobserver" concept, which is described in the Sharing Job Slots with GNU make. Since all Gnuastro programs use a single function to spin-off threads, it should be relatively easy to implement this in Gnuastro; so it integrates with GNU Make and doesn't add too many jobs to the queue.

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

Attached Files

This item currently has no attached files.

 

Dependencies

This item does not depend on any other items.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    Votes

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    History

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.16-11ef.
    Corresponding source code