taskGNU Astronomy Utilities - Tasks: task #16251, Using Make's jobserver for...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #16251: Using Make's jobserver for parallelization

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Sun 04 Sep 2022 12:50:05 PM UTC
   
 
Should Start On:  Sat 03 Sep 2022 11:00:00 PM UTC Should be Finished on:  Sat 03 Sep 2022 11:00:00 PM UTC
Category:  All Gnuastro Priority:  5 - Normal
Item Group:  Enhancement Status:  None
Privacy:  Public Assigned to:  None
Percent Complete:  0% Open/Closed:  Open
Effort:  0.00

Sun 04 Sep 2022 12:50:05 PM UTC, original submission:  

One of the most common high-level frameworks where Gnuastro is run is GNU Make since it manages dependencies and can run in Parallel.

When running in Parallel, by default, each program counts the total number of CPU threads and uses all of them. So if a CPU has 8 cores, and a user runs Make with '-j8' (to use all those 8 threads), and each rule has one NoiseChisel call within it (which will use all 8 cores by default), the system will have a queue of 8x8=64 jobs! It is possible to manage this by setting all Gnuastro programs to use one thread in a 'gnuastro.conf' configuration file, but that is not usually the most effective way either (there might be times that more than one thread is available).

For such situations, Make has the "jobserver" concept, which is described in the Sharing Job Slots with GNU make. Since all Gnuastro programs use a single function to spin-off threads, it should be relatively easy to implement this in Gnuastro; so it integrates with GNU Make and doesn't add too many jobs to the queue.

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code