GNU Astronomy Utilities - Tasks: task #16251, Using Make's jobserver for...
You are not allowed to post comments on this tracker with your current authentication level.
task #16251: Using Make's jobserver for parallelization
Submitter: | Mohammad Akhlaghi <makhlaghi> | ||
Submitted: | Sun 04 Sep 2022 12:50:05 PM UTC | ||
Should Start On: | Sat 03 Sep 2022 11:00:00 PM UTC | Should be Finished on: | Sat 03 Sep 2022 11:00:00 PM UTC |
Category: | All Gnuastro | Priority: | 5 - Normal |
Item Group: | Enhancement | Status: | None |
Privacy: | Public | Assigned to: | None |
Percent Complete: | 0% | Open/Closed: | Open |
Effort: | 0.00 |
No files currently attached
Depends on the following items: None found
Items that depend on this one: None found
Carbon-Copy List
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
No changes have been made to this item
Powered by Savane 3.13-758e.
Corresponding source code
One of the most common high-level frameworks where Gnuastro is run is GNU Make since it manages dependencies and can run in Parallel.
When running in Parallel, by default, each program counts the total number of CPU threads and uses all of them. So if a CPU has 8 cores, and a user runs Make with '-j8' (to use all those 8 threads), and each rule has one NoiseChisel call within it (which will use all 8 cores by default), the system will have a queue of 8x8=64 jobs! It is possible to manage this by setting all Gnuastro programs to use one thread in a 'gnuastro.conf' configuration file, but that is not usually the most effective way either (there might be times that more than one thread is available).
For such situations, Make has the "jobserver" concept, which is described in the Sharing Job Slots with GNU make. Since all Gnuastro programs use a single function to spin-off threads, it should be relatively easy to implement this in Gnuastro; so it integrates with GNU Make and doesn't add too many jobs to the queue.