bugmake - Bugs: bug #51200, Improvement suggestion: listen to...

 
 

bug #51200: Improvement suggestion: listen to signals to adjust number of jobs

Submitter:  None
Submitted:  Wed 07 Jun 2017 06:43:43 PM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.3 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 20 Mar 2023 05:58:21 PM UTC, comment #3: 

My latest patch signal_num_jobs6.patch is almost a complete rewrite to work better with recursive calls to make. Jobs are simply added by putting tokens in the pipe from the signal routine for SIGUSR2. Revoking jobs are slightly more tricky, a separate process is spawned with fork which tries to remove a token from the pipe. However, some jobs might be able to start before the process is able to revoke a token. Making that process busy waiting on the pipe much increases the probability that it will become the first process to get a token, but I didn't want to do that. Another option might have been to make the ordinary make processes sleep some ms each time before they pick a token, but I didn't want to do that either. My choice was to live with the fact that it might take a long time before the number of jobs are decreased.

An slightly intrusive change from this patch is that the job server is always setup initially. This is as it cannot be done in a signal safe way from the signal handlers, it has to be ready when the signal handlers need them. This also means that the job server is setup when -j is called without a value. That was supposed to give an infinite number of jobs, but the job server has to have a limit. My choice for now was to set that limit to 10000 jobs. In practice 10000 jobs are enough to be backwards compatible with the old behavior of make to by default become kind of a fork bomb on a big source tree to build when no value is given to -j. By definition 10000 jobs are not an infinite number of jobs, but in practice I see no big difference. Maybe I would prefer to by default limit the number of jobs to a low number and let the people who really know what they are doing specify higher numbers if they want to, but that would break backwards compatibility.

(file #54516)

Henrik Carlqvist <henca>
Sat 11 Mar 2023 05:15:46 PM UTC, comment #2: 

The latest version of the patch (signal_num_jobs5.patch) no longer immediately increases or decreases the number of jobs when signals USR2 or USR1 are received, instead, for signal safety, the number of running jobs will be adjusted when a job is finished.

For those who do not want to apply the patch themselves, I have cloned the git repo at https://github.com/henca/Henriks-make/tags and applied my patch to tag 4.4.1h.

Henrik Carlqvist <henca>
Mon 20 Nov 2017 05:58:31 AM UTC, comment #1: 

As the directory structure recently changed in git, would you like me to send an updated patch?

regards Henrik

Henrik Carlqvist <henca>
Wed 07 Jun 2017 06:43:43 PM UTC, original submission:  

Many times after starting make I have wished that I started it with more parallel jobs than I did. A few time I have also wished that I started make with fewer parallel jobs.

The attached patch allows you to send SIGUSR2 to your top level make process to add one more parallel job.

Once a SIGUSR2 is received the functionality of SIGUSR1 also changes from toggling debug output to decrease the number of jobs by 1. No running job will be killed, but when the next target is finished one less job will be started again.

If make was started with parallel jobs SIGUSR2 will immediately increase the number of jobs with one. If make was started without -j SIGUSR2 will not cause make to work in parallel until the current target is finished.

The patch was written against sources from git, but also works with latest stable version 4.2.1.

I hope that you will find my patch useful and that it somehow will make it
into upcoming stable releases of make.

Please feel free to discuss improvements of the patch, I have subscribed to the bug-make mailing list.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #54516:  signal_num_jobs6.patch added by henca (5KiB - text/x-patch - Adjusting the number of jobs with signal safe calls from the signal functions to work better with recursive calls of make. This patch applies cleanly to tag 4.4.1 which at the time of this writing is the latest commit to master.)
file #54515:  signal_num_jobs6.patch added by henca (5KiB - text/x-patch - Adjusting the number of jobs with signal safe calls from the signal functions to work better with recursive calls of make. This patch applies cleanly to tag 4.4.1 which at the time of this writing is the latest commit to master.)
file #49430:  signal_num_jobs5.patch added by henca (3KiB - text/x-patch - Updated patch with current directory structure as of version 4.3 of make. Also avoiding all non-signal-safe function calls from signal functions.)
file #40877:  signal_num_jobs3.patch added by None (3KiB - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by henca (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-03-20 henca Attached File- Added signal_num_jobs6.patch, #54516
    2023-03-20 henca Attached File- Added signal_num_jobs6.patch, #54515
    2020-07-04 henca Attached File- Added signal_num_jobs5.patch, #49430
    2017-06-07 None Attached File- Added signal_num_jobs3.patch, #40877

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code