bugmake - Bugs: bug #57178, Allow $(shell $(MAKE) ...) to...

 
 

bug #57178: Allow $(shell $(MAKE) ...) to participate in jobserver

Submitter:  Paul Berg <paulberg>
Submitted:  Wed 06 Nov 2019 11:07:42 AM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  4.2.1 Operating System:  Any
Fixed Release:  4.4 Triage Status:  Medium Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 03 Aug 2022 03:43:23 AM UTC, comment #4: 

The next release of GNU make supports the jobserver running using a named pipe; this will allow this to work properly.

Paul D. Smith <psmith>
Group administrator
Tue 17 Dec 2019 09:51:40 PM UTC, comment #3: 

That makes sense. My use case was not critical, it just appeared to be odd behavior. Perhaps it should be added to the documentation?

My use case was that I needed a third party component built which utilized make prior before my makefile's macros could be fully resolved and I was trying to 1) avoid a full second pass restart and 2) respect the jobserver flags passed in when calling make on the 3rd party component. An unrelated dependency ended up requiring a full restart anyway so I ended up tucking the 3rd party component in that first pass as a standard submake, so this currently isn't affecting me.

I can see an argument for adding this behavior, but it is clear now that it is intended and would be a feature request, not a bug. I do think a blub in the docs would be helpful for those who come after me though and I'd agree this can be resolved.

Paul Berg <paulberg>
Tue 17 Dec 2019 08:48:33 PM UTC, comment #2: 

This issue isn't really about "the first pass", it's about the shell function.  Even if you were to use the shell function (outside of a recipe) in the second pass, for example via secondary expansion, you'd still see this same behavior.

It doesn't have anything to do with the setting of MAKEFLAGS.

Make disables the jobserver when it determines that the process it's about to invoke is not a sub-make.  This is needed because some programs do not expect to have extra file descriptors available and may even corrupt the jobserver pipe.

In recipes make detects this by looking for the '+' recipe prefix or else detecting $(MAKE) or ${MAKE} in the recipe.

However, this detection is never done for the shell function: we always disable jobserver information even if the text of the shell function contains $(MAKE) or ${MAKE}.

This could be resolved.  However, I do urge you to consider a different method for solving your real problem, rather than running a subshell inside a shell function.  There are a lot of issues with this (for example, error handling just to name one).

Paul D. Smith <psmith>
Group administrator
Fri 08 Nov 2019 08:36:45 PM UTC, comment #1: 

Looks like MAKEFLAGS var is not set at this point. Possibly related?

Paul Berg <paulberg>
Wed 06 Nov 2019 11:07:42 AM UTC, original submission:  

In a recipe, a call to a sub-make process using the $(MAKE) macro coordinates with the job server to maintain a maximum number of parallel jobs as instructed by the -j parameter. It is expected that any use of this macro would do the same, but in the initial expansion, use of this macro does not coordinate and is restricted to -j1.

To Repro use 2 files (below) Makefile and test.mk. test.mk called with -j10 will return 10 normally (some race conditions possible). However, if Makefile is called with -j10, it invokes test.mk as a sub-make and test.mk will report that only -j1 was invoked, indicating that the job server is not being utilized.

Expected: This test when invoked with -j n should report n (or close to n, race conditions permitting) since the job server should coordinate the number of available jobs and the sub process should parallelize accordingly.

FILE1 Makefile:

$(info $(shell $(MAKE) -f test.mk all))

all:
@:

FILE2 test.mk:

all: | parallel ; @echo $(JOB_COUNT)

parallel: .parallel ; @$(eval JOB_COUNT := $(shell sort -n $< | tail -n 1))
.parallel: FORCE ; @$(MAKE) -f test.mk --no-print-directory par 2>/dev/null >$@ || true
FORCE:

to_n = $(words $2) $(if $(filter-out $1,$(words x $2)),$(call to_n,$1,x $2))

PAR_COUNT :=
par: $(addprefix par-,$(call to_n,32))
par-%: ; @$(eval PAR_COUNT += x)@echo $(words $(PAR_COUNT)) && sleep 1 && false

Paul Berg <paulberg>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by paulberg (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-08-03 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.4
        Triage StatusNone Medium Effort
    2019-12-17 psmith Item GroupBug Enhancement
        SummaryJob server not invoked in initial pass Allow $(shell $(MAKE) ...) to participate in jobserver

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code