bugmake - Bugs: bug #57178, Allow $(shell $(MAKE) ...) to...

 
 

bug #57178: Allow $(shell $(MAKE) ...) to participate in jobserver

Submitted by:  Paul Berg <paulberg>
Submitted on:  Wed 06 Nov 2019 11:07:42 AM UTC  
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.2.1 Operating System:  Any
Fixed Release:  None Triage Status:  None

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 17 Dec 2019 09:51:40 PM UTC, comment #3: 

That makes sense. My use case was not critical, it just appeared to be odd behavior. Perhaps it should be added to the documentation?

My use case was that I needed a third party component built which utilized make prior before my makefile's macros could be fully resolved and I was trying to 1) avoid a full second pass restart and 2) respect the jobserver flags passed in when calling make on the 3rd party component. An unrelated dependency ended up requiring a full restart anyway so I ended up tucking the 3rd party component in that first pass as a standard submake, so this currently isn't affecting me.

I can see an argument for adding this behavior, but it is clear now that it is intended and would be a feature request, not a bug. I do think a blub in the docs would be helpful for those who come after me though and I'd agree this can be resolved.

Paul Berg <paulberg>
Tue 17 Dec 2019 08:48:33 PM UTC, comment #2: 

This issue isn't really about "the first pass", it's about the shell function.  Even if you were to use the shell function (outside of a recipe) in the second pass, for example via secondary expansion, you'd still see this same behavior.

It doesn't have anything to do with the setting of MAKEFLAGS.

Make disables the jobserver when it determines that the process it's about to invoke is not a sub-make.  This is needed because some programs do not expect to have extra file descriptors available and may even corrupt the jobserver pipe.

In recipes make detects this by looking for the '+' recipe prefix or else detecting $(MAKE) or ${MAKE} in the recipe.

However, this detection is never done for the shell function: we always disable jobserver information even if the text of the shell function contains $(MAKE) or ${MAKE}.

This could be resolved.  However, I do urge you to consider a different method for solving your real problem, rather than running a subshell inside a shell function.  There are a lot of issues with this (for example, error handling just to name one).

Paul D. Smith <psmith>
Project Administrator
Fri 08 Nov 2019 08:36:45 PM UTC, comment #1: 

Looks like MAKEFLAGS var is not set at this point. Possibly related?

Paul Berg <paulberg>
Wed 06 Nov 2019 11:07:42 AM UTC, original submission:  

In a recipe, a call to a sub-make process using the $(MAKE) macro coordinates with the job server to maintain a maximum number of parallel jobs as instructed by the -j parameter. It is expected that any use of this macro would do the same, but in the initial expansion, use of this macro does not coordinate and is restricted to -j1.

To Repro use 2 files (below) Makefile and test.mk. test.mk called with -j10 will return 10 normally (some race conditions possible). However, if Makefile is called with -j10, it invokes test.mk as a sub-make and test.mk will report that only -j1 was invoked, indicating that the job server is not being utilized.

Expected: This test when invoked with -j n should report n (or close to n, race conditions permitting) since the job server should coordinate the number of available jobs and the sub process should parallelize accordingly.

FILE1 Makefile:

$(info $(shell $(MAKE) -f test.mk all))

all:
@:

FILE2 test.mk:

all: | parallel ; @echo $(JOB_COUNT)

parallel: .parallel ; @$(eval JOB_COUNT := $(shell sort -n $< | tail -n 1))
.parallel: FORCE ; @$(MAKE) -f test.mk --no-print-directory par 2>/dev/null >$@ || true
FORCE:

to_n = $(words $2) $(if $(filter-out $1,$(words x $2)),$(call to_n,$1,x $2))

PAR_COUNT :=
par: $(addprefix par-,$(call to_n,32))
par-%: ; @$(eval PAR_COUNT += x)@echo $(words $(PAR_COUNT)) && sleep 1 && false

Paul Berg <paulberg>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by paulberg (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-17 psmith Item GroupBug => Enhancement
        SummaryJob server not invoked in initial pass => Allow $(shell $(MAKE) ...) to participate in jobserver

    Back to the top


    Powered by Savane 3.5