bugmake - Bugs: bug #27714, expansion of $(shell) in target...

 
 

bug #27714: expansion of $(shell) in target forces serialization of targets

Submitter:  Mike Frysinger <vapier>
Submitted:  Fri 16 Oct 2009 07:50:48 AM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  Wont Fix Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Component Version:  3.81 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 02 Jul 2010 04:07:54 PM UTC, comment #10: 

I'm going to close this as "won't fix" because I don't see any reasonable way to implement it that doesn't involve massive changes to the way make works, and I don't think the capability to have "parallel function expansion" is worth the effort.

If you want to invoke shell commands in a recipe you don't have to run $(shell ...)... the recipe itself is run by the shell so why use the shell function?  Just put the commands that the shell function would run directly into the recipe.  Then they will be passed to the subshell and run in parallel just like the rest of the command script.

Paul D. Smith <psmith>
Group administrator
Sat 21 Nov 2009 02:38:27 PM UTC, comment #9: 

No, that's not equivalent.  First, your GNU make syntax is wrong; you mean "FOO = $(shell echo bar)".  And second, as far as I can see from the pmake documentation "FOO != echo bar" is actually equivalent to "FOO := $(shell echo bar)", so it doesn't apply to the issue discussed in this bug (which is that the expansion of the shell happens when the recipe is evaluated, not when the makefile is read in).

Paul D. Smith <psmith>
Group administrator
Sat 21 Nov 2009 09:24:46 AM UTC, comment #8: 

# gmake
FOO=   $(echo bar)

# pmake (BSD)
FOO!=   echo bar

Anonymous
Mon 19 Oct 2009 09:47:46 PM UTC, comment #7: 

I'm surprised you say you can do this with BSD make; I'm not aware that BSD make has any feature like the $(shell ...) function, that can appear inside a recipe.  What syntax do you use for this in BSD make?

We can't expand the recipe in a forked version of GNU make, because the expansion of the recipe can have side-effects (for example, if an $(eval ...) is expanded); if the expansion happens in a forked process then those side effects are not available in the parent.

Paul D. Smith <psmith>
Group administrator
Mon 19 Oct 2009 07:45:54 PM UTC, comment #6: 

i dont really know how GNU make is architected, but i dont see why make cant fork the process to handle the target and then the subprocess does both deferred expansion and running the commands listed for that target.

i'd also point out that other make implementations (such as BSD makes) handle this the way i expect -- in parallel.

Mike Frysinger <vapier>
Mon 19 Oct 2009 06:35:05 PM UTC, comment #5: 

Hrm.  The only way this could be "fixed" would be to make the $(shell ...) function a true make job, so that it takes a job token, can run in the background asynchronously, and is basically handled as an anonymous recipe line that is run before the "real" recipe line.

This would be almost impossible to do, because it would mean we'd have to introduce an entirely new concept in make: the ability to have "deferred variable expansion", where we say "please expand this string" and instead of getting the expanded string back, we'd get back a response saying "I started it, come back later".   Then later, sometime, we'd have to check to see if it completed, and somehow restart our expansion at that time.  Allowing expansion to happen asynchronously would mean completely rewriting a very significant portion of make.

Paul D. Smith <psmith>
Group administrator
Mon 19 Oct 2009 12:18:34 AM UTC, comment #4: 

i understand what you're saying, but i think the behavior you describe as "expected" is actually a bug.  deferred variable expansion for a target shouldnt affect make's ability to run other targets.

Mike Frysinger <vapier>
Fri 16 Oct 2009 11:34:21 AM UTC, comment #3: 

Hi Mike; no, sorry, I should have given an explicit example.  Your new test case is not correct and doesn't prove anything.  I explicitly said do NOT put the sleep inside the $(shell).  It's important to understand that make itself is single threaded (it doesn't use pthread() etc.).  Its parallelism is gained by having the recipes it invokes be run in parallel.  But before it can invoke the recipe it has to expand all the variables and functions.  That includes $(shell ...) functions.  Variables and functions (including $(shell ...) functions) are NOT expanded "in parallel"; they are expanded one at a time as they are seen in the recipe.

After the entire recipe is expanded, THEN make invokes the command, then it leaves the command running in the background while it locates the next possible recipe to start, evaluates it, then runs it in the background, etc.

So, causing the recipe expansion to take a long time (which is what you're doing by adding the sleep to the $(shell ...) function) actually decreases the amount of parallelism.  Hopefully that's clear.

What you need to do to show the make is working correctly is increase the time it takes the recipe itself to run: that is, add the sleep OUTSIDE the $(shell ...).  Something like:


$(t): ; @printf $@ $(shell :); sleep 5; printf $@


You should see the recipes all run in parallel (the first printf's all run, then a pause, then the second printfs all run).

Paul D. Smith <psmith>
Group administrator
Fri 16 Oct 2009 08:13:12 AM UTC, comment #2: 

i didnt think it a race condition because the original test case had a fairly lengthy $(shell) code and the targets always ran in the same order (and obviously only one cpu at a time -- the other cpus stayed idle)

using sleep shows that it is indeed a make bug.  i see one target show up every 10 seconds:
$ cat makefile
t = 0 a b c 1 d e f 2 g h i 3 j k l 4 m n o 5 p q r 6
all: $(t) ; @echo
$(t): ; @printf $@ $(shell sleep 10)

$ make -f makefile
<10 seconds>0<10 seconds>a<10 seconds>b............

Mike Frysinger <vapier>
Fri 16 Oct 2009 07:58:27 AM UTC, comment #1: 

My suspicion is that this is just an artifact of your test case.  In your scenario, make must evaluate the $(shell ...) function (invoking the shell) while it's expanding the command to be invoked.  What I think is happening here is that your actual command is so fast that by the time make is ready to invoke the next command (after it ran the $(shell :) for that target), the previous command is already completed.  So, it looks like things are happening serially even though make is doing its best to run them in parallel.

Please try this test again, but have the command you invoke take longer; for example add a "sleep 1" or something after your printf (not inside the $(shell ...) function!), then print it again maybe.  You should see all the first prints show up (running in parallel) then all the second prints.

Paul D. Smith <psmith>
Group administrator
Fri 16 Oct 2009 07:50:48 AM UTC, original submission:  

i noticed a package recently that would build serially even when i gave it -j (on a quad machine).  looking into it, the reason was that $(shell) was appended to a compiler flag leading to it being expanded when the target was run:
CFLAGS += $(shell .....)

forcing the $(shell) to be evaluated immediately allowed things to build in parallel:
SOME_CHECK := $(shell .....)
CFLAGS += $(SOME_CHECK)

behavior is seen with make-3.80 and make-3.81

here is a simple example:
$ cat makefile
t = 0 a b c 1 d e f 2 g h i 3 j k l 4 m n o 5 p q r 6
all: $(t)
        @echo
$(t):
        @printf $@ $(shell :)

$ while make -f makefile -j ; do : ; done
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6

removing the $(shell :) allows for parallel evaluation:
0abc1de2gi3hjkl4mnoqr56p
0abc1de2ghi3jkl4mn5pqor6
0acde21gbhi3jkl4mno5pqr6
0abc1de2ghi3jkl4mn5qr6po
0abc1d2gehi3jkl4o5mpnqr6
0abc1de2ghi3jkl4mno5pq6r
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jl4n5pkqorm6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mnpqor56

Mike Frysinger <vapier>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by vapier (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-07-02 psmith StatusNone Wont Fix
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code