bugmake - Bugs: bug #27714, expansion of $(shell) in target...

 
 

bug #27714: expansion of $(shell) in target forces serialization of targets

Submitted by:  Mike Frysinger <vapier>
Submitted on:  Fri 16 Oct 2009 07:50:48 AM UTC  
 
Severity: 3 - NormalItem Group: Bug
Status: Wont FixPrivacy: Public
Assigned to: NoneOpen/Closed: Closed
Component Version: 3.81Operating System: POSIX-Based
Fixed Release: NoneTriage Status: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Fri 02 Jul 2010 04:07:54 PM UTC, comment #10:

I'm going to close this as "won't fix" because I don't see any reasonable way to implement it that doesn't involve massive changes to the way make works, and I don't think the capability to have "parallel function expansion" is worth the effort.

If you want to invoke shell commands in a recipe you don't have to run $(shell ...)... the recipe itself is run by the shell so why use the shell function? Just put the commands that the shell function would run directly into the recipe. Then they will be passed to the subshell and run in parallel just like the rest of the command script.

Paul D. Smith <psmith>
Project Administrator
Sat 21 Nov 2009 02:38:27 PM UTC, comment #9:

No, that's not equivalent. First, your GNU make syntax is wrong; you mean "FOO = $(shell echo bar)". And second, as far as I can see from the pmake documentation "FOO != echo bar" is actually equivalent to "FOO := $(shell echo bar)", so it doesn't apply to the issue discussed in this bug (which is that the expansion of the shell happens when the recipe is evaluated, not when the makefile is read in).

Paul D. Smith <psmith>
Project Administrator
Sat 21 Nov 2009 09:24:46 AM UTC, comment #8:

# gmake
FOO= $(echo bar)

# pmake (BSD)
FOO!= echo bar

Anonymous
Mon 19 Oct 2009 09:47:46 PM UTC, comment #7:

I'm surprised you say you can do this with BSD make; I'm not aware that BSD make has any feature like the $(shell ...) function, that can appear inside a recipe. What syntax do you use for this in BSD make?

We can't expand the recipe in a forked version of GNU make, because the expansion of the recipe can have side-effects (for example, if an $(eval ...) is expanded); if the expansion happens in a forked process then those side effects are not available in the parent.

Paul D. Smith <psmith>
Project Administrator
Mon 19 Oct 2009 07:45:54 PM UTC, comment #6:

i dont really know how GNU make is architected, but i dont see why make cant fork the process to handle the target and then the subprocess does both deferred expansion and running the commands listed for that target.

i'd also point out that other make implementations (such as BSD makes) handle this the way i expect -- in parallel.

Mike Frysinger <vapier>
Mon 19 Oct 2009 06:35:05 PM UTC, comment #5:

Hrm. The only way this could be "fixed" would be to make the $(shell ...) function a true make job, so that it takes a job token, can run in the background asynchronously, and is basically handled as an anonymous recipe line that is run before the "real" recipe line.

This would be almost impossible to do, because it would mean we'd have to introduce an entirely new concept in make: the ability to have "deferred variable expansion", where we say "please expand this string" and instead of getting the expanded string back, we'd get back a response saying "I started it, come back later". Then later, sometime, we'd have to check to see if it completed, and somehow restart our expansion at that time. Allowing expansion to happen asynchronously would mean completely rewriting a very significant portion of make.

Paul D. Smith <psmith>
Project Administrator
Mon 19 Oct 2009 12:18:34 AM UTC, comment #4:

i understand what you're saying, but i think the behavior you describe as "expected" is actually a bug. deferred variable expansion for a target shouldnt affect make's ability to run other targets.

Mike Frysinger <vapier>
Fri 16 Oct 2009 11:34:21 AM UTC, comment #3:

Hi Mike; no, sorry, I should have given an explicit example. Your new test case is not correct and doesn't prove anything. I explicitly said do NOT put the sleep inside the $(shell). It's important to understand that make itself is single threaded (it doesn't use pthread() etc.). Its parallelism is gained by having the recipes it invokes be run in parallel. But before it can invoke the recipe it has to expand all the variables and functions. That includes $(shell ...) functions. Variables and functions (including $(shell ...) functions) are NOT expanded "in parallel"; they are expanded one at a time as they are seen in the recipe.

After the entire recipe is expanded, THEN make invokes the command, then it leaves the command running in the background while it locates the next possible recipe to start, evaluates it, then runs it in the background, etc.

So, causing the recipe expansion to take a long time (which is what you're doing by adding the sleep to the $(shell ...) function) actually decreases the amount of parallelism. Hopefully that's clear.

What you need to do to show the make is working correctly is increase the time it takes the recipe itself to run: that is, add the sleep OUTSIDE the $(shell ...). Something like:

You should see the recipes all run in parallel (the first printf's all run, then a pause, then the second printfs all run).

Paul D. Smith <psmith>
Project Administrator
Fri 16 Oct 2009 08:13:12 AM UTC, comment #2:

i didnt think it a race condition because the original test case had a fairly lengthy $(shell) code and the targets always ran in the same order (and obviously only one cpu at a time -- the other cpus stayed idle)

using sleep shows that it is indeed a make bug. i see one target show up every 10 seconds:
$ cat makefile
t = 0 a b c 1 d e f 2 g h i 3 j k l 4 m n o 5 p q r 6
all: $(t) ; @echo
$(t): ; @printf $@ $(shell sleep 10)

$ make -f makefile
<10 seconds>0<10 seconds>a<10 seconds>b............

Mike Frysinger <vapier>
Fri 16 Oct 2009 07:58:27 AM UTC, comment #1:

My suspicion is that this is just an artifact of your test case. In your scenario, make must evaluate the $(shell ...) function (invoking the shell) while it's expanding the command to be invoked. What I think is happening here is that your actual command is so fast that by the time make is ready to invoke the next command (after it ran the $(shell :) for that target), the previous command is already completed. So, it looks like things are happening serially even though make is doing its best to run them in parallel.

Please try this test again, but have the command you invoke take longer; for example add a "sleep 1" or something after your printf (not inside the $(shell ...) function!), then print it again maybe. You should see all the first prints show up (running in parallel) then all the second prints.

Paul D. Smith <psmith>
Project Administrator
Fri 16 Oct 2009 07:50:48 AM UTC, original submission:

i noticed a package recently that would build serially even when i gave it -j (on a quad machine). looking into it, the reason was that $(shell) was appended to a compiler flag leading to it being expanded when the target was run:
CFLAGS += $(shell .....)

forcing the $(shell) to be evaluated immediately allowed things to build in parallel:
SOME_CHECK := $(shell .....)
CFLAGS += $(SOME_CHECK)

behavior is seen with make-3.80 and make-3.81

here is a simple example:
$ cat makefile
t = 0 a b c 1 d e f 2 g h i 3 j k l 4 m n o 5 p q r 6
all: $(t)
@echo
$(t):
@printf $@ $(shell :)

$ while make -f makefile -j ; do : ; done
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mno5pqr6

removing the $(shell :) allows for parallel evaluation:
0abc1de2gi3hjkl4mnoqr56p
0abc1de2ghi3jkl4mn5pqor6
0acde21gbhi3jkl4mno5pqr6
0abc1de2ghi3jkl4mn5qr6po
0abc1d2gehi3jkl4o5mpnqr6
0abc1de2ghi3jkl4mno5pq6r
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jl4n5pkqorm6
0abc1de2ghi3jkl4mno5pqr6
0abc1de2ghi3jkl4mnpqor56

Mike Frysinger <vapier>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by psmith (Posted a comment)
  • -unavailable- added by vapier (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Fri 02 Jul 2010 04:07:54 PM UTCpsmithStatusNone=>Wont Fix
      Open/ClosedOpen=>Closed

    Back to the top


    Powered by Savane 3.1-cleanup1