Hi!  I am sad to say, I missed that this bug had been fixed at the time.  Thank you so much for incorporating the fix!

I can confirm that using make 4.2.1 fixes this problem for us, and we no longer have to carry this patch locally.

Additionally, I checked whether the subsequent, not yet released, change to posix_spawn affects performance.  It appears to be a slight regression, with 4.2.1 taking about 16.3s on my current test system (8-core digitalocean "CPU optimized" droplet) and git taking 17.5s.  Using "./configure --disable-posix-spawn" goes back down to 16.4s, so most of the slowdown is due to posix_spawn. (run to run variance is on the order of .1s at most, so 16.3 -> 17.5 is actually a difference, but 16.3 vs 16.4 may not be)

(these timings are on the "mkmk.sh" testcase I posted in an earlier comment, on a digitalocean 8-core "CPU-optimized" droplet running Debian 10 "Stretch")

This has become off-topic for the current bug report and anyway it was discussed exhaustively back then; details are in mail archives. But in any case, whether you agree or disagree after reading, it's water well past the bridge by now.

Bisecting between 3.81 and current Git HEAD will not be very productive.  3.81 was released almost 10 years ago.  There have been many performance regressions, and many performance improvements over that time.  It's likely that some of the regressions are related to new features, and so it's not possible to just remove those changes (I know, for example, that the new feature allowing for pattern rules to match most specifically rather than "first rule matches" added a lot of complexity and had some performance impacts, especially if you have lots of pattern rules--I had an idea just the other day that might simplify this but I'd need to play with it).

Rather than spend time bisecting it's much, much better to spend time looking at the current behavior of Git HEAD with perf tools and discover what is slow in your build environment.  Reports like that can, hopefully, be addressed.

OK, I'll put it back for the next release.

I see absolutely the same results as noted.

And there is NO DOUBT that the fork/vfork is to blame!

I tested today on any single commit since 3.75(over 1000 versions!), and every single one of those versions up to 94735f0ad7f67c56afa1513381c73e8f62cd44fc (3.82+156, i.e. the 156th commit after 3.82) had almost identical results, where at that version it suddenly bloated to those tedious time-frames noted by others.

To summarize: bring back vfork()!

I can confirm the bug report and that reverting 94735f0 solves the problem. For us this is a very critical problem, since build times are increased so drastically.

We've now deployed our patched GNU make with the patch from comment #2 and the results are positive--the performance regression seen in GNU make 4.x is fixed.

Is there any additional information I can provide to allow you to incorporate this fix in GNU make, or to help you find an alternative fix that is more acceptable to you?

On each system I've tested, the script in comment #2 clearly demonstrates the performance regression, even if I don't have any understanding of why it causes a performance regression.

I share your surprise that fork vs vfork could possibly have an important performance impact, but
*it's what git bisect told me
*testing immediately before the commit, I did not encounter the problem
*testing at the tip of master with the commit reverted, I did not encounter the problem

I believe I have created a standalone test which captures what is going on for us.

"mkmk.sh" creates a lot of dummy .c and .h files and also prints a Makefile on its stdout.  Like our software build systems, there are a lot of object files all with a lot of dependencies. (I had originally written some paragraphs describing our build system, but a runnable Makefile has got to be worth a thousand words of description)

With make.git at 4.1-11-ga80a8b8, "make -j10" typically takes about 3 minutes.  With the attached patch applied, "make -j10" typically takes about 11s.  If the only time consumed was by 'sleep .01', at 10x parallelism, it would ideally take 5s.  Fully serialized, it would take 500s.

Increasing NI/NJ (number of sources and number of includes) and the number of repetitions of '@mkdir .' all seem to make the problem worse.

$ mkmk.sh > Makefile
$ rm -f *.o default
$ time make.bad -j12 ;# 4.1-11-ga80a8b8
real        2m58.167s
user        0m2.536s
sys        2m55.375s
$ rm -f *.o default
$ time make.good -j12 ;# patch applied
real        0m10.869s
user        0m2.120s
sys        0m1.564s
$ rm -f *.o default
$ time /usr/bin/make -j10 ;# debian make 3.81
real        0m10.727s
user        0m2.244s
sys        0m1.404s

I agree we're getting into weird territory here.  It could be libc or kernel specific.  As I mentioned initially, we're on Debian 7.8.

Linux parrot 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64 GNU/Linux
GNU C Library (Debian EGLIBC 2.13-38+deb7u8) stable release version 2.13, by Roland McGrath et al.

I also tested on a digitalocean.com virtual server instance with 12 threads and Ubuntu 14.10 (linux 3.16, glibc 2.19) and got 36s vs 6m19 wall time (also at -j10 parallelism)

I also tested on an atlantic.net virtual server instance with 1 thread and Debian Testing (linux 3.18, libc 2.19) and got 1m16 vs 4m52 wall time (also at -j10 parallelism)

So whatever's going on, it affects a range of libc and kernel versions too.  (however, I did just copy the make.good and make.bad binaries, so it could still be something that is specific to the debian 7 build environment)

(file #33381, file #33382)

I maintain the build system for a commercial software product.  Developers build the system on Debian 7

We recently upgraded from 3.81 to 4.1.  Unfortunately, this version of make frequently fails to use the requested level of parallelism, falling back to one job at a time.

Unfortunately, I haven't been able to distill this down to a test case that runs independent of our development environment, so I have no testcase to submit.  But what I know is:

• the project has about 6000 object files
• developer desktops are 6 cores / 12 threads
• we request -j15 -l26 parallelism
• thanks to make parallelism + ccache, a full build is <2minutes
  • but is >10 minutes when this bug bites
• we consistently reproduce the problem by:

    git checkout branch1; make clean; make
    git checkout branch2; make

• it's important not to clean before the second build
• some headers have been removed in branch2 compared to branch1; we have a rule 'include/%.h:; echo $@: missing or removed' so that the build proceeds in this case (we do not use gcc -MP because we found that parsing all those additional lines had a measurable impact on the time of a do-nothing make)
• we use gcc-generated dependency information
• we autogenerate some header files by having the main Makefile depend on those header files
• The problem does not occur if this one commit is reverted:

    94735f0 Remove use of vfork().