I think this is not an enhancement request but rather a bug-fix request.

Regarding efficiency:

First, the main point of my report is not "alloca sucks" but rather
that setrlimit is an unexpected thing for make to do.

But having opened up the can of worms, let me play with them a little
(I don't really want to start a big argument, but I do want to defend
my previous statements):

In my experience, the dominant effect on program performance, after
choice of algorithm, is data locality: frequently re-using the same or
nearby memory locations.  Instruction count is at best a second-order
effect.  Andrew Appel has a lovely paper where he "proves" that (of
all things) garbage collection is faster than stack allocation, by
exclusively focusing on instruction count:

  http://citeseer.ist.psu.edu/appel87garbage.html
 
Of course, the analysis is fatally flawed by ignoring locality.

My point regarding alloca is that if I have a lot of data, it cannot
all be "hot" (high locality).  If I put cold data on the stack, the
stack as a whole has poorer locality, because the hot regions (stack
frames) are interspersed with cold regions (the big chunks of data).
So if I intend to allocate infrequently-accessed data, it is better to
put it on the heap where it won't interfere with the locality of stack
accesses.

You say that only "medium" sized data ends up on the stack, which
certainly sounds like a reasonable approach, but if there is enough of
it to warrant calling setrlimit, then I have to suspect that some of
it would be better placed elsewhere, from a strictly
performance-oriented point of view.

Regarding portability:

Indeed, there is no portable way to detect running out of stack space,
whether by alloca or not, though alloca obviously makes running out of
stack space more likely.  Of course, setrlimit is not all that
portable either, so I was assuming that nonportable solutions were on
the table.  Toward that end, an appropriate signal handler (which will
need to use nonportable tricks to distinguish stack overflow from
other types of segfaults), or the GCC -fstack-check argument (when
compiling with GCC, which is after all the common case) may be options.

Moreover, I would argue that the architectural decision to make heavy
reliance on alloca is what leads us into nonportable territory in the
first place.  It is not uncommon to find systems with ~10MB or less of
stack space.  The setrlimit call is a band-aid; in some circumstances
it will help, but that call is not an antidote to running out of stack
space.

Of course, the big advantage of alloca is that it is easy to program
with.  I would not advocate suddenly rewriting GNU make to use malloc
everywhere, since (as you say) that will likely introduce many more
serious bugs than this one in the short term.  (If it was written in
C++ there would be better options, but that ship too has sailed.)

So, at most, I would suggest doing some profiling to find the biggest
consumers of alloca space and change just them (hopefully only a
handful) to use [x]malloc instead.  The primary motivation would be
enhanced portability; the efficiency argument is just meant to allay
fears that this will make everything slower.

Regarding setrlimit itself:

Yes, I would consider this bug (and I do consider it a bug...) to be
fixed if 'make' would reliably return the resource limits to their
original settings before exec'ing its children.

> If 'make' needs to allocate a large amount (megabytes) of data,
> it would be better to do so on the heap, both from a
> portability standpoint (the stack size) and from a performance
> standpoint (it messes up the normally good locality of stack
> access).

> Alternatively, if it must allocate on the stack, then detecting
> and complaining about a too-low limit would be better in my
> opinion than silently changing it. It's easy to uncap the stack
> size explicitly in build scripts and whatnot when truly
> necessary.

Unfortunately, none of the above is true.  Make needs extra stack space because it makes extensive use of the alloca() function.  It does not allocate huge chunks of memory on the stack: if large amounts of memory are needed they are allocated on the stack.  alloca() is used for modest-sized memory chunks, to hold filenames and smaller strings (make is, at heart, a string manipulation program).  Using heap, which requires a system call to get more memory, for all of these small allocations would be much less efficient than using the stack.  Not to mention the overhead of having to allocate these memory segments to be used for a short time, then freeing them again, over and over.

However, because make is also very recursive, even though no single alloca() call is very large it's quite possible for the entirety of the alloca() invocations for a complex makefile to use quite a bit of stack.

Of course, there's no way to determine how much stack will be used a priori, since it depends entirely on the construction of the makefile, exactly which functions are invoked in which order, and even the current state of the build (whether various targets need to be rebuilt or not).  Further, there is no portable way that I'm aware of to determine how much stack is left before the program runs out.

Finally, there is no way to detect an out of stack error and exit gracefully with a warning as you suggest: the behavior of alloca() is undefined if you run out of stack space (it doesn't just return NULL as malloc() etc. do).

So.  In order to avoid the need for extra stack in GNU make all the invocations of alloca() would need to be rewritten to use xmalloc(), and those functions would need to be changed to be careful to free all that memory whenever the function exited to avoid memory leaks... with what must certainly be a decrease in performance (although of how much we can't be sure).

On the other hand, as you point out, the problem on Solaris is a bug in the kernel which you can hardly blame GNU make for.

However, your point about programs invoked by make inheriting the setrlimit() is definitely something that seems problematic.  Perhaps GNU make could change the stack limit back to what it was after it forks but before it execs its child.  I wonder what happens if you change a limit to something too small for the current processes resources?