Thank you for looking into this and showing the committed change. I'll update our port to include this.

I'll reopen this so I don't lose track of it.

For the next release I wanted to completely rewrite the signal handling in GNU make, because it's utterly broken (see various still-open bugs about easy ways to get make to hang on ^C etc.)  The code makes the cardinal sin of doing a lot of crazy stuff inside a signal handler which only leads to tears.

I have an implementation that reduces the signal handler to just set a flag, which is then checked at strategic places to determine whether a fatal signal was received, then introduces a "critical section" which can be declared to ensure that these lists are not mismanaged.  This would alleviate your issue here since we'd never need to change the signal handlers once they were set up when the program started.

Unfortunately, it's not all completely working yet.

I'll either install a fix like this (maybe not quite exactly this) or get my full solution working before the next release.

Surely Kurt was running with that?  Mind, he didn't say.  I haven't seen my Linux problem in the 2+ years since applying b552b0525198.

We continue to see gmake 4.2.1 get stuck with zombie's on OpenBSD with pselect(2).

comment #4:

> I took a look through the code.  I could to do a more comprehensive review but at first glance it looks like everything is OK: if PSELECT is enabled then we block SIGCHLD right at the start of the make process and we never unblock it except as a parameter to pselect().

While SIGCHLD is blocked at the start of the make process, it gets unblocked in various places which breaks the invariant that it must remain blocked for pselect(2) to atomically receive all SIGCHLD events. This was determined by using strace(1) on gmake on OpenBSD.

I have a proposed patch to keep SIGCHLD blocked and will attach to the bug report.

I have built several large software suites (including KDE Frameworks and Plasma Workspaces) since applying this patch, and I have not seen the zombie problem recur.

Any updates on this?  Have you seen the pselect() zombie issue re-appear after applying the patch?

Thanks for testing!

A fix for bug #51159 has been pushed to the Git repository.

I have a strong suspicion that this is the same problem we're seeing here and that fix will also fix this issue.

Knowing the problem I thought I might be able to conjure a scenario to force a repro case for this bug on my system, and indeed I could easily construct scenarios where the read token was stolen, but I failed to reproduce this bug (make always made progress eventually and completed; there were no outstanding zombies) so I can't be sure.

However, if people who do see this can apply the change in Git SHA b552b0525198 to their versions and retry with pselect() enabled I would appreciate it.

Matt's work around worked for my latest recurrence.  I wonder if it gives us a clue as to the cause?  I hadn't thought to try it and it's helpful not to have to kill the build, so thanks.

> Also I suppose Linux 3.2 is pretty old at this point

I'm now on:

Linux swiftboat 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux

... and still seeing the same behavior as I reported in comment #3, several times now.  That too is no spring chicken, it's true.  I wonder if it would be worth my while trying to reproduce it on something with fewer feet in the grave:

Linux balance 4.9.0-1-amd64 #1 SMP Debian 4.9.2-2 (2017-01-12) x86_64 GNU/Linux

It can't be implemented in userland.  What pselect() does is unblock the signal, call select, then have the signal blocked again on return.  The point of using pselect() is that the signal unblock/block must be atomic with the select() system call.  Otherwise between the instant you unblock the signal and invoke select(), the signal could arrive and you miss it.  Ditto for after select() returns and before the signal is blocked again.  There's no way to avoid that from userland.  The Linux man page for pselect has a good explanation, as does this LWN article: https://lwn.net/Articles/176911/

I looked at the NetBSD man page but it didn't say clearly one way or the other whether it used a system call or not.  If you've checked and verified it uses a system call that's good.

I have to say I still don't see what the bug could be.  Stripping out all the ifdefs etc. the handling of SIGCHLD is very simple when HAVE_PSELECT is set so it's hard to know what might be wrong.  I think we may have to involve the NetBSD devs to talk about how pselect() works there.

I took a look through the code.  I could to do a more comprehensive review but at first glance it looks like everything is OK: if PSELECT is enabled then we block SIGCHLD right at the start of the make process and we never unblock it except as a parameter to pselect().

However, I think that pselect() in the *BSD systems is not POSIX-compliant: according to the FreeBSD 5.0 man pages:

The pselect() function is implemented in the C library as a wrapper around select().

It is not possible to correctly implement pselect() in the C library using select(); it MUST have kernel system call support.  Otherwise there is a very real race condition that cannot be closed.  I'm not sure that there's any way to detect pselect() without kernel syscall support from autoconf, so the best I can do (assuming that my diagnosis is correct) would be to detect the target platform and disable pselect() support on BSD targets.  If you elect to discuss this issue on any BSD mailing lists please CC me.

Regarding the Linux issue with SIGSTOP and SIGCONT, I can't explain that.  I hate to suggest a kernel bug because 99.9% of the time you end up with egg on your face, but I do wonder if there's an issue with SIGSTOP/SIGCONT and pselect() where it's losing a signal somewhere.  Also I suppose Linux 3.2 is pretty old at this point; perhaps someone on the kernel lists might have a thought.

Without pselect, I semi-reliable get errors like:

gmake[3]: * duping jobs pipe: Bad file descriptor.  Stop.

variants with ENOENT etc exists. According to

https://secure.freshbsd.org/commit/freebsd-ports/r421562

this happens on FreeBSD with pselect as well.

When using the new pselect based job server logic, I regularly see hanging pkgsrc bulk builds on NetBSD 7.0, both i386 and amd64. Most common victims are huge packages like libreoffice. The symptoms are gmake processes with exactly one zombie child per concurrent job. Since disable pselect by overriding the configure test, the problem has disappeared.