bugmake - Bugs: bug #49014, Zombies in parallel builds with...

 
 

bug #49014: Zombies in parallel builds with pselect code

Submitter:  Jörg Sonnenberger <joerg>
Submitted:  Tue 06 Sep 2016 12:38:41 PM UTC
Votes: 100
 
Severity:  3 - Normal Item Group:  Bug
Status:  Duplicate Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Component Version:  4.2.1 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 09 Sep 2019 01:37:16 PM UTC, comment #24: 

Thank you for looking into this and showing the committed change. I'll update our port to include this.

Kurt Miller <kurtmiller>
Sat 07 Sep 2019 11:45:26 PM UTC, comment #23: 

I'm re-closing this :).

The problem Kurt reports is a duplicate of bug #51400.

It was fixed with http://git.savannah.gnu.org/cgit/make.git/commit/?id=78b5fec6898c26956d00548427cda1101cb80f8a

Paul D. Smith <psmith>
Group administrator
Fri 06 Sep 2019 11:22:36 PM UTC, comment #22: 

I'll reopen this so I don't lose track of it.

For the next release I wanted to completely rewrite the signal handling in GNU make, because it's utterly broken (see various still-open bugs about easy ways to get make to hang on ^C etc.)  The code makes the cardinal sin of doing a lot of crazy stuff inside a signal handler which only leads to tears.

I have an implementation that reduces the signal handler to just set a flag, which is then checked at strategic places to determine whether a fatal signal was received, then introduces a "critical section" which can be declared to ensure that these lists are not mismanaged.  This would alleviate your issue here since we'd never need to change the signal handlers once they were set up when the program started.

Unfortunately, it's not all completely working yet.

I'll either install a fix like this (maybe not quite exactly this) or get my full solution working before the next release.

Paul D. Smith <psmith>
Group administrator
Fri 06 Sep 2019 11:05:04 PM UTC, comment #21: 

Thank you for the replies. Yes, the referenced commit (b552b0525198) was tried:

https://github.com/openbsd/ports/commit/29ba9c2b7a4577f5aa1d85b6c9bf4696423886d8#diff-b2f76d26babda853136a6c3517055ebb

but the pselect/zombie issue continued to show up in our ports bulk builds.

While the problem may not be tripping on other platforms, the fact that SIGCHLD does not remain blocked should be visible via strace(1). Since applying the submitted patch our pselect/zombie issue has not resurfaced.

Kurt Miller <kurtmiller>
Fri 06 Sep 2019 10:22:35 PM UTC, comment #20: 

Surely Kurt was running with that?  Mind, he didn't say.  I haven't seen my Linux problem in the 2+ years since applying b552b0525198.

Martin Dorey <mdorey>
Fri 06 Sep 2019 10:16:32 PM UTC, comment #19: 

Please check the referenced commit. We've been running with it for over a year in pkgsrc now and the problems disappeared.

Jörg Sonnenberger <joerg>
Fri 06 Sep 2019 09:55:19 PM UTC, comment #18: 

We continue to see gmake 4.2.1 get stuck with zombie's on OpenBSD with pselect(2).

comment #4:

> I took a look through the code.  I could to do a more comprehensive review but at first glance it looks like everything is OK: if PSELECT is enabled then we block SIGCHLD right at the start of the make process and we never unblock it except as a parameter to pselect().


While SIGCHLD is blocked at the start of the make process, it gets unblocked in various places which breaks the invariant that it must remain blocked for pselect(2) to atomically receive all SIGCHLD events. This was determined by using strace(1) on gmake on OpenBSD.

I have a proposed patch to keep SIGCHLD blocked and will attach to the bug report.

Kurt Miller <kurtmiller>
Mon 10 Jul 2017 02:01:59 AM UTC, comment #17: 

I'm going to close this as resolved by the fix for bug #51159.  If this is seen again please add a comment.

Thanks for all the testing and investigation!

Paul D. Smith <psmith>
Group administrator
Fri 23 Jun 2017 05:37:24 AM UTC, comment #16: 

I have built several large software suites (including KDE Frameworks and Plasma Workspaces) since applying this patch, and I have not seen the zombie problem recur.

Matt Whitlock <mwhitlock>
Thu 22 Jun 2017 06:06:57 PM UTC, comment #15: 

I've been running with the git head as of your 2017-06-04 request.  The problem hasn't recurred for me.  I don't think that means as much as it would have a few months ago because I've had less cause to run my "pgid=$(find-pid-of-root-of-big-recursive-make-job); while pid-still-going $pgid; do kill -STOP -$pgid; sleep 1; kill -CONT -$pgid; sleep 1; done" script recently.  Still, looks hopeful.

Martin Dorey <mdorey>
Thu 22 Jun 2017 12:14:09 PM UTC, comment #14: 

Any updates on this?  Have you seen the pselect() zombie issue re-appear after applying the patch?

Thanks for testing!

Paul D. Smith <psmith>
Group administrator
Mon 05 Jun 2017 06:58:12 AM UTC, comment #13: 

I've applied b552b0525198 to Make 4.2.1 on my Gentoo system (and have re-enabled HAVE_PSELECT). I will report back if I experience another deadlock.

Thanks for the (probable) fix!

Matt Whitlock <mwhitlock>
Sun 04 Jun 2017 11:12:55 PM UTC, comment #12: 

A fix for bug #51159 has been pushed to the Git repository.

I have a strong suspicion that this is the same problem we're seeing here and that fix will also fix this issue.

Knowing the problem I thought I might be able to conjure a scenario to force a repro case for this bug on my system, and indeed I could easily construct scenarios where the read token was stolen, but I failed to reproduce this bug (make always made progress eventually and completed; there were no outstanding zombies) so I can't be sure.

However, if people who do see this can apply the change in Git SHA b552b0525198 to their versions and retry with pselect() enabled I would appreciate it.

Paul D. Smith <psmith>
Group administrator
Mon 08 May 2017 03:47:49 PM UTC, comment #11: 

SIGCHLD doesn't always unstick the make process, though. It worked for me when I had a make with two zombie children, but it didn't work on another make that had only one zombie child.

I think I'll try comment #0's suggestion of overriding the HAVE_PSELECT autoconf test so I can go back to the tried and true job control code.

Matt Whitlock <mwhitlock>
Mon 08 May 2017 05:52:53 AM UTC, comment #10: 

Matt's work around worked for my latest recurrence.  I wonder if it gives us a clue as to the cause?  I hadn't thought to try it and it's helpful not to have to kill the build, so thanks.

Martin Dorey <mdorey>
Sat 06 May 2017 06:54:21 PM UTC, comment #9: 

This issue also affects Make 4.2.1 on Linux 4.11.0-gentoo. I had a parallel build of Firefox stall in a make process with two unreaped children. Manually sending a SIGCHLD to the zombies' parent (make) process got everything moving again.

Matt Whitlock <mwhitlock>
Wed 08 Feb 2017 12:49:46 AM UTC, comment #8: 


> Also I suppose Linux 3.2 is pretty old at this point


I'm now on:

Linux swiftboat 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux

... and still seeing the same behavior as I reported in comment #3, several times now.  That too is no spring chicken, it's true.  I wonder if it would be worth my while trying to reproduce it on something with fewer feet in the grave:

Linux balance 4.9.0-1-amd64 #1 SMP Debian 4.9.2-2 (2017-01-12) x86_64 GNU/Linux

Martin Dorey <mdorey>
Wed 14 Sep 2016 02:32:41 PM UTC, comment #7: 

It's a red herring. FreeBSD has been providing a system call since at least version 8.1 and 9.0. A correct userland implementation can be done on top of other primitives like ppoll or kqueue, but that's not relevant here.

The common element for all the reports so far is that the file descriptor for the jobs pipe got closed and something else is reusing the FD. This is much more noticable when disabling the pselect use it seems.

Jörg Sonnenberger <joerg>
Wed 14 Sep 2016 01:55:29 PM UTC, comment #6: 

It can't be implemented in userland.  What pselect() does is unblock the signal, call select, then have the signal blocked again on return.  The point of using pselect() is that the signal unblock/block must be atomic with the select() system call.  Otherwise between the instant you unblock the signal and invoke select(), the signal could arrive and you miss it.  Ditto for after select() returns and before the signal is blocked again.  There's no way to avoid that from userland.  The Linux man page for pselect has a good explanation, as does this LWN article: https://lwn.net/Articles/176911/

I looked at the NetBSD man page but it didn't say clearly one way or the other whether it used a system call or not.  If you've checked and verified it uses a system call that's good.

I have to say I still don't see what the bug could be.  Stripping out all the ifdefs etc. the handling of SIGCHLD is very simple when HAVE_PSELECT is set so it's hard to know what might be wrong.  I think we may have to involve the NetBSD devs to talk about how pselect() works there.

Paul D. Smith <psmith>
Group administrator
Tue 13 Sep 2016 04:08:41 PM UTC, comment #5: 

(1) I'm pretty sure that pselect can be implemented correctly in userland.
(2) It doesn't matter as the original bug report was from NetBSD, where pselect is a system call.

Jörg Sonnenberger <joerg>
Tue 13 Sep 2016 02:33:23 PM UTC, comment #4: 

I took a look through the code.  I could to do a more comprehensive review but at first glance it looks like everything is OK: if PSELECT is enabled then we block SIGCHLD right at the start of the make process and we never unblock it except as a parameter to pselect().

However, I think that pselect() in the *BSD systems is not POSIX-compliant: according to the FreeBSD 5.0 man pages:

The pselect() function is implemented in the C library as a wrapper around select().

It is not possible to correctly implement pselect() in the C library using select(); it MUST have kernel system call support.  Otherwise there is a very real race condition that cannot be closed.  I'm not sure that there's any way to detect pselect() without kernel syscall support from autoconf, so the best I can do (assuming that my diagnosis is correct) would be to detect the target platform and disable pselect() support on BSD targets.  If you elect to discuss this issue on any BSD mailing lists please CC me.

Regarding the Linux issue with SIGSTOP and SIGCONT, I can't explain that.  I hate to suggest a kernel bug because 99.9% of the time you end up with egg on your face, but I do wonder if there's an issue with SIGSTOP/SIGCONT and pselect() where it's losing a signal somewhere.  Also I suppose Linux 3.2 is pretty old at this point; perhaps someone on the kernel lists might have a thought.

Paul D. Smith <psmith>
Group administrator
Tue 13 Sep 2016 01:50:11 AM UTC, comment #3: 

Happened to me again.  All the zombies were preceded in the process tree by a make blocked on the read of job_fds[0] in the HAVE_PSELECT jobserver_acquire.  I had, once again, sent the whole process group SIGSTOP and SIGCONT at some point.

Martin Dorey <mdorey>
Thu 08 Sep 2016 05:18:44 PM UTC, comment #2: 

Without pselect, I semi-reliable get errors like:

gmake[3]: * duping jobs pipe: Bad file descriptor.  Stop.

variants with ENOENT etc exists. According to

https://secure.freshbsd.org/commit/freebsd-ports/r421562

this happens on FreeBSD with pselect as well.

Jörg Sonnenberger <joerg>
Tue 06 Sep 2016 06:38:50 PM UTC, comment #1: 

I've had a couple of hung makes in the last couple of months where I've noticed a prominent number of zombies.  I see my from-git make from 2016-06-11, reporting itself as 4.2.1, has HAVE_PSELECT=1 in config.status and imports that symbol from glibc.  The hang has always been deep in a big build, I think after I've suspended and resumed the build with SIGSTOP and SIGCONT to a large process group.  Neither time did I find anything tractable, feared it might be a kernel bug in:

Linux swiftboat 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

Guess I should try strace()ing the daunting number of make processes next time and looking at the signal masking in /proc.  Sorry for the lame me-too.

Martin Dorey <mdorey>
Tue 06 Sep 2016 12:38:41 PM UTC, original submission:  

When using the new pselect based job server logic, I regularly see hanging pkgsrc bulk builds on NetBSD 7.0, both i386 and amd64. Most common victims are huge packages like libreoffice. The symptoms are gmake processes with exactly one zombie child per concurrent job. Since disable pselect by overriding the configure test, the problem has disappeared.

Jörg Sonnenberger <joerg>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #47461:  gmake.pselect.diff added by kurtmiller (2KiB - text/x-patch - Keep SIGCHLD blocked when using pselect(2))

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by kurtmiller (Posted a comment)
  • -email is unavailable- added by mwhitlock (Posted a comment)
  • -email is unavailable- added by mwhitlock
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by mdorey (Posted a comment)
  • -email is unavailable- added by joerg (Submitted the item)
  •  

    There are 100 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-09-07 psmith Open/ClosedOpen Closed
    2019-09-06 psmith Open/ClosedClosed Open
    2019-09-06 kurtmiller Attached File- Added gmake.pselect.diff, #47461
    2017-07-10 psmith StatusNone Duplicate
        Open/ClosedOpen Closed
    2017-05-06 mwhitlock Carbon-Copy- Added mwhitlock

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code