bugmake - Bugs: bug #49014, Zombies in parallel builds with...

 
 

bug #49014: Zombies in parallel builds with pselect code

Submitted by:  Jörg Sonnenberger <joerg>
Submitted on:  Tue 06 Sep 2016 12:38:41 PM UTC  
Votes:  100  
 
Severity: 3 - NormalItem Group: Bug
Status: DuplicatePrivacy: Public
Assigned to: NoneOpen/Closed: Closed
Component Version: 4.2.1Operating System: POSIX-Based
Fixed Release: NoneTriage Status: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon 10 Jul 2017 02:01:59 AM UTC, comment #17:

I'm going to close this as resolved by the fix for bug #51159. If this is seen again please add a comment.

Thanks for all the testing and investigation!

Paul D. Smith <psmith>
Project Administrator
Fri 23 Jun 2017 05:37:24 AM UTC, comment #16:

I have built several large software suites (including KDE Frameworks and Plasma Workspaces) since applying this patch, and I have not seen the zombie problem recur.

Matt Whitlock <mwhitlock>
Thu 22 Jun 2017 06:06:57 PM UTC, comment #15:

I've been running with the git head as of your 2017-06-04 request. The problem hasn't recurred for me. I don't think that means as much as it would have a few months ago because I've had less cause to run my "pgid=$(find-pid-of-root-of-big-recursive-make-job); while pid-still-going $pgid; do kill -STOP -$pgid; sleep 1; kill -CONT -$pgid; sleep 1; done" script recently. Still, looks hopeful.

Martin Dorey <mdorey>
Thu 22 Jun 2017 12:14:09 PM UTC, comment #14:

Any updates on this? Have you seen the pselect() zombie issue re-appear after applying the patch?

Thanks for testing!

Paul D. Smith <psmith>
Project Administrator
Mon 05 Jun 2017 06:58:12 AM UTC, comment #13:

I've applied b552b0525198 to Make 4.2.1 on my Gentoo system (and have re-enabled HAVE_PSELECT). I will report back if I experience another deadlock.

Thanks for the (probable) fix!

Matt Whitlock <mwhitlock>
Sun 04 Jun 2017 11:12:55 PM UTC, comment #12:

A fix for bug #51159 has been pushed to the Git repository.

I have a strong suspicion that this is the same problem we're seeing here and that fix will also fix this issue.

Knowing the problem I thought I might be able to conjure a scenario to force a repro case for this bug on my system, and indeed I could easily construct scenarios where the read token was stolen, but I failed to reproduce this bug (make always made progress eventually and completed; there were no outstanding zombies) so I can't be sure.

However, if people who do see this can apply the change in Git SHA b552b0525198 to their versions and retry with pselect() enabled I would appreciate it.

Paul D. Smith <psmith>
Project Administrator
Mon 08 May 2017 03:47:49 PM UTC, comment #11:

SIGCHLD doesn't always unstick the make process, though. It worked for me when I had a make with two zombie children, but it didn't work on another make that had only one zombie child.

I think I'll try comment #0's suggestion of overriding the HAVE_PSELECT autoconf test so I can go back to the tried and true job control code.

Matt Whitlock <mwhitlock>
Mon 08 May 2017 05:52:53 AM UTC, comment #10:

Matt's work around worked for my latest recurrence. I wonder if it gives us a clue as to the cause? I hadn't thought to try it and it's helpful not to have to kill the build, so thanks.

Martin Dorey <mdorey>
Sat 06 May 2017 06:54:21 PM UTC, comment #9:

This issue also affects Make 4.2.1 on Linux 4.11.0-gentoo. I had a parallel build of Firefox stall in a make process with two unreaped children. Manually sending a SIGCHLD to the zombies' parent (make) process got everything moving again.

Matt Whitlock <mwhitlock>
Wed 08 Feb 2017 12:49:46 AM UTC, comment #8:

> Also I suppose Linux 3.2 is pretty old at this point


I'm now on:

Linux swiftboat 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux

... and still seeing the same behavior as I reported in comment #3, several times now. That too is no spring chicken, it's true. I wonder if it would be worth my while trying to reproduce it on something with fewer feet in the grave:

Linux balance 4.9.0-1-amd64 #1 SMP Debian 4.9.2-2 (2017-01-12) x86_64 GNU/Linux

Martin Dorey <mdorey>
Wed 14 Sep 2016 02:32:41 PM UTC, comment #7:

It's a red herring. FreeBSD has been providing a system call since at least version 8.1 and 9.0. A correct userland implementation can be done on top of other primitives like ppoll or kqueue, but that's not relevant here.

The common element for all the reports so far is that the file descriptor for the jobs pipe got closed and something else is reusing the FD. This is much more noticable when disabling the pselect use it seems.

Jörg Sonnenberger <joerg>
Wed 14 Sep 2016 01:55:29 PM UTC, comment #6:

It can't be implemented in userland. What pselect() does is unblock the signal, call select, then have the signal blocked again on return. The point of using pselect() is that the signal unblock/block must be atomic with the select() system call. Otherwise between the instant you unblock the signal and invoke select(), the signal could arrive and you miss it. Ditto for after select() returns and before the signal is blocked again. There's no way to avoid that from userland. The Linux man page for pselect has a good explanation, as does this LWN article: https://lwn.net/Articles/176911/

I looked at the NetBSD man page but it didn't say clearly one way or the other whether it used a system call or not. If you've checked and verified it uses a system call that's good.

I have to say I still don't see what the bug could be. Stripping out all the ifdefs etc. the handling of SIGCHLD is very simple when HAVE_PSELECT is set so it's hard to know what might be wrong. I think we may have to involve the NetBSD devs to talk about how pselect() works there.

Paul D. Smith <psmith>
Project Administrator
Tue 13 Sep 2016 04:08:41 PM UTC, comment #5:

(1) I'm pretty sure that pselect can be implemented correctly in userland.
(2) It doesn't matter as the original bug report was from NetBSD, where pselect is a system call.

Jörg Sonnenberger <joerg>
Tue 13 Sep 2016 02:33:23 PM UTC, comment #4:

I took a look through the code. I could to do a more comprehensive review but at first glance it looks like everything is OK: if PSELECT is enabled then we block SIGCHLD right at the start of the make process and we never unblock it except as a parameter to pselect().

However, I think that pselect() in the *BSD systems is not POSIX-compliant: according to the FreeBSD 5.0 man pages:

The pselect() function is implemented in the C library as a wrapper around select().

It is not possible to correctly implement pselect() in the C library using select(); it MUST have kernel system call support. Otherwise there is a very real race condition that cannot be closed. I'm not sure that there's any way to detect pselect() without kernel syscall support from autoconf, so the best I can do (assuming that my diagnosis is correct) would be to detect the target platform and disable pselect() support on BSD targets. If you elect to discuss this issue on any BSD mailing lists please CC me.

Regarding the Linux issue with SIGSTOP and SIGCONT, I can't explain that. I hate to suggest a kernel bug because 99.9% of the time you end up with egg on your face, but I do wonder if there's an issue with SIGSTOP/SIGCONT and pselect() where it's losing a signal somewhere. Also I suppose Linux 3.2 is pretty old at this point; perhaps someone on the kernel lists might have a thought.

Paul D. Smith <psmith>
Project Administrator
Tue 13 Sep 2016 01:50:11 AM UTC, comment #3:

Happened to me again. All the zombies were preceded in the process tree by a make blocked on the read of job_fds[0] in the HAVE_PSELECT jobserver_acquire. I had, once again, sent the whole process group SIGSTOP and SIGCONT at some point.

Martin Dorey <mdorey>
Thu 08 Sep 2016 05:18:44 PM UTC, comment #2:

Without pselect, I semi-reliable get errors like:

gmake[3]: *** duping jobs pipe: Bad file descriptor. Stop.

variants with ENOENT etc exists. According to

https://secure.freshbsd.org/commit/freebsd-ports/r421562

this happens on FreeBSD with pselect as well.

Jörg Sonnenberger <joerg>
Tue 06 Sep 2016 06:38:50 PM UTC, comment #1:

I've had a couple of hung makes in the last couple of months where I've noticed a prominent number of zombies. I see my from-git make from 2016-06-11, reporting itself as 4.2.1, has HAVE_PSELECT=1 in config.status and imports that symbol from glibc. The hang has always been deep in a big build, I think after I've suspended and resumed the build with SIGSTOP and SIGCONT to a large process group. Neither time did I find anything tractable, feared it might be a kernel bug in:

Linux swiftboat 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

Guess I should try strace()ing the daunting number of make processes next time and looking at the signal masking in /proc. Sorry for the lame me-too.

Martin Dorey <mdorey>
Tue 06 Sep 2016 12:38:41 PM UTC, original submission:

When using the new pselect based job server logic, I regularly see hanging pkgsrc bulk builds on NetBSD 7.0, both i386 and amd64. Most common victims are huge packages like libreoffice. The symptoms are gmake processes with exactly one zombie child per concurrent job. Since disable pselect by overriding the configure test, the problem has disappeared.

Jörg Sonnenberger <joerg>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mwhitlock (Posted a comment)
  • -unavailable- added by mwhitlock
  • -unavailable- added by psmith (Posted a comment)
  • -unavailable- added by mdorey (Posted a comment)
  • -unavailable- added by joerg (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 100 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 10 Jul 2017 02:01:59 AM UTCpsmithStatusNone=>Duplicate
      Open/ClosedOpen=>Closed
    Sat 06 May 2017 06:54:21 PM UTCmwhitlockCarbon-Copy-=>Added mwhitlock

    Back to the top


    Powered by Savane 3.1-cleanup1