bugmake - Bugs: bug #49014, Zombies in parallel builds with...

 
 

bug #49014: Zombies in parallel builds with pselect code

Submitted by:  Jörg Sonnenberger <joerg>
Submitted on:  Tue Sep 6 12:38:41 2016  
Votes:  100  
 
Severity: 3 - NormalItem Group: Bug
Status: NonePrivacy: Public
Assigned to: NoneOpen/Closed: Open
Component Version: 4.2.1Operating System: POSIX-Based
Fixed Release: NoneTriage Status: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Mon May 8 15:47:49 2017, comment #11:

SIGCHLD doesn't always unstick the make process, though. It worked for me when I had a make with two zombie children, but it didn't work on another make that had only one zombie child.

I think I'll try comment #0's suggestion of overriding the HAVE_PSELECT autoconf test so I can go back to the tried and true job control code.

Matt Whitlock <mwhitlock>
Mon May 8 05:52:53 2017, comment #10:

Matt's work around worked for my latest recurrence. I wonder if it gives us a clue as to the cause? I hadn't thought to try it and it's helpful not to have to kill the build, so thanks.

Martin Dorey <mdorey>
Sat May 6 18:54:21 2017, comment #9:

This issue also affects Make 4.2.1 on Linux 4.11.0-gentoo. I had a parallel build of Firefox stall in a make process with two unreaped children. Manually sending a SIGCHLD to the zombies' parent (make) process got everything moving again.

Matt Whitlock <mwhitlock>
Wed Feb 8 00:49:46 2017, comment #8:

> Also I suppose Linux 3.2 is pretty old at this point


I'm now on:

Linux swiftboat 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux

... and still seeing the same behavior as I reported in comment #3, several times now. That too is no spring chicken, it's true. I wonder if it would be worth my while trying to reproduce it on something with fewer feet in the grave:

Linux balance 4.9.0-1-amd64 #1 SMP Debian 4.9.2-2 (2017-01-12) x86_64 GNU/Linux

Martin Dorey <mdorey>
Wed Sep 14 14:32:41 2016, comment #7:

It's a red herring. FreeBSD has been providing a system call since at least version 8.1 and 9.0. A correct userland implementation can be done on top of other primitives like ppoll or kqueue, but that's not relevant here.

The common element for all the reports so far is that the file descriptor for the jobs pipe got closed and something else is reusing the FD. This is much more noticable when disabling the pselect use it seems.

Jörg Sonnenberger <joerg>
Wed Sep 14 13:55:29 2016, comment #6:

It can't be implemented in userland. What pselect() does is unblock the signal, call select, then have the signal blocked again on return. The point of using pselect() is that the signal unblock/block must be atomic with the select() system call. Otherwise between the instant you unblock the signal and invoke select(), the signal could arrive and you miss it. Ditto for after select() returns and before the signal is blocked again. There's no way to avoid that from userland. The Linux man page for pselect has a good explanation, as does this LWN article: https://lwn.net/Articles/176911/

I looked at the NetBSD man page but it didn't say clearly one way or the other whether it used a system call or not. If you've checked and verified it uses a system call that's good.

I have to say I still don't see what the bug could be. Stripping out all the ifdefs etc. the handling of SIGCHLD is very simple when HAVE_PSELECT is set so it's hard to know what might be wrong. I think we may have to involve the NetBSD devs to talk about how pselect() works there.

Paul D. Smith <psmith>
Project Administrator
Tue Sep 13 16:08:41 2016, comment #5:

(1) I'm pretty sure that pselect can be implemented correctly in userland.
(2) It doesn't matter as the original bug report was from NetBSD, where pselect is a system call.

Jörg Sonnenberger <joerg>
Tue Sep 13 14:33:23 2016, comment #4:

I took a look through the code. I could to do a more comprehensive review but at first glance it looks like everything is OK: if PSELECT is enabled then we block SIGCHLD right at the start of the make process and we never unblock it except as a parameter to pselect().

However, I think that pselect() in the *BSD systems is not POSIX-compliant: according to the FreeBSD 5.0 man pages:

The pselect() function is implemented in the C library as a wrapper around select().

It is not possible to correctly implement pselect() in the C library using select(); it MUST have kernel system call support. Otherwise there is a very real race condition that cannot be closed. I'm not sure that there's any way to detect pselect() without kernel syscall support from autoconf, so the best I can do (assuming that my diagnosis is correct) would be to detect the target platform and disable pselect() support on BSD targets. If you elect to discuss this issue on any BSD mailing lists please CC me.

Regarding the Linux issue with SIGSTOP and SIGCONT, I can't explain that. I hate to suggest a kernel bug because 99.9% of the time you end up with egg on your face, but I do wonder if there's an issue with SIGSTOP/SIGCONT and pselect() where it's losing a signal somewhere. Also I suppose Linux 3.2 is pretty old at this point; perhaps someone on the kernel lists might have a thought.

Paul D. Smith <psmith>
Project Administrator
Tue Sep 13 01:50:11 2016, comment #3:

Happened to me again. All the zombies were preceded in the process tree by a make blocked on the read of job_fds[0] in the HAVE_PSELECT jobserver_acquire. I had, once again, sent the whole process group SIGSTOP and SIGCONT at some point.

Martin Dorey <mdorey>
Thu Sep 8 17:18:44 2016, comment #2:

Without pselect, I semi-reliable get errors like:

gmake[3]: *** duping jobs pipe: Bad file descriptor. Stop.

variants with ENOENT etc exists. According to

https://secure.freshbsd.org/commit/freebsd-ports/r421562

this happens on FreeBSD with pselect as well.

Jörg Sonnenberger <joerg>
Tue Sep 6 18:38:50 2016, comment #1:

I've had a couple of hung makes in the last couple of months where I've noticed a prominent number of zombies. I see my from-git make from 2016-06-11, reporting itself as 4.2.1, has HAVE_PSELECT=1 in config.status and imports that symbol from glibc. The hang has always been deep in a big build, I think after I've suspended and resumed the build with SIGSTOP and SIGCONT to a large process group. Neither time did I find anything tractable, feared it might be a kernel bug in:

Linux swiftboat 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

Guess I should try strace()ing the daunting number of make processes next time and looking at the signal masking in /proc. Sorry for the lame me-too.

Martin Dorey <mdorey>
Tue Sep 6 12:38:41 2016, original submission:

When using the new pselect based job server logic, I regularly see hanging pkgsrc bulk builds on NetBSD 7.0, both i386 and amd64. Most common victims are huge packages like libreoffice. The symptoms are gmake processes with exactly one zombie child per concurrent job. Since disable pselect by overriding the configure test, the problem has disappeared.

Jörg Sonnenberger <joerg>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mwhitlock (Posted a comment)
  • -unavailable- added by mwhitlock
  • -unavailable- added by psmith (Posted a comment)
  • -unavailable- added by mdorey (Posted a comment)
  • -unavailable- added by joerg (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 100 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat May 6 18:54:21 2017mwhitlockCarbon-Copy-=>Added mwhitlock

    Back to the top


    Powered by Savane 3.1-cleanup1