I applied patches to master and screen-v4 branches. Everyone are welcome to test it.

Thanks a lot.

comment #24 :

> It doesn't seem to depend on the OS, so far it seems to depend on two things:
>
> a) using libutempter (a compile time decision by the configure script), and
> b) using a slow computer or a VM.

This is actually more complex. To reproduce the issue, the SIGCHLD needs to be received while the action has been set to SIG_DFL by libutempter, basically while the libutempter helper is running. I had noticed that reducing or disabling the compiler optimizations for screen made the zombies less likely to appear on my VM. And this probably depends very much on the hardware (e.g., on my laptop, I recently noticed that a race condition in the display manager triggered an issue only after I changed the SSD disk, just because the new SSD disk is faster).

> The exact part which "needs" to be slow on a host to trigger the race condition or why it shows up especially on VMs is though unclear to me. I was so far unable to reproduce this on neither a Xen VM nor on a KVM (ProxMox) based VM nor on real hardware.

My VM (at Gandi) is based on Xen. And the issue is 100% reproducible.

I've just tried to reproduce the issue on my laptop by adding a "sleep(1);" after the SIGCHLD action has been set to SIG_DFL in libutempter (iface.c), but I couldn't. I suppose that the SIGCHLD arrives earlier. I don't know which stress-ng options could be used to have it delayed ("--all 1" completely freezes my machine).

comment #9:

> That would be a lot easier if I could reproduce it...
>

Maybe I'm wrong, but I think Amadeusz uses gentoo.
Anyway...

Can someone reproduce this issue on NOT_Debian system?
Because I also CAN NOT reproduce this problem (tested on openSUSE and OpenBSD).

Axel: This actually started with this bug report by a Mandriva user in 2008 (no-one else confirmed at that time). Then this became silent until you mentioned the bug in October 2019 (thus 11 years later) for Debian 10 Buster. The 11-year gap after a single report is a lot. Many things could have happened and the code has probably much changed. If this 2008 issue had a different cause, this very old bug may already have been fixed (no reports for 11 years) and we are seeing something different as of Debian 10.

Hi Vincent,

comment #14:

> I've found a clash between GNU Screen and libutempter (used in utmp.c): libutempter needs to execute some helper, and temporarily resets the SIGCHLD handler to the default action. If the signal from the terminated shell process arrives at this time, the handler set up by GNU Screen will not be called.

I doubt that this will solve the case (completely) either because this issue already happened in Debian 9 Stretch before I enabled libutempter support in Debian in Debian 10 Buster.

At some point in the past (when I had to deal with the CVE-2021-26937 fix in early 2021, i.e. before your Debian bug report in summer 2021) I set screen zombies to be ignored in our monitoring system. I will now reenable this monitoring check and start tracking this issue again, also by trying out your patch on that one machine where me and colleagues run into it, once we ran into it again.

Really attaching the patch: file #51740

Note also that, as I've said in my Debian bug report, GotSigChld (which is modified in a signal handler) needs to be declared as volatile in order to disable some compiler optimizations (currently not affecting the code, but this might happen in the future).

I'm wondering how to fix the real issue. One knows that a SIGCHLD may have been lost by some libutempter functions called in pututslot() (defined in utmp.c). Thus pututslot() could call DoWait() directly or set GotSigChld, if this is valid. To be able to do that, one would also need to export DoWait() or GotSigChld. Alternatively, send a SIGCHLD to oneself.

Yay! Thanks for fixing this decade old bug.

I can't find a reason, why it is done a way it is currently... as it seems to have been done this way from the beginning.

So, let's try just running SIGCHLD handler directly instead of doing scheduling with GotSigChld variable.

Let's check first if this fixes it and then I'll think further.

try3.patch in attachment

(file #48133)

That would be a lot easier if I could reproduce it...

Anyway I suspect that with patch counter may still be decreased to early.

I attached new one, can you try with it?

(file #48132)

As mentioned on mailing list, it is a bad solution.
Try newly attached patch instead.

(file #47810)

Hi Amadeusz,

comment #4:

> from quick look I would say that WNOHANG in waitpid() looks suspicious:
> https://git.savannah.gnu.org/cgit/screen.git/tree/src/screen.c?h=screen-v4#n1495
>
> Can you try removing it and see if it still happens?

Bingo! This indeed seems to have solved the issue.

At least I can no more reproduce it as soon as I close all my screen session, install the patched version (via patched Debian package) and try to reproduce it again.

For reproducing, I start a screen session with a lot of windows (87× the line "screen" in my .screenrc for testing purposes, around 30 probably already suffice), then wait until all shells have properly started up, then I do C-a C-p until I'm at approximately window 52. If I then press C-d inside a shell to close one or more of them, with the unpatched version I get as many zombie process as I closed windows. They vanish all at once when I have close the last shell inside the screen session together with the session. And the patched version works as expected, no zombie processes.

After testing with the patched version, I closed my screen session, installed the unpatched screen version again, and I was able to reproduce the issue immediately again.

JFTR: this is the patch I applied:

--- a/screen.c
+++ b/screen.c
@@ -1492,7 +1492,7 @@
   if (!wstat_valid && p->w_pid > 0) {
       /* EOF on file descriptor. The process is probably also dead.
        * try a waitpid */
-    if (waitpid(p->w_pid, &wstat, WNOHANG | WUNTRACED) == p->w_pid) {
+    if (waitpid(p->w_pid, &wstat, WUNTRACED) == p->w_pid) {
       p->w_pid = 0;
       wstat_valid = 1;
     }

> You can also try building screen 4.7.0, it should do better system detection now, so maybe it gets fixed by that.

Didn't change with the screen package from Debian Unstable/Testing (4.7.0-1) built and installed on Debian 10 Buster.

So I applied the above patch to the 4.7.0-1 Debian package, too, and the issue was gone, too.

We're now running the patched 4.6.2-3 in production. So far my coworker couldn't reproduce the issue instantly, but then again, he never tried to reproduce it on purpose before. (Just I did.)

A coworker ran into this on Debian 10 Buster (amd64) with Screen 4.6.2 (Debian package version 4.6.2-3) inside a (production) VM (ESX) and I was able to reproduce this under my user on the same machine.

I was able to reproduce it by opening many (like 20 or 30 or so) screen windows and then closing some of them (not the most recently created ones) again. Not every closed shell became a zombie processes, though.

Looks like this at the moment:

→ ps auxwwwf | egrep 'Z|defunct|SCREEN' | fgrep -v grep
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
usera    32453  0.0  0.7  23776 15320 ?        Ss   Sep11  17:09 SCREEN
usera    22527  0.0  0.0      0     0 ?        Zs   Oct07   0:00  \_ [bash] <defunct>
usera    17308  0.0  0.0      0     0 ?        Zs   Oct08   0:00  \_ [bash] <defunct>
userb    17774  0.0  0.6  20440 13972 ?        Ss   Oct09   0:08 SCREEN
userb    17876  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    17902  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    17927  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    18124  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    18150  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    18174  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    20071  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    20104  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    20137  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>

Those process zombies have no according screen window anymore, i.e. are not "screen zombie windows". From the user perspective it seems as if the according shell has exited as expected and the according screen window has been closed.

Maybe also worth to note is that these zombies process are not just there for a few minutes and then get cleaned up. Most of them (but not all) stay for at least days, if not until I exit the screen session. 

I though wasn't yet able to reproduce it on any other machine, neither with the same Debian release, the same Debian screen package (but on real hardware). So I'm still unsure what actually triggers this issue, but it only seems to surface under specific conditions, but it's still unclear under which conditions.

"usera" has rather short and not fancy .screenrc file:

   → cat ~usera/.screenrc
   autodetach on
   hardstatus alwayslastline "%{+b dr}${USER}@%H: %n %t | %w | %l | %c:%s"
   startup_message off
  
   screen 0

And userb has no .screenrc at all on this host.

The relevant parts of the /etc/screenrc present on that system (actually Debian's default /etc/screenrc) are:

→ egrep -v '^$|^#' /etc/screenrc
deflogin on
vbell on
vbell_msg "   Wuff  ----  Wuff!!  "
defscrollback 1024
bind ^k
bind ^\
bind \\ quit
bind K kill
bind I login on
bind O login off
bind } history
termcapinfo vt100 dl=5\E[M
hardstatus off
termcapinfo xterm*|rxvt*|kterm*|Eterm* hs:ts=\E]0;:fs=\007:ds=\E]0;\007
hardstatus string "%h%? users: %u%?"
termcapinfo xterm*|linux*|rxvt*|Eterm* OP
termcapinfo xterm 'is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;4;6l'
defnonblock 5


It also seems to be independent of the login shell. My coworker uses bash, I use zsh, and the issue is present with both.

I also wondered what this box might have in common with the original reporter's box, but the only very vague connection is that a VM might be also considered a "slow" machine.

So I also tried it on another, less potent VM (Xen this time) with Debian 10 Buster amd64, but wasn't able to reproduce this issue there either.

No ideas, really, other than staring at the source for a while to see if anything jumps out (or, perhaps, to see if gdb allows it to be reproduced, unlike strace).