bugGNU Screen - Bugs: bug #25089, screen produces zombies

 
 

bug #25089: screen produces zombies

Submitted by:  Michal Svoboda <pht1>
Submitted on:  Sun 14 Dec 2008 07:36:20 AM UTC  
 
Category:  None Severity:  3 - Normal
Priority:  5 - Normal Status:  Need Investigation
Privacy:  Public Assigned to:  None
Open/Closed:  Open Release:  4.0.3
Fixed Release:  None Planned Release:  Later
Work Required:  1 - Days

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 29 Dec 2019 08:42:58 AM UTC, comment #13: 

Yay! Thanks for fixing this decade old bug.

Michal Svoboda <pht1>
Sat 28 Dec 2019 07:34:59 PM UTC, comment #12: 

The reason for this is that in the C language only some very narrow set of library functions can be invoked from the signal handler

FreeBSD sigaction(2) manpage lists them and adds:

https://www.freebsd.org/cgi/man.cgi?query=sigaction&apropos=0&sektion=2&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html

> All functions not in the above lists are considered to be unsafe with
> respect to signals.  That is to say, the behaviour of such functions is
> undefined when they are called from a signal handler that interrupted an
> unsafe function.  In general though, signal handlers should do little
> more than set a flag; most other actions are not safe.


Therefore it was the best practice in the signal handlers only to   set the variable and deal with the actual handling later.

Marcin Cieślak <saper>
Sat 28 Dec 2019 07:23:15 PM UTC, comment #11: 

I can't find a reason, why it is done a way it is currently... as it seems to have been done this way from the beginning.

So, let's try just running SIGCHLD handler directly instead of doing scheduling with GotSigChld variable.

Let's check first if this fixes it and then I'll think further.

try3.patch in attachment

(file #48133)

Amadeusz Sławiński <amade>
Project Administrator
Sat 28 Dec 2019 06:02:55 PM UTC, comment #10: 

Actually, ignore it for now, I hurried up a bit and forgot to test run it properly and screen hangs... will send better one

Amadeusz Sławiński <amade>
Project Administrator
Sat 28 Dec 2019 06:00:38 PM UTC, comment #9: 

That would be a lot easier if I could reproduce it...

Anyway I suspect that with patch counter may still be decreased to early.

I attached new one, can you try with it?

(file #48132)

Amadeusz Sławiński <amade>
Project Administrator
Fri 22 Nov 2019 02:59:37 PM UTC, comment #8: 

comment #7:

> As mentioned on mailing list, it is a bad solution.
> Try newly attached patch instead.
>
> (file #47810)


As mentioned on mailing list, I've done that now.
Unfortunately the new patch doesn't fix the issue.

Axel Beckert <abe>
Wed 06 Nov 2019 08:36:58 PM UTC, comment #7: 

As mentioned on mailing list, it is a bad solution.
Try newly attached patch instead.

(file #47810)

Amadeusz Sławiński <amade>
Project Administrator
Wed 23 Oct 2019 12:41:20 PM UTC, comment #6: 

Hi again,

comment #5:

> We're now running the patched 4.6.2-3 in production. So far my coworker couldn't reproduce the issue instantly, but then again, he never tried to reproduce it on purpose before. (Just I did.)


JFTR: No more sightings of screen child zombies on that VM since then. So I consider the patch to be sufficient to fix this issue.

Axel Beckert <abe>
Mon 14 Oct 2019 04:48:36 PM UTC, comment #5: 

Hi Amadeusz,

comment #4:

> from quick look I would say that WNOHANG in waitpid() looks suspicious:
> https://git.savannah.gnu.org/cgit/screen.git/tree/src/screen.c?h=screen-v4#n1495
>
> Can you try removing it and see if it still happens?


Bingo! This indeed seems to have solved the issue.

At least I can no more reproduce it as soon as I close all my screen session, install the patched version (via patched Debian package) and try to reproduce it again.

For reproducing, I start a screen session with a lot of windows (87× the line "screen" in my .screenrc for testing purposes, around 30 probably already suffice), then wait until all shells have properly started up, then I do C-a C-p until I'm at approximately window 52. If I then press C-d inside a shell to close one or more of them, with the unpatched version I get as many zombie process as I closed windows. They vanish all at once when I have close the last shell inside the screen session together with the session. And the patched version works as expected, no zombie processes.

After testing with the patched version, I closed my screen session, installed the unpatched screen version again, and I was able to reproduce the issue immediately again.

JFTR: this is the patch I applied:

--- a/screen.c
+++ b/screen.c
@@ -1492,7 +1492,7 @@
   if (!wstat_valid && p->w_pid > 0) {
       /* EOF on file descriptor. The process is probably also dead.
        * try a waitpid */
-    if (waitpid(p->w_pid, &wstat, WNOHANG | WUNTRACED) == p->w_pid) {
+    if (waitpid(p->w_pid, &wstat, WUNTRACED) == p->w_pid) {
       p->w_pid = 0;
       wstat_valid = 1;
     }

> You can also try building screen 4.7.0, it should do better system detection now, so maybe it gets fixed by that.


Didn't change with the screen package from Debian Unstable/Testing (4.7.0-1) built and installed on Debian 10 Buster.

So I applied the above patch to the 4.7.0-1 Debian package, too, and the issue was gone, too.

We're now running the patched 4.6.2-3 in production. So far my coworker couldn't reproduce the issue instantly, but then again, he never tried to reproduce it on purpose before. (Just I did.)

Axel Beckert <abe>
Fri 11 Oct 2019 06:04:58 PM UTC, comment #4: 

Well...
from quick look I would say that WNOHANG in waitpid() looks suspicious:
https://git.savannah.gnu.org/cgit/screen.git/tree/src/screen.c?h=screen-v4#n1495

Can you try removing it and see if it still happens?

You can also try building screen 4.7.0, it should do better system detection now, so maybe it gets fixed by that.

Amadeusz Sławiński <amade>
Project Administrator
Fri 11 Oct 2019 01:55:36 PM UTC, comment #3: 

A coworker ran into this on Debian 10 Buster (amd64) with Screen 4.6.2 (Debian package version 4.6.2-3) inside a (production) VM (ESX) and I was able to reproduce this under my user on the same machine.

I was able to reproduce it by opening many (like 20 or 30 or so) screen windows and then closing some of them (not the most recently created ones) again. Not every closed shell became a zombie processes, though.

Looks like this at the moment:

→ ps auxwwwf | egrep 'Z|defunct|SCREEN' | fgrep -v grep
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
usera    32453  0.0  0.7  23776 15320 ?        Ss   Sep11  17:09 SCREEN
usera    22527  0.0  0.0      0     0 ?        Zs   Oct07   0:00  \_ [bash] <defunct>
usera    17308  0.0  0.0      0     0 ?        Zs   Oct08   0:00  \_ [bash] <defunct>
userb    17774  0.0  0.6  20440 13972 ?        Ss   Oct09   0:08 SCREEN
userb    17876  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    17902  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    17927  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    18124  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    18150  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    18174  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    20071  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    20104  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>
userb    20137  0.0  0.0      0     0 ?        Zs   Oct09   0:00  \_ [zsh] <defunct>

Those process zombies have no according screen window anymore, i.e. are not "screen zombie windows". From the user perspective it seems as if the according shell has exited as expected and the according screen window has been closed.

Maybe also worth to note is that these zombies process are not just there for a few minutes and then get cleaned up. Most of them (but not all) stay for at least days, if not until I exit the screen session. 

I though wasn't yet able to reproduce it on any other machine, neither with the same Debian release, the same Debian screen package (but on real hardware). So I'm still unsure what actually triggers this issue, but it only seems to surface under specific conditions, but it's still unclear under which conditions.

"usera" has rather short and not fancy .screenrc file:

   → cat ~usera/.screenrc
   autodetach on
   hardstatus alwayslastline "%{+b dr}${USER}@%H: %n %t | %w | %l | %c:%s"
   startup_message off

  

   screen 0

And userb has no .screenrc at all on this host.

The relevant parts of the /etc/screenrc present on that system (actually Debian's default /etc/screenrc) are:

→ egrep -v '^$|^#' /etc/screenrc
deflogin on
vbell on
vbell_msg "   Wuff  ----  Wuff!!  "
defscrollback 1024
bind ^k
bind ^\
bind \\ quit
bind K kill
bind I login on
bind O login off
bind } history
termcapinfo vt100 dl=5\E[M
hardstatus off
termcapinfo xterm*|rxvt*|kterm*|Eterm* hs:ts=\E]0;:fs=\007:ds=\E]0;\007
hardstatus string "%h%? users: %u%?"
termcapinfo xterm*|linux*|rxvt*|Eterm* OP
termcapinfo xterm 'is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;4;6l'
defnonblock 5

It also seems to be independent of the login shell. My coworker uses bash, I use zsh, and the issue is present with both.

I also wondered what this box might have in common with the original reporter's box, but the only very vague connection is that a VM might be also considered a "slow" machine.

So I also tried it on another, less potent VM (Xen this time) with Debian 10 Buster amd64, but wasn't able to reproduce this issue there either.

Axel Beckert <abe>
Mon 26 Jan 2009 09:54:58 PM UTC, comment #2: 

Could the 'zombie' command be responsible for this? Could you try unsetting the zombie setting (using the 'zombie' command without any params), then terminating the process?

Sadrul Habib Chowdhury <sadrul>
Project Administrator
Thu 15 Jan 2009 10:55:24 PM UTC, comment #1: 

No ideas, really, other than staring at the source for a while to see if anything jumps out (or, perhaps, to see if gdb allows it to be reproduced, unlike strace).

Micah Cowan <micahcowan>
Sun 14 Dec 2008 07:36:20 AM UTC, original submission:  

I came across a funny thing: each process started by screen in a new window (eg. /bin/bash) will become a zombie after exiting it (eg. typing exit).

It looks like some sort of a race condition, because when I tried to strace the SCREEN process, it would correctly wait() and reclaim the process (in fact, all of the zombies were reclaimed). Also, my computer is quite slow:

model name      : Pentium II (Klamath)
cpu MHz         : 233.284

Other useful info:
Linux 2.6.27.8 (vanilla)
Screen version 4.00.03 (FAU) 23-Oct-06 (from Mandriva 2009.0)

Any ideas where to look at, how to pinpoint the problem?

Michal Svoboda <pht1>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #48133:  try3.patch added by amade (2KiB - text/x-patch - Just run handler directly instead using GotSigChld to schedule)
file #48132:  try2.patch added by amade (769B - text/x-patch - Decrease GotSigChld counter in better place)
file #47810:  0001-Fix-screen-leaving-zombie-processes.patch added by amade (1KiB - text/x-patch - New version of patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by saper (Posted a comment)
  • -email is unavailable- added by amade (Posted a comment)
  • -email is unavailable- added by abe (Posted a comment)
  • -email is unavailable- added by sadrul (Posted a comment)
  • -email is unavailable- added by pht1 (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-12-30 micahcowan Carbon-CopyRemoved 60294 => -
    2019-12-28 amade Attached File- => Added try3.patch, #48133
    2019-12-28 amade Attached File- => Added try2.patch, #48132
    2019-11-06 amade Attached File- => Added 0001-Fix-screen-leaving-zombie-processes.patch, #47810
    2009-01-15 micahcowan StatusNone => Need Investigation
        Planned ReleaseNone => Later
        Work RequiredNone => 1 - Days

    Back to the top


    Powered by Savane 3.5