bugGNU Octave - Bugs: bug #37591, system with async option results...

 
 

bug #37591: system with async option results in zombie processes

Submitter:  Mike Miller <mtmiller>
Submitted:  Thu 18 Oct 2012 12:37:48 AM UTC
   
 
Category:  Libraries Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Other
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Fri 22 Sep 2017 04:56:37 PM UTC, comment #13: 

I agree that callbacks are cleaner, generally speaking.  (At very low levels like assembly instructions and specific architecture the equivalent is interrupts.)  Polling isn't a good use of resources, and there is always the question of how often something should be polled.  Plus, most system environments and frameworks (Qt, etc.) have moved away from polling so the programming of callbacks isn't challenging.

As for the severity of this zombie behavior, as noted, if forked processes do in fact terminate taking all the memory along with it then it isn't much consequence.  However, I could still imagine someone doing a very long batch process that involves partial Octave code and then a system call to some other program:


REPEAT(hundreds_of_files) {
  RUN_OCTAVE_PROCESSING;
  SAVE_RESULTS_TO_FILE;
  cmd = sprintf("post_process OUTFILE%d", i);
  system (cmd, [], "async");
}


Do this in a multi-user environment, e.g., mainframe, and the administrator might look at the PID list and wonder what's happening.

Anyway, more importantly, the callback concept relates to signal handling, which Octave never seemed to get quite right.  For example, we've often run into problems with Cntrl-C, complicated by the fact there is a GUI present.  And if the SIGINT were to be revamped, might as well look at SIGCHLD at the same time.

For example, what is this "octave_quit()" starting to show up all over the code?  I've looked at that function, and it appears that it traps the SIGINT such that the SIGINT can only be handled by calling octave_quit().  If I'm understanding correctly, that isn't very elegant programming, i.e., having to remember to place an octave_quit() here and there.  There must be a better way of dealing with SIGINT.

Dan Sebald <sebald>
Fri 22 Sep 2017 03:52:59 PM UTC, comment #12: 

Because this is a potentially valuable feature I don't see a reason to remove it.  It is true that, in general, callbacks are cleaner than polling protocols, but it is also dependent on the particulars of the design in question.

There are reasons to believe that this is not a tremendous problem for Octave.  First, there hasn't been a rash of bug reports reporting problems with resource usage.  Second, we know that the zombie process doesn't consume large amounts of memory, disk space, pixels, or some other finite resource.  Rather it takes up just a small amount of space in the process table.  There is a limit to the size of the kernel process table, but again it seems unlikely that Octave would be spawning hundreds of subprocesses.

Octave is all volunteer, so if this is the "itch" you want to scratch then go for it.  But to my mind there are many more impactful bugs that could be tackled first.

For this one, I think at least a first pass should be a documentation fix.  The docstring for system has no seealso link to waitpid, and there is no discussion at all about the repercussions of choosing "async".

Rik <rik5>
Group administrator
Fri 22 Sep 2017 03:34:44 PM UTC, comment #11: 

Re Anonymous - yes, that's what I was proposing, and it sounds like it's a bad idea because users may rely on the process ID of the async'd process for information. So I think we should leave things the way they are.

Re Dan - background periodic reaping is not complicated at all, the code is already in Octave to do exactly this. I think adding a user callback function support would make this much more complicated.

I agree with comment #9, the existing behavior should be kept in place as it is now, and maybe a new option or a new function for adding process IDs to be harvested could be added.

This could be a new option on the system function, something like


system ("xterm", [], "async", "cleanup");


Or it could be a completely new function to let the caller tell Octave that it doesn't care about the process ID and wants it to be harvested whenever possible, something like


pid = system ("xterm", [], "async");
waitpid_later (pid);


This function would not block, it would simply add the process ID to Octave's child_list to be cleaned up whenever it exits eventually.

Mike Miller <mtmiller>
Group Member
Fri 22 Sep 2017 07:24:26 AM UTC, comment #10: 

Well, I'm sort of "thinking" more about this one right now rather than "acting".  I see in the synchronous case, Octave is using


  iprocstream *cmd = new iprocstream (cmd_str.c_str ());


one of the C++ interfaces to processes.  So, maybe that takes care of all the cleanup.  I see that is being used by _gnuplot_version_() to run gnuplot and inquire its version.  I don't see any zombie for that sub-process.

Maybe there is an asynchronous C++ interface that we could use instead of execl().

In any case, I've a feeling the proper way to do this is not the periodic reaping of processes in the background.  First, it seems like a rather complex hunk of code to deal with.  Second, we don't know how long these processes are going to last, so how do we tell when they are done?  I'd rather avoid thant.

I was thinking, after the discussion and patch for https://savannah.gnu.org/bugs/index.php?52084, that the most straightforward thing would be to handle that SIGCHLD interrupt when the process is complete.  By "handle", I mean THEN reap the sub-process; but in addition to that given what you've said in the previous two posts we could maybe add a callback function of some sort:

system('ls',0,'async',@callbackfunction);

where @callbackfunction is psuedocode for however the figures implement callback functions in which the arguments of the function should be the exit-status and so on.  The reaping would be done automatically, but the exit status info is present as Octave variables.

With such a setup, rather than you doing the polling approach on long processes, you could simply use callback functions for a much cleaner solution.  That sounds like better-placed effort to me.

Dan Sebald <sebald>
Fri 22 Sep 2017 06:57:14 AM UTC, comment #9: 

With respect to comment #5: are you proposing to periodically reap asynchronously started child processes in the background? Actually, I have code that relies on the present behaviour: I use octave to run a number of processes in the background (lengthy calculations in external programs) in parallel. I start a number of these processes asynchronously, and then periodically check with waitpid(..,WNOHANG) whether one has exited, in which case I start the next one.

So yes, I see that it is problematic if octave is gathering zombie processes. So if you introduce automatic periodic reaping of these processes, then please make provisions to allow also for excluding specific processes from that, for instance by introducing further type options in the system call, for instance "async_noreap".

Anonymous
Fri 22 Sep 2017 06:53:26 AM UTC, comment #8: 

Yes, there is nothing you can do in the child to influence this behavior, it has to be done by Octave.

If you are calling system directly, you can reap these defunct processes with


>> pid = system ("ls", [] "async")
pid =  16256
>> system ps;
  PID TTY          TIME CMD
16235 pts/0    00:00:00 octave-cli-4.3.
16256 pts/0    00:00:00 sh <defunct>
16259 pts/0    00:00:00 sh
16260 pts/0    00:00:00 ps
29689 pts/0    00:00:00 bash
>> waitpid (pid);
>> system ps;
  PID TTY          TIME CMD
16235 pts/0    00:00:01 octave-cli-4.3.
16307 pts/0    00:00:00 sh
16308 pts/0    00:00:00 ps
29689 pts/0    00:00:00 bash


I think this leads to the decision point of this problem, should system(..., "async") automatically reap all child processes automatically behind the scenes? Or should it be the responsibility of the caller, at the interpreter level, to call waitpid(pid) for every invocation of system(..., "async")?

It can't be done both ways. If Octave is in charge of reaping these defunct processes, then the caller will never have access to the exit status, etc, they will be truly async with no information available about whether they ever ran or are still running.

Mike Miller <mtmiller>
Group Member
Fri 22 Sep 2017 06:43:50 AM UTC, comment #7: 

OK, then that's why the amount of memory used by the "sh" processes in the lists indicates "N/A" rather than actually memory volumes.  The processes are gone, in fact, but the remnants are still there.

Dan Sebald <sebald>
Fri 22 Sep 2017 06:39:55 AM UTC, comment #6: 

Actually, the shell "sh" probably is exiting with "-c" option.  The following terminal commands should illustrate:


sebald@ ~/octave/octave/octave $ sh
$ exit
sebald@ ~/octave/octave/octave $ sh -c "ls"
aclocal.m4
sebald@ ~/octave/octave/octave $


However, the execl documentation states:

"
Return Value

The exec() functions only return if an error has occurred. The return value is -1, and errno is set to indicate the error.
"

So, I've placed ";exit 1" at the end of the command.  That definitely causes different behavior, with error messages, but it still doesn't get rid of the forked processes.

Dan Sebald <sebald>
Fri 22 Sep 2017 06:35:08 AM UTC, comment #5: 

The shell has terminated. The process is no longer running. The only thing remaining is the operating system bookkeeping of the dead child process that the parent needs to reap (things like exit or fatal signal status).

When a child process completes, it's done. The shell is no longer running. If the parent that created the child is still around, it is required to call wait() on the child to tell the operating system that it can be cleaned up. If the parent is no longer around, then the child is reparented to the init process, which in turn calls wait() and cleans up the dead process.

In this case, Octave has started a process, the process has finished, and Octave is not calling wait() on it. Octave does call wait on children that it starts synchronously, just not the async ones.

Octave already maintains a table of child processes that are periodically reaped. If we could arrange for the child process IDs to be taken care of in the same way, this should take care of itself.

Mike Miller <mtmiller>
Group Member
Fri 22 Sep 2017 06:17:28 AM UTC, comment #4: 

I made some comments in thread https://savannah.gnu.org/bugs/index.php?52084 about "sh" hangs around after an asynchronous system().  Any insight into this?  It sort of makes sense that they should be there because Octave is launching a shell as follows:


  if (pid == 0)
    execl (SHELL_PATH, "sh", "-c", cmd, (char *) (0));


Notice there is nothing in that command that tells the shell (i.e., "sh") to exit once the command "cmd" is complete.  So, my thinking is that the shell is just sitting there.  I've attempted a couple things.  One, I tagged ";exit" on to the end of the command cmd, and two, made the change


    execl (SHELL_PATH, "sh", "-c", "-e", "-i", cmd, (char *) (0));


The -e is supposed to terminate if the system command fails (a good idea, as the user could type in any nonsensical words).  The -i is to force the shell to behave interactive, as the -c places the shell into non-interactive state.  I tried with and without the -i.  Basically, I can't seem to get that shell to terminate when it seems like it should be easy to do so.

Dan Sebald <sebald>
Sat 19 Nov 2016 10:45:59 PM UTC, comment #3: 

This is still happening with Octave 4.2.0 (on linux).

Hartmut <hardy>
Wed 09 Jul 2014 03:03:11 PM UTC, comment #2: 

In bug #42665, it was reported that Octave can also create gnuplot zombie processes if gnuplot is killed outside of Octave's knowledge. If Octave then tries to interact with the plot, it gets a broken pipe error and gets stuck somewhere, but this is a separate issue. If Octave subsequently closes the figure, the process is cleaned up.

Mike Miller <mtmiller>
Group Member
Thu 17 Oct 2013 09:19:11 PM UTC, comment #1: 

Finally got around to testing this and I can confirm that Octave is leaving zombies around all the time.

Rik <rik5>
Group administrator
Thu 18 Oct 2012 12:37:48 AM UTC, original submission:  

Each call of


system (cmd, [], "async");


leaves a zombie process on Linux, probably MacOS and other unices as well. The sigchld_handler function is called for each process that exits, but nothing is done to clean up these processes until Octave exits. Exiting Octave simply allows init to clean up the leftovers.

Example:


octave:1> system ("ps");
  PID TTY          TIME CMD
  528 pts/3    00:00:00 lt-octave
 3357 pts/3    00:00:00 sh
 3358 pts/3    00:00:00 ps
19089 pts/3    00:00:00 bash
octave:2> system ("uname", [], "async");
octave:3> system ("uname", [], "async");
octave:4> system ("uname", [], "async");
octave:5> system ("uname", [], "async");
octave:6> system ("ps");
  PID TTY          TIME CMD
  528 pts/3    00:00:00 lt-octave
 3375 pts/3    00:00:00 sh <defunct>
 3377 pts/3    00:00:00 sh <defunct>
 3380 pts/3    00:00:00 sh <defunct>
 3384 pts/3    00:00:00 sh <defunct>
 3393 pts/3    00:00:00 sh
 3394 pts/3    00:00:00 ps
19089 pts/3    00:00:00 bash


A more common way to trigger this behavior is the edit command, which defaults to async mode when starting the editor.

If edit or system are called enough times, the user can overrun the process limit and no more processes can be started until Octave exits.

Mike Miller <mtmiller>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Carbon-Copy List
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by hardy (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-10-24 mmuetzel Dependencies- bugs #63256 is dependent
    2019-02-26 mtmiller Carbon-CopyRemoved 80942 -
    2014-07-09 mtmiller Dependencies- bugs #42665 is dependent
    2013-10-17 rik5 StatusNone Confirmed

    Back to the top

    Powered by Savane 3.13-54b4.
    Corresponding source code