bugGNU Octave - Bugs: bug #58641, Java crashes...

 
 

bug #58641: Java crashes (OutOfMemoryError/StackOverflowError) when invoking a child process

Submitted by:  None
Submitted on:  Mon 22 Jun 2020 03:43:01 PM UTC  
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Unexpected Error
Status:  Fixed Assigned to:  None
Originator Name:  Benjamin Moody Originator Email:  -email is unavailable-
Open/Closed:  Closed Release:  6.0.90
Operating System:  GNU/Linux

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Fri 03 Jul 2020 09:33:35 AM UTC, comment #21: 

Compile times of the build bot are back to normal.

If the only difference is whether ccache has hits or misses in its cache, it is very impressive how much faster it is...

Closing as fixed.

Markus Mützel <mmuetzel>
Project Member
Fri 03 Jul 2020 07:06:28 AM UTC, comment #20: 

The buildbots took very long to compile after this change. E.g. here:
http://buildbot.octave.org:8010/#/builders/1/builds/528

Given that the config header changed, I expected less hits from the ccache - so a somewhat longer build time. But about one and a half hours for a step that previously took less than 4 minutes seems a lot of difference...

Looking at the changes, I have no idea what might have caused this.
Are the buildbots ok otherwise?

Markus Mützel <mmuetzel>
Project Member
Thu 02 Jul 2020 04:17:55 PM UTC, comment #19: 

Thank you for testing.

I pushed the change (minus the debugging setting) here:
https://hg.savannah.gnu.org/hgweb/octave/rev/9e7b2625e574

Marking as ready for test.

Markus Mützel <mmuetzel>
Project Member
Wed 01 Jul 2020 06:35:22 PM UTC, comment #18: 

I can confirm the original issue happens with the current stable branch (on Debian 10, amd64, using openjdk-11), and after applying the patch it's fixed.

I'm not sure what $OCTAVE_JAVA_DIR should be, but I can override the settings by creating a file 'scripts/java/java.opts'.  If the setting is "false" then creating a process gives a StackOverflowError.

Benjamin Moody <bmoody>
Wed 01 Jul 2020 06:31:36 AM UTC, comment #17: 

I am no longer able to reproduce the original crash (even without the patch). So I wasn't sure if we'd need another mechanism to override the default value for the Java setting "jdk.lang.processReaperUseDefaultStackSize".
I thought the debugging messages could help if someone who is still affected by the original issue tries the patch (and the override mechanism outlined in comment #12).
But you are right: A final patch should not change Vdebug_java.

Markus Mützel <mmuetzel>
Project Member
Tue 30 Jun 2020 06:24:16 PM UTC, comment #16: 

Markus: the change looks good to me.  Did you mean to leave Vdebug_java set to true or was that just left over from debugging?

John W. Eaton <jwe>
Project Administrator
Tue 30 Jun 2020 06:19:01 PM UTC, comment #15: 

Sorry.
I hope this time I correctly attached the updated patch.

(file #49401)

Markus Mützel <mmuetzel>
Project Member
Tue 30 Jun 2020 05:28:29 PM UTC, comment #14: 

mmuetzel: Did you mean to attach a second patch?  I only see your first patch (bug58641_java_stacksize.patch, from comment #3).

Benjamin Moody <bmoody>
Mon 29 Jun 2020 02:52:10 PM UTC, comment #13: 

> Is this confined to OpenJDK, does Oracle Java behave differently?


I'm pretty sure that the Oracle JDK is actually built from the same code base as OpenJDK; it differs only in licensing and some deployment & packaging stuff added in. OpenJDK and Oracle JDK can be expected to behave the same way in pretty much all respects, including internal JVM behaviors like this.

Andrew Janke <apjanke>
Fri 26 Jun 2020 04:54:09 PM UTC, comment #12: 

I'm not sure what changed. But I'm no longer able to reproduce the original error.

With the attached patch, the workaround is used if the test from comment #7 fails (i.e. for all glibc versions for now).
With it, Octave should no longer crash or block for the initial test.

We already use $OCTAVE_JAVA_DIR/java.opts to pass user arguments to the JVM (with a fallback to OCTAVE_FCNFILEDIR/java/java.opts if $OCTAVE_JAVA_DIR isn't set).

Can someone who can still reproduce the crash please test if the attached patch solves the issue? (Run bootstrap, configure and make with the usual parameters)
If it does, does it crash again after executing the following and restarting Octave:

system('echo -Djdk.lang.processReaperUseDefaultStackSize=false > $OCTAVE_JAVA_DIR/java.opts')

The patch also activates debug messages for Java. A final patch should not do that.

Markus Mützel <mmuetzel>
Project Member
Fri 26 Jun 2020 07:14:41 AM UTC, comment #11: 

If I understood correctly, setting that JVM options only takes any effect if the JVM spawns new threads. If that option is not set on system with the buggy glibc, the JVM (and with it Octave) will crash.
So I'm not sure, we need an option to change JVM start parameters to fix this bug here. But I agree that such an option could be useful.

What about fixing this error (with the hard coded option on affected systems) on stable?
We could add the interface that you are proposing on default afterwards.

Markus Mützel <mmuetzel>
Project Member
Thu 25 Jun 2020 09:42:22 PM UTC, comment #10: 

The test program could be useful for setting default option values.

I was thinking that it should be possible to set the list of options for the JVM in an interpreter config file.

Is there a place other than the initialize_jvm function in ov-java.cc where JVM options are used?  If not, then it seems we could provide a function that users could call from startup files (or any time before the JVM is initialized) that could set the internal list of options.  If this function is called after the JVM is initialized, it could issue a warning and do nothing.

We could still provide a default set of options and allow this new function to return current set of options.  That way you could append/modify or completely override the defaults without having to rebuild Octave.

John W. Eaton <jwe>
Project Administrator
Tue 23 Jun 2020 04:53:37 PM UTC, comment #9: 

@bmoody: Thanks for the test. It looks like it should be straight forward to make this a configure test.

@jwe: Is having that as a configure test what we want?
Do we have an options file or something similar for liboctave to keep track of such a setting? Maybe an environment variable to overwrite whatever the configure test would suggest?
I remember that we needed something similar for the interpreter to store the mfile encoding to be able to correctly parse the startup files (bug #57235). But I don't know if that was done.

Markus Mützel <mmuetzel>
Project Member
Mon 22 Jun 2020 10:27:23 PM UTC, comment #8: 

comment #1:

> Out of curiosity:
> Is this confined to OpenJDK, does Oracle Java behave differently?


Oracle JDK 14.0.1 appears to use the same stack size (139264) as OpenJDK 11.  Haven't tried it with Octave but I would expect to see the same problems.

I'd guess you probably wouldn't see these problems on Windows, or other non-GNU systems, but I don't know.

Benjamin Moody <bmoody>
Mon 22 Jun 2020 09:45:10 PM UTC, comment #7: 

I suppose it's not too complicated to test whether glibc has this issue (see attachment.)

Testing whether the JRE is affected by it seems trickier.

FTR, the underlying bug in glibc is https://sourceware.org/bugzilla/show_bug.cgi?id=11787 (comments there also mention a somewhat scary workaround being used by the Rust compiler.)

(file #49352)

Benjamin Moody <bmoody>
Mon 22 Jun 2020 08:22:53 PM UTC, comment #6: 

Or, maybe we could also make these options more easily configurable by allowing them to be set in a startup file instead of hard coded into the ov-java.cc file.

John W. Eaton <jwe>
Project Administrator
Mon 22 Jun 2020 08:01:32 PM UTC, comment #5: 

"__GLIBC__" (for glibc >=6) or "__GNU_LIBRARY__" (for glibc <= 5) is defined in <features.h>. But that header doesn't exist on all platforms. The standard header <limits> includes that header in glibc (see e.g. [1]).
Alternatively, we could add a configure check for the features header.

Reading the bug reports, it looks like older Java versions didn't limit the stack size of the reaper thread as much.
Maybe a Java expert can provide a test.

[1]: https://sourceforge.net/p/predef/wiki/Libraries/

Markus Mützel <mmuetzel>
Project Member
Mon 22 Jun 2020 07:41:02 PM UTC, comment #4: 

Why is <limits> needed for this change?

I recommend adding some explanation about the purpose of this option somewhere in the changeset.  Maybe it would be best to make a note in a comment inside the #if block to explain why this option is needed for glibc.

Better yet, is there a way to test for the bug and only add the option if the bug is really present?  Maybe different versions of  glibc or Java don't require it?

John W. Eaton <jwe>
Project Administrator
Mon 22 Jun 2020 07:29:44 PM UTC, comment #3: 

The attached patch works as a workaround to avoid the issue for me.

(file #49351)

Markus Mützel <mmuetzel>
Project Member
Mon 22 Jun 2020 06:48:17 PM UTC, comment #2: 

On Centos 8.2 with
java -version
openjdk version "11.0.7" 2020-04-14 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.7+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.7+10-LTS, mixed mode, sharing)

On 6.0.1 I get:

octave:1> usejava ("jvm");
octave:2> b = javaObject('java.lang.ProcessBuilder', {'/bin/true'});
octave:3> p = b.start();

Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "process reaper"

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 22 Jun 2020 05:37:30 PM UTC, comment #1: 

Thanks for this bug report and your analysis of the issue.

Out of curiosity:
Is this confined to OpenJDK, does Oracle Java behave differently?

I'm asking because while AFAICS on Linux OpenJDK works reliably for me, I've seen some fairly bad OpenJDK implementations on Windows.

Anyway, if you can supply a changeset (preferrably) or a patch we can try it out and see (1) if we can reproduce the bug and (2) if the cset/patch fixes it.

FYI, the "Archived non-system classes are disabled" warning you mentioned in the first URL in your bug report is bug #55174 here; coincidentally just fixed yesterday.

Philip Nienhuis <philipnienhuis>
Project Member
Mon 22 Jun 2020 03:43:01 PM UTC, original submission:  

When using a recent version of OpenJDK, if an Octave program invokes a Java function that in turn creates a child process, it can cause a stack overflow in an internal Java thread.  This can cause the program to crash or to freeze when waiting for the child process to finish.

A simple example is to run the following:

b = javaObject('java.lang.ProcessBuilder', {'/bin/true'});
p = b.start();
p.waitFor()

On older versions of Octave and OpenJDK, this should return 0.  On newer versions, it will either show an error message, or freeze the Octave process completely.

The reason appears to be twofold:

  • newer versions of Octave are using more thread-local storage than older versions did;
  • when creating a "process reaper" thread, newer versions of OpenJDK allocate less stack space (which must include thread-local storage space) than older versions of OpenJDK did.

More details in the Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961681

Also see the Ubuntu bug report: https://bugs.launchpad.net/ubuntu/+source/octave/+bug/1866453

It seems to me that the simplest solution would be for Octave to set the Java system property "jdk.lang.processReaperUseDefaultStackSize".  If this property is set to "true", Java will use the same stack size for "reaper" threads as it uses for other threads.  Otherwise, "reaper" threads use a much smaller stack size (currently 128k.)

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49352:  pthread-stacksize-test.c added by bmoody (1KiB - text/x-csrc)
file #49351:  bug58641_java_stacksize.patch added by mmuetzel (1KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by bmoody (Updated the item)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-07-03 mmuetzel StatusReady For Test => Fixed
        Open/ClosedOpen => Closed
    2020-07-02 mmuetzel StatusNeed Info => Ready For Test
    2020-06-30 mmuetzel Attached File- => Added bug58641_java_reaper_thread_stacksize_v2.patch, #49401
    2020-06-26 mmuetzel StatusIn Progress => Need Info
        Release5.2.0 => 6.0.90
    2020-06-22 bmoody Attached File- => Added pthread-stacksize-test.c, #49352
    2020-06-22 mmuetzel Attached File- => Added bug58641_java_stacksize.patch, #49351
        StatusNone => In Progress

    Back to the top


    Powered by Savane 3.5