This fixes it! Thank you!

To summarize: In order to use javaruntime.exec from within Octave (default Octave package) on Ubuntu 20.04, make sure that a file named /usr/share/octave/5.2.0/m/java/java.opts exists and contains the line

-Djdk.lang.processReaperUseDefaultStackSize=true

Thank you dear Octave developers for fixing this Octave bug somewhere between Octave-5.2.0 and Octave-6.0.90:

While using the same openjdk 11.0.11 with both versions, this bug manifests with Octave-5.2.0 but not anymore with Octave-6.0.90.

Would you please point me to what you have changed to fix this bug so that maybe we can provide an updated 5.2.0 with a bugfix for our users that might still use ubuntu 20.04 and therefore Octave-5.2.0 for the next 4 years?

Thank you very much, T Herzke

(By the way, the suggested java.opts settings have had no effect).

maybe use a java.opts file to control initial heap allocation?
See also bug #53943

My java.opts has this:

-Xms64m
-Xmx128m

This was working fine in octave 4.2.2 with openjdk 11 (default versions on ubuntu 18.04).

It fails with octave 5.2.0 with openjdk 11 (default versions on ubuntu 20.04). It also fails when replacing openjdk 11 with openjdk 8.

All tests were performed with standard ubuntu packages for octave and default-jre-headless on the respective releases 18.04 and 20.04.

Steps to reproduce with a simpe example:

• In ubuntu 20.04, install octave and java:

apt install octave default-jre-headless

• Start octave, enter these three commands:

javaruntime = javaMethod('getRuntime','java.lang.Runtime');
command = 'touch /tmp/testfile1';
process = javaruntime.exec(command)


Expected results:
1. A file /tmp/testfile1 should have been created.
2. Variable "process" should now reference a java object of type java.lang.ProcessImpl.
3. No error occurs.

Actual results:
1. A file /tmp/testfile1 has actually been created as expected. The child process has successfully executed.
2. Variable "process" does not exist because javaruntime.exec() was terminated with an error.
3. First a warning occurs, then an error occurs. Warning and error messages:

[0.063s][warning][os,thread] Failed to start thread - pthread_create failed (EINVAL) for attributes: stacksize: 136k, guardsize: 0k, detached.
error: [java] java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

I have tried this on multiple systems running ubuntu focal, including fresh installations, therefore this is not caused by some system with quirky thread creation limits.

When running this example in octave with strace

strace -f --trace=clone octave --eval example

(where example.m contains the three lines from above),
I can see that the child process is created successfully and exits with exit code 0 before the warning about thread creation is printed.