bugGNU Octave - Bugs: bug #55395, Java tests messes up the stack...

 
 

bug #55395: Java tests messes up the stack limits, causes failure of sparse/bicgstab.m

Submitted by:  Stefan Brüns <stefanbruens>
Submitted on:  Sat 05 Jan 2019 12:49:23 AM UTC  
 
Category:  Test Suite Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  4.4.1 Operating System:  GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 06 Jan 2019 03:22:04 AM UTC, comment #13:

Aha, I get the segmentation fault your originally reported now, only if I run octave-cli instead of octave-gui.

Mike Miller <mtmiller>
Project Administrator
Sun 06 Jan 2019 01:55:25 AM UTC, comment #12:

Each thread has its own stack ...

You are running octave-gui, while I am running octave-cli. With octave-gui, the JVM obviously is not created from the main thread:

Mapping for the JVM threads stack (see stack pointer register):

Mapping for the main threads stack:

Stefan Brüns <stefanbruens>
Sun 06 Jan 2019 01:36:10 AM UTC, comment #11:

Currently testing for this bug with Octave 4.4.1:

I do get stopped on a breakpoint set at 'JavaThread::create_stack_guard_pages', three times in the interpreter thread if I keep continuing execution, plus multiple times in several other spawned threads. Here is the top of the call stack of the first instance in the interpreter thread when loading Java the first time:

The stack space in /proc/N/maps remains 140 kB on my system:

Mike Miller <mtmiller>
Project Administrator
Sat 05 Jan 2019 10:13:35 PM UTC, comment #10:

@mtmiller - which version of octave are you running?

If your JVM does not insert a stack guard page for the main (Java calls in primordial) thread, there must be a difference.

The relevant code likely is here:
https://github.com/JetBrains/jdk8u_hotspot/commit/8302c38e12be9a7e0259c81b6ca426cd7f79f53d#diff-852a551ce56c81e8f2f0770135243c14

Can you set a breakpoint on 'JavaThread::create_stack_guard_pages()', and see if there are any differences:

maps:

The JVM is obviously created from the main thread.

Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 08:13:02 PM UTC, comment #9:

Ok, the segfault in the JVM cpu probing was a red herring. I am elaborating here for future reference, although it is not actually related to this bug.

The forced segfault was introduced here:
https://github.com/JetBrains/jdk8u_hotspot/commit/d2dc33eab19e26360bbaa15450b0686b69cb282c#diff-ade79d50bf46023a5dd26524c1a7945b

and is used to disable AVX when a linux kernel bug is present

"Some OSs" actually is 32bit userspace using a 32bit kernel on a 64bit machine (implied by AVX).

The kernel bug has been fixed 3 years ago:
https://lore.kernel.org/patchwork/patch/616662/
I.e. anyone using Linux >= 4.4-rc2 or a 64bit kernel does not need this piece of code.

Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 06:10:32 PM UTC, comment #8:

The Debian openblas package is compiled with NUM_THREADS=64, and I cannot reproduce this issue on my system. After 'usejava("jvm")', Octave's stack size is the same as before, 140 kB.

Mike Miller <mtmiller>
Project Administrator
Sat 05 Jan 2019 06:54:20 AM UTC, comment #7:

Are you sure it is not the same as:

https://bugzilla.redhat.com/show_bug.cgi?id=1572811

(See comment #7 by Severin Gehwolf)

Dmitri,
--

Dmitri A. Sergatskov <dasergatskov>
Sat 05 Jan 2019 03:37:26 AM UTC, comment #6:

This is a genuine Java bug, it also segfaults (typically not visible, as it has a signal handler installed) when running standalone.

You can see this when running e.g.

Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 03:27:42 AM UTC, comment #5:

Ok, dug a little bit deeper ...

Apparently, the JVM installs a temporary protection page during initalization. Now, when probing for CPU features, it gets a SEGFAULT - see https://savannah.gnu.org/bugs/?func=detailitem&item_id=52061#comment19 , and fails to remove the protection page in its fault handler.

As a side effect, the JVM stack size detection later fails, and the stack size stays wrong.

This is on a Haswell here.

Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 02:23:25 AM UTC, comment #4:

Fedora has NUM_THREADS=128.
On my machine javamem returns:

octave:1> javamem
OpenJDK 64-Bit Server VM warning: Archived non-system classes are disabled because the java.system.class.loader property is specified (value = "org.octave.OctClassLoader"). To use archived non-system classes, this property must be not be set

Java virtual machine (JVM) memory info:
Maximum available memory: 3980 MiB;
(...running garbage collector...)
OK, current status:
Total memory in virtual machine: 14 MiB;
Free memory in virtual machine: 12.9078 MiB;
8 CPUs available.

OPENBLAS_NUM_THREADS seems to be a run-time option, but I could not get a crash with either.

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Sat 05 Jan 2019 01:59:48 AM UTC, comment #3:

NUM_THREADS is not a runtime, but compile time option of OpenBLAS. If fedora uses <~ 40 or > 80, you won't see the issue.

Whats the output of javamem?

(using java.opts: -Xss8m)

Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 01:29:01 AM UTC, comment #2:

I am not sure what I am doing but I cannot reproduce this with 5.01:

This is on Fedora 29 java-11 octave-5.0.1 (pre-release)

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 05 Jan 2019 01:23:54 AM UTC, comment #1:

This can be worked around by adding a java.opts with a proper limit, e.g. 8 MByte.

It is somewhat surprising loading the JVM leads to crashes in completely unrelated code ...

Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 12:49:23 AM UTC, original submission:

After running anything involing the JVM, tests involving OpenBLAS may fail. This depends on the compilation options of OpenBLAS, specifically its NUM_THREADS option, which influences the required stack size. NUM_THREADS <= 32 is fine (~260 kByte required), as is NUM_THREADS > 80 (OpenBLAS allocates on the heap). See https://github.com/xianyi/OpenBLAS/issues/246
Windows is not affected, as the heap allocation threshold is 32, not 80.

The issue can be reproduced as follows:

1. start octave:
/bin/sh ../run-octave -g --norc --silent --no-history

2. diagnostic - show process memory map:
cat /proc/<pid>/maps | tail

3. load JVM
octave:1> usejava ("jvm")
ans = 1

4. new process mapping:

=> loading the JVM fixes the stack size to 1 MByte.

5. Run sparse/bicgstab.m test:
octave:2> test ../scripts/sparse/bicgstab.m
Segmentation fault (core dumped)

Stefan Brüns <stefanbruens>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by stefanbruens (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-06 mtmiller StatusWorks For Me => Confirmed
    2019-01-05 mtmiller StatusNone => Works For Me

    Back to the top


    Powered by Savane 3.4