bugGNU Octave - Bugs: bug #55395, Java tests messes up the stack...

 
 

bug #55395: Java tests messes up the stack limits, causes failure of sparse/bicgstab.m

Submitter:  Stefan Brüns <stefanbruens>
Submitted:  Sat 05 Jan 2019 12:49:23 AM UTC
   
 
Category:  Test Suite Severity:  3 - Normal
Priority:  5 - Normal Item Group:  None
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * 4.4.1 Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 06 Jan 2019 03:22:04 AM UTC, comment #13: 

Aha, I get the segmentation fault your originally reported now, only if I run octave-cli instead of octave-gui.


#0  0x00007fef12f295e9 in dgetrf_parallel () from /lib/x86_64-linux-gnu/libopenblas.so.0
#1  0x00007fef12f29795 in dgetrf_parallel () from /lib/x86_64-linux-gnu/libopenblas.so.0
#2  0x00007fef16f5a548 in dgetrf_ () from /lib/x86_64-linux-gnu/liblapack.so.3
#3  0x00007fef1957a400 in Matrix::fsolve (this=<optimized out>, mattype=..., b=..., info=@0x7fff8f1bf260: 0,
    rcon=@0x7fff8f1bf268: 0, sing_handler=0x7fef1ae387f0 <solve_singularity_warning(double)>, calc_cond=true)
    at ../liboctave/array/dMatrix.cc:1689
#4  0x00007fef1957c2e6 in Matrix::solve (this=this@entry=0x7fff8f1bf2e0, mattype=..., b=...,
    info=@0x7fff8f1bf260: 0, rcon=@0x7fff8f1bf268: 0,
    sing_handler=0x7fef1ae387f0 <solve_singularity_warning(double)>, singular_fallback=true, transt=blas_no_trans)
    at ../liboctave/array/dMatrix.cc:1806
#5  0x00007fef1ae3a8cc in xleftdiv (a=..., b=..., typ=..., transt=transt@entry=blas_no_trans)
    at ../libinterp/corefcn/xdiv.cc:352


Mike Miller <mtmiller>
Group Member
Sun 06 Jan 2019 01:55:25 AM UTC, comment #12: 

Each thread has its own stack ...

You are running octave-gui, while I am running octave-cli. With octave-gui, the JVM obviously is not created from the main thread:


Thread 7 "QThread" hit Breakpoint 1, 0x00007fffd3b83c24 in JavaThread::create_stack_guard_pages() () from /usr/lib64/jvm/java-11-openjdk-11/lib/server/libjvm.so
(gdb) info threads
  Id   Target Id                                          Frame
  1    Thread 0x7fffec857b00 (LWP 2963) "octave-gui"      0x00007ffff55132c9 in poll () from /lib64/libc.so.6
  2    Thread 0x7fffec82d700 (LWP 2972) "octave-gui"      0x00007ffff31ea4dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3    Thread 0x7fffec02c700 (LWP 2973) "octave-gui"      0x00007ffff31ea4dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4    Thread 0x7fffe782b700 (LWP 2974) "octave-gui"      0x00007ffff31ea4dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5    Thread 0x7fffe24d9700 (LWP 2975) "QXcbEventQueue"  0x00007ffff55132c9 in poll () from /lib64/libc.so.6
  6    Thread 0x7fffdafa6700 (LWP 2977) "QDBusConnection" 0x00007ffff55132c9 in poll () from /lib64/libc.so.6
* 7    Thread 0x7fffd9d95700 (LWP 2978) "QThread"         0x00007fffd3b83c24 in JavaThread::create_stack_guard_pages() () from /usr/lib64/jvm/java-11-openjdk-11/lib/server/libjvm.so
  8    Thread 0x7fffd941a700 (LWP 2979) "QThread"         0x00007ffff545be2c in sigtimedwait () from /lib64/libc.so.6
(gdb) p/x $rsp
$1 = 0x7fffd9d7c510


Mapping for the JVM threads stack (see stack pointer register):

grep -C2 7fffd9d  /proc/$(pgrep octave-gui)/maps
7fffd9596000-7fffd9597000 rw-p 00000000 00:00 0
7fffd9597000-7fffd959b000 ---p 00000000 00:00 0
7fffd959b000-7fffd9d96000 rw-p 00000000 00:00 0              <== 0x7fffd9d7c510
7fffd9d96000-7fffda7a6000 rw-s 00000000 fe:03 76421698                   /home/stefan/.cache/icon-cache.kcache
7fffda7a6000-7fffda7a7000 ---p 00000000 00:00 0
7fffda7a7000-7fffdafa7000 rw-p 00000000 00:00 0


Mapping for the main threads stack:

grep -C2 stack /proc/$(pgrep octave-gui)/maps
7ffff7ffd000-7ffff7ffe000 rw-p 00026000 00:2f 4047686                    /lib64/ld-2.27.so
7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


Stefan Brüns <stefanbruens>
Sun 06 Jan 2019 01:36:10 AM UTC, comment #11: 

Currently testing for this bug with Octave 4.4.1:


>> version
ans = 4.4.1
>> version -hgid
ans = 1f46d371968c
>> version -java
ans = Java 11.0.1+13-Debian-3 with Oracle Corporation OpenJDK 64-Bit Server VM mixed mode, sharing


I do get stopped on a breakpoint set at 'JavaThread::create_stack_guard_pages', three times in the interpreter thread if I keep continuing execution, plus multiple times in several other spawned threads. Here is the top of the call stack of the first instance in the interpreter thread when loading Java the first time:


#0  0x00007fa35ab87f90 in JavaThread::create_stack_guard_pages() (this=0x7fa354653000)
    at ./src/hotspot/os/linux/os_linux.inline.hpp:47
#1  0x00007fa35ab883d3 in Threads::create_vm(JavaVMInitArgs*, bool*)
    (args=args@entry=0x7fa3546510e0, canTryAgain=canTryAgain@entry=0x7fa370a9a347)
    at ./src/hotspot/share/runtime/thread.cpp:3703
#2  0x00007fa35a733441 in JNI_CreateJavaVM_inner (args=args@entry=0x7fa3546510e0, penv=0x7fa370a9a6e0,
    penv@entry=0x7fa370a9a0b0, vm=vm@entry=0x7fa39576cf80 <jvm>) at ./src/hotspot/share/prims/jni.cpp:3945
#3  0x00007fa35a733441 in JNI_CreateJavaVM(JavaVM**, void**, void*)
    (vm=vm@entry=0x7fa39576cf80 <jvm>, penv=penv@entry=0x7fa370a9a6e0, args=args@entry=0x7fa370a9a5e0)
    at ./src/hotspot/share/prims/jni.cpp:4036
#4  0x00007fa394ecdea8 in initialize_jvm() () at ../libinterp/octave-value/ov-java.cc:230
#5  0x00007fa394ece5ab in initialize_java() () at ../libinterp/octave-value/ov-java.cc:1876


The stack space in /proc/N/maps remains 140 kB on my system:


$ grep -C2 stack /proc/$(pgrep octave-gui)/maps
7fa395aff000-7fa395b00000 rw-p 00027000 fe:01 6824752                    /usr/lib/x86_64-linux-gnu/ld-2.28.so
7fa395b00000-7fa395b01000 rw-p 00000000 00:00 0
7ffce506c000-7ffce508e000 rw-p 00000000 00:00 0                          [stack]
7ffce5130000-7ffce5133000 r--p 00000000 00:00 0                          [vvar]
7ffce5133000-7ffce5135000 r-xp 00000000 00:00 0                          [vdso]


Mike Miller <mtmiller>
Group Member
Sat 05 Jan 2019 10:13:35 PM UTC, comment #10: 

@mtmiller - which version of octave are you running?

If your JVM does not insert a stack guard page for the main (Java calls in primordial) thread, there must be a difference.

The relevant code likely is here:
https://github.com/JetBrains/jdk8u_hotspot/commit/8302c38e12be9a7e0259c81b6ca426cd7f79f53d#diff-852a551ce56c81e8f2f0770135243c14

Can you set a breakpoint on 'JavaThread::create_stack_guard_pages()', and see if there are any differences:


(gdb) bt 7
#0  0x00007fffe71dec42 in JavaThread::create_stack_guard_pages() () from /usr/lib64/jvm/java-11-openjdk-11/lib/server/libjvm.so
#1  0x00007fffe71df063 in Threads::create_vm(JavaVMInitArgs*, bool*) () from /usr/lib64/jvm/java-11-openjdk-11/lib/server/libjvm.so
#2  0x00007fffe6d88492 in JNI_CreateJavaVM () from /usr/lib64/jvm/java-11-openjdk-11/lib/server/libjvm.so
#3  0x00007ffff74f96c8 in ?? () from /usr/lib64/octave/4.4.1/liboctinterp.so.6
#4  0x00007ffff74f9e05 in ?? () from /usr/lib64/octave/4.4.1/liboctinterp.so.6
#5  0x00007ffff74fa1be in FjavaMethod(octave_value_list const&, int) () from /usr/lib64/octave/4.4.1/liboctinterp.so.6
#6  0x00007ffff7458f08 in octave_builtin::call(octave::tree_evaluator&, int, octave_value_list const&) () from /usr/lib64/octave/4.4.1/liboctinterp.so.6
(More stack frames follow...)
(gdb) info threads
  Id   Target Id                                      Frame
* 1    Thread 0x7fffef84aa00 (LWP 17550) "octave-cli" 0x00007fffe71dec42 in JavaThread::create_stack_guard_pages() () from /usr/lib64/jvm/java-11-openjdk-11/lib/server/libjvm.so
  2    Thread 0x7fffef829700 (LWP 17551) "octave-cli" 0x00007ffff42614dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3    Thread 0x7fffef028700 (LWP 17552) "octave-cli" 0x00007ffff42614dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4    Thread 0x7fffea827700 (LWP 17553) "octave-cli" 0x00007ffff42614dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  5    Thread 0x7fffe7eac700 (LWP 17554) "octave-cli" 0x00007ffff57c2e2c in sigtimedwait () from /lib64/libc.so.6
(gdb) p/x $rsp
$5 = 0x7fffffffb9b0

 
maps:

7ffff7ffe000-7ffff7fff000 rw-p 00000000 00:00 0
7fffff805000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]


The JVM is obviously created from the main thread.

Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 08:13:02 PM UTC, comment #9: 

Ok, the segfault in the JVM cpu probing was a red herring. I am elaborating here for future reference, although it is not actually related to this bug.

The forced segfault was introduced here:
https://github.com/JetBrains/jdk8u_hotspot/commit/d2dc33eab19e26360bbaa15450b0686b69cb282c#diff-ade79d50bf46023a5dd26524c1a7945b

and is used to disable AVX when a linux kernel bug is present

    // Some OSs have a bug when upper 128bits of YMM
    // registers are not restored after a signal processing.
    // Generate SEGV here (reference through NULL)
    // and check upper YMM bits after it.
    //


"Some OSs" actually is 32bit userspace using a 32bit kernel on a 64bit machine (implied by AVX).

The kernel bug has been fixed 3 years ago:
https://lore.kernel.org/patchwork/patch/616662/
I.e. anyone using Linux >= 4.4-rc2 or a 64bit kernel does not need this piece of code.

Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 06:10:32 PM UTC, comment #8: 

The Debian openblas package is compiled with NUM_THREADS=64, and I cannot reproduce this issue on my system. After 'usejava("jvm")', Octave's stack size is the same as before, 140 kB.


>> javamem

Java virtual machine (JVM) memory info:
Maximum available memory:         3992 MiB;
   (...running garbage collector...)
OK, current status:
Total memory in virtual machine:    14 MiB;
Free memory in virtual machine:  12.8955 MiB;
8 CPUs available.


Mike Miller <mtmiller>
Group Member
Sat 05 Jan 2019 06:54:20 AM UTC, comment #7: 

Are you sure it is not the same as:

https://bugzilla.redhat.com/show_bug.cgi?id=1572811

(See comment #7 by Severin Gehwolf)

Dmitri,
--

Dmitri A. Sergatskov <dasergatskov>
Sat 05 Jan 2019 03:37:26 AM UTC, comment #6: 

This is a genuine Java bug, it also segfaults (typically not visible, as it has a signal handler installed) when running standalone.

You can see this when running e.g.

$> gdb -q --batch -ex run -ex "bt 6" -ex quit java

Thread 2 "java" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5e80700 (LWP 26318)]
0x00007fffd863d526 in ?? ()
#0  0x00007fffd863d526 in ?? ()
#1  0x0000000000000202 in ?? ()
#2  0x00007fffd863d280 in ?? ()
#3  0x00007ffff7386374 in Abstract_VM_Version::_vm_major_version () from /usr/lib64/jvm/java-11-openjdk-11/lib/server/libjvm.so
#4  0x00007ffff5e7f480 in ?? ()
#5  0x00007ffff6fd3899 in VM_Version::get_processor_features() () from /usr/lib64/jvm/java-11-openjdk-11/lib/server/libjvm.so


Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 03:27:42 AM UTC, comment #5: 

Ok, dug a little bit deeper ...

Apparently, the JVM installs a temporary protection page during initalization. Now, when probing for CPU features, it gets a SEGFAULT - see https://savannah.gnu.org/bugs/?func=detailitem&item_id=52061#comment19 , and fails to remove the protection page in its fault handler.

As a side effect, the JVM stack size detection later fails, and the stack size stays wrong.

This is on a Haswell here.

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf


Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 02:23:25 AM UTC, comment #4: 

Fedora has NUM_THREADS=128.
On my machine javamem returns:


octave:1> javamem
OpenJDK 64-Bit Server VM warning: Archived non-system classes are disabled because the java.system.class.loader property is specified (value = "org.octave.OctClassLoader"). To use archived non-system classes, this property must be not be set

Java virtual machine (JVM) memory info:
Maximum available memory:         3980 MiB;
   (...running garbage collector...)
OK, current status:
Total memory in virtual machine:    14 MiB;
Free memory in virtual machine:  12.9078 MiB;
8 CPUs available.


OPENBLAS_NUM_THREADS seems to be a run-time option, but I could not get a crash with either.

Dmitri.


Dmitri A. Sergatskov <dasergatskov>
Sat 05 Jan 2019 01:59:48 AM UTC, comment #3: 

NUM_THREADS is not a runtime, but compile time option of OpenBLAS. If fedora uses <~ 40 or > 80, you won't see the issue.

Whats the output of javamem?


octave:1> javamem

Java virtual machine (JVM) memory info:
Maximum available memory:         1972 MiB;
   (...running garbage collector...)
OK, current status:
Total memory in virtual machine:     8 MiB;
Free memory in virtual machine:  7.29816 MiB;
4 CPUs available.


(using java.opts: -Xss8m)

Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 01:29:01 AM UTC, comment #2: 

I am not sure what I am doing but I cannot reproduce this with 5.01:


NUM_THREADS=100 LD_PRELOAD=/usr/lib64/libopenblasp.so ./run-octave -q -f


octave:1> usejava ("jvm")
OpenJDK 64-Bit Server VM warning: Archived non-system classes are disabled because the java.system.class.loader property is specified (value = "org.octave.OctClassLoader"). To use archived non-system classes, this property must be not be set
ans = 1
octave:2> test ../scripts/sparse/bicgstab.m
PASSES 12 out of 12 tests
octave:3>

=======
before usejava

cat /proc/27230/maps | tail
7f5ac3c8f000-7f5ac3caf000 r-xp 00001000 08:03 6306371                    /usr/lib64/ld-2.28.so
7f5ac3caf000-7f5ac3cb7000 r--p 00021000 08:03 6306371                    /usr/lib64/ld-2.28.so
7f5ac3cb7000-7f5ac3cb8000 r--s 00000000 00:31 248902                     /run/user/1001/dconf/user
7f5ac3cb8000-7f5ac3cb9000 r--p 00029000 08:03 6306371                    /usr/lib64/ld-2.28.so
7f5ac3cb9000-7f5ac3cba000 rw-p 0002a000 08:03 6306371                    /usr/lib64/ld-2.28.so
7f5ac3cba000-7f5ac3cbb000 rw-p 00000000 00:00 0
7fffd726e000-7fffd7291000 rw-p 00000000 00:00 0                          [stack]
7fffd72df000-7fffd72e2000 r--p 00000000 00:00 0                          [vvar]
7fffd72e2000-7fffd72e4000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

after usejava
cat /proc/27230/maps | tail
7f5ac3c8f000-7f5ac3caf000 r-xp 00001000 08:03 6306371                    /usr/lib64/ld-2.28.so
7f5ac3caf000-7f5ac3cb7000 r--p 00021000 08:03 6306371                    /usr/lib64/ld-2.28.so
7f5ac3cb7000-7f5ac3cb8000 r--s 00000000 00:31 248902                     /run/user/1001/dconf/user
7f5ac3cb8000-7f5ac3cb9000 r--p 00029000 08:03 6306371                    /usr/lib64/ld-2.28.so
7f5ac3cb9000-7f5ac3cba000 rw-p 0002a000 08:03 6306371                    /usr/lib64/ld-2.28.so
7f5ac3cba000-7f5ac3cbb000 rw-p 00000000 00:00 0
7fffd726e000-7fffd7291000 rw-p 00000000 00:00 0                          [stack]
7fffd72df000-7fffd72e2000 r--p 00000000 00:00 0                          [vvar]
7fffd72e2000-7fffd72e4000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


This is on Fedora 29 java-11 octave-5.0.1 (pre-release)

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 05 Jan 2019 01:23:54 AM UTC, comment #1: 

This can be worked around by adding a java.opts with a proper limit, e.g. 8 MByte.

It is somewhat surprising loading the JVM leads to crashes in completely unrelated code ...

Stefan Brüns <stefanbruens>
Sat 05 Jan 2019 12:49:23 AM UTC, original submission:  

After running anything involing the JVM, tests involving OpenBLAS may fail. This depends on the compilation options of OpenBLAS, specifically its NUM_THREADS option, which influences the required stack size. NUM_THREADS <= 32 is fine (~260 kByte required), as is NUM_THREADS > 80 (OpenBLAS allocates on the heap). See https://github.com/xianyi/OpenBLAS/issues/246
Windows is not affected, as the heap allocation threshold is 32, not 80.

The issue can be reproduced as follows:

1. start octave:
/bin/sh ../run-octave -g  --norc --silent --no-history

2. diagnostic - show process memory map:
cat /proc/<pid>/maps | tail


7fa92239f000-7fa9223a0000 rw-p 00000000 00:00 0
7ffe49c2e000-7ffe49c50000 rw-p 00000000 00:00 0                          [stack]
7ffe49c80000-7ffe49c83000 r--p 00000000 00:00 0                          [vvar]
7ffe49c83000-7ffe49c85000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


3. load JVM
octave:1> usejava ("jvm")
ans = 1

4. new process mapping:


7ffe49b50000-7ffe49b54000 ---p 00000000 00:00 0
7ffe49b54000-7ffe49c50000 rw-p 00000000 00:00 0                          [stack]
7ffe49c80000-7ffe49c83000 r--p 00000000 00:00 0                          [vvar]
7ffe49c83000-7ffe49c85000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


=> loading the JVM fixes the stack size to 1 MByte.

5. Run sparse/bicgstab.m test:
octave:2> test ../scripts/sparse/bicgstab.m
Segmentation fault (core dumped)

Stefan Brüns <stefanbruens>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by stefanbruens (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-06 mtmiller StatusWorks For Me Confirmed
    2019-01-05 mtmiller StatusNone Works For Me

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code