bugGNU Octave - Bugs: bug #53733, Panic/segfault when calling Java...

 
 

bug #53733: Panic/segfault when calling Java method after forge:im[read|write]

Submitted by:  Adrian <adaerr>
Submitted on:  Mon 23 Apr 2018 10:38:07 PM UTC  
 
Category:  Libraries Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Segfault, Bus Error, etc.
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  4.4.0 Operating System:  GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 23 Dec 2018 10:04:20 PM UTC, comment #81:

I had to kill -9 mine manually after about ~50 minutes running.
It was hanging at N = 1e9. If I run directly (w/o loop)
with N=1e10 I get:

octave:1> javaaddpath (".")
OpenJDK 64-Bit Server VM warning: Archived non-system classes are disabled because the java.system.class.loader property is specified (value = "org.octave.OctClassLoader"). To use archived non-system classes, this property must be not be set
octave:2> fmt = imformats ();
octave:3> graph = javaObject ("Baz");
octave:4> graph.addNodes (1:1e10);
error: out of memory or dimension too large for Octave's index type
octave:4>

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 23 Dec 2018 09:45:56 PM UTC, comment #80:

When I use the script from comment #75, I also DO get a crash at N=10^9 with Octave 5.0.1 (hg-id 9644c0b7efd8). The call for imformats was included. It takes 2-3 minutes.

Sorry, so this issue seems to be still alive :(

Hartmut <hardy>
Sun 23 Dec 2018 09:39:50 PM UTC, comment #79:

Less than a second before crashing at N=1e5 when I call imformats first. About 3-5 seconds to crash at N=1e9 when I don't call imformats.

Mike Miller <mtmiller>
Project Administrator
Sun 23 Dec 2018 08:58:35 PM UTC, comment #78:

OK. This is better. How long does it take before it crashes?
Mine is sitting at N=10^9 for about 10 minutes already with 100% cpu load and 16GB virt and 8GB res memory in use...

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 23 Dec 2018 08:43:30 PM UTC, comment #77:

Dmitri - you need to download and compile the Baz.java attachment.

Mike Miller <mtmiller>
Project Administrator
Sun 23 Dec 2018 08:28:25 PM UTC, comment #76:

This is what i got:

octave:1> javaaddpath (".")
OpenJDK 64-Bit Server VM warning: Archived non-system classes are disabled because the java.system.class.loader property is specified (value = "org.octave.OctClassLoader"). To use archived non-system classes, this property must be not be set
octave:2> fmt = imformats ();
octave:3> graph = javaObject ("Baz");
error: [java] java.lang.ClassNotFoundException: Baz
octave:3> for k=1:10

> N=10^k
> graph.addNodes (1:N);
> endfor

N = 10
warning: the 'graph' function is not yet implemented in Octave

Please read <https://www.octave.org/missing.html> to learn how you can
contribute missing functionality.
error: 'graph' undefined near line 3 column 3
octave:3>

This is with java-11

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 23 Dec 2018 08:15:27 PM UTC, comment #75:

Please try the code example from comment #59:

This still crashes Octave 5.0.1 (d00577d95369) with a segmentation fault for me.

With the call to imformats, it crashes for me at N = 10^5. If the imformats line is removed, it crashes for me at N = 10^9.

Mike Miller <mtmiller>
Project Administrator
Sun 23 Dec 2018 07:42:47 PM UTC, comment #74:

I followed the procedure outlined in comment #0 and could NOT generate any crash of a self-compiled Octave 5.0.1 (hg-id 9644c0b7efd8) on Ubuntu 18.04 with an image-2.10 pre-release package. (Sorry, I haven't read the 72 comments inbetween...)

Can anyone confirm, that this bug is GONE in Octave 5.0.1? (Without anyone intentionally solving it...)

Hartmut <hardy>
Thu 24 May 2018 10:12:50 PM UTC, comment #73:

Another observation: graphicsmagick plays quite nicely when it comes
to signal handling: it refrains from installing its own handler when
one is already installed. Or to be precise: because there is no
function to simply inquire about the existing handler, graphicsmagick
installs its own signal handler, checks whether there was a
non-default handler installed, and restores the old handler if there
was one. See the source at:

http://hg.code.sf.net/p/graphicsmagick/code/file/356b0eba718c/magick/magick.c#l633

In any case, gdb confirms that the initialisation of
ImageMagick does not modify the signal handler for SIGSEGV:

Does this mean that the SEGV signal handling is not the culprit in
this bug, despite java's shenanigans in throwing that signal for an
argument size of 10^5 elements (about equal to the stack size) ?

The series of instructions

produces a SIGSEGV at the last line, but executes fine if the second
line is omitted. The table of signal handlers however is exactly the
same in both cases (whether ImageMagick is initialised or not).

A detailled debugging session follows for the script above.

BTW, if someone has a nicer way (than setting a breakpoint on __GI___poll as above) of returning to gdb after octave finishes executing one instruction and returns to the prompt, I'd be happy to learn.

Adrian <adaerr>
Thu 24 May 2018 09:01:39 PM UTC, comment #72:

TLDR; Compiling with AddressSanitizer prevents premature SEGV stop.

I built the same source (current tip a few days ago: hg_id = 2cf750f5cb7d) with java-10 once with AdressSanitizer options:

and once without:

Then the following test script is executed on both builds:

The non-ASan version stops through a SEGV at k=5, while the build linked with ASan runs up to k=9 before stopping through a SEGV. No leak is reported.

Here's the output. Note the ASAN option to exclude SEGV signal handling by that tool.

Adrian <adaerr>
Mon 07 May 2018 03:18:50 PM UTC, comment #71:

TLDR; In essence java 10 behaves as java 9 with respect to SIGSEGV
emission for the test script triggering bug at hand.

This was tested by downloading OpenJDK 64bit version "10.0.1"
2018-04-17 from java.net and compiling the current octave development
tip (af64921cdb28) against it. The machine has Intel Xeon E5520 CPUs
and is running Debian stable with kernel 4.9.0.

There are minor differences to java 9, e.g. the critical argument size
in the method call that triggers a SIGSEGV signal is decreased from
1e6 (java-9) to 1e5 (java-10).

Here is the output:

Without the call to imformats, the addNodes(1:10^k) instructions
succeed for k=1..8, where octave takes several seconds to
complete the k=8 case. The segfault then occurs for an argument
size of 1e9, after somme calculation and swapping.

With gdb attached, and omitting the call to imformats, a SIGSEGV is
received for k=5, but can be ignored (either via "handle SIGSEGV
nostop print pass" in .gdbinit, or by typing "continue" at the gdb
prompt). Output follows (gdb is attached and the process execution
continued before pasting the test script lines at the octave prompt;
the 51 "New Thread" creations are in particular the result of the
instruction 'javaaddpath(".")'):

With the call to imformats the SIGSEGV occuring
for "graph.addNodes(1:1e5);" is fatal, though only after
continuing twice in gdb:

Adrian <adaerr>
Thu 03 May 2018 05:14:39 PM UTC, comment #70:

From http://www.graphicsmagick.org/NEWS.html

<<<<
Signal Handling: Signal handling is now more robust and handles SIGSEGV and other critical signals. The sole purpose of the default signal handling is to remove any temporary files and quit. An informative message is printed for signals other than SIGINT.

>>>>


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 03 May 2018 05:07:25 PM UTC, comment #69:

BTW -- according to redhat bugzilla thread (and confirmed by java.com) -- java 9 is dead long live java 10.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 03 May 2018 05:05:35 PM UTC, comment #68:

Before the call to javaaddpath, there are two threads, the octave-cli main thread and the octave-cli signal_watcher thread.

After calling 'javaaddpath .', 32 new threads are started by Java (Java 9.0.4 in this case).

Mike Miller <mtmiller>
Project Administrator
Thu 03 May 2018 04:57:31 PM UTC, comment #67:

And, add threads...

What is starting all those threads shown in the gdb transcript? Is that the JVM or GraphicsMagick or Octave or something else?

John W. Eaton <jwe>
Project Administrator
Thu 03 May 2018 04:22:12 PM UTC, comment #66:

Yes, Octave has a way of handling SIGSEGV, and I'm starting to suspect that GraphicsMagick also has a separate SIGSEGV handler, and Java is using/handling SIGSEGV in yet another way.

Mike Miller <mtmiller>
Project Administrator
Thu 03 May 2018 03:56:28 PM UTC, comment #65:

Another observation: If you call imformats() before javaaddpath("."), then you have the same behaviour as when you don't call it at all (comment #63). Could it be that both the jvm and graphicsmagick change signal handling in such a way that it affects the other ? Is signal handling global, i.e. shared between these parts and octave ? So if graphicsmagick initialises before the jvm is created, the JVM's SIGSEGV-trickery works fine, but otherwise graphicsmagick's initialisation code modifies signal handlers in such a way that the signal that the JVM was supposed to catch and handle internally for its stack banging escapes the JVM and kills octave ?

In accordance with the idea that the order of initialisation matters, if imformats() is called before javaaddpath("."), then it may be called again afterwards without reducing the argument size producing a fatal segfault. That is both

and

go up to N=1e9 before crashing octave with a segfault (as discussed in comment #63, a continuable segfault is caught by gdb for N=1e5, and is not affecting the test script when running outside gdb).

Adrian <adaerr>
Thu 03 May 2018 03:24:01 PM UTC, comment #64:

Interesting indeed. Same observations (as comment #62 and comment #63) for release-4-2-1 compiled against java-8 respectively java-9. I tried 4.2.1 specifically because I had previously (comment #34) observed that 4.2.1 behaved differently from 4.2.2. This seems to be due only to the difference in the java version that the Debian octave packages were built with (java-8 for 4.2.1, java-9 for 4.2.2).

Adrian <adaerr>
Thu 03 May 2018 07:33:08 AM UTC, comment #63:

This might be important. With Java 9, when I comment out the call to imformats, I get the following output in gdb

Notice the additional SIGSEGV received and handled at N=10^5. I do not see that with Java 8. It could be that this additional SIGSEGV is supposed to be a normal part of the Java stack probing behavior, that for some reason turns into a fatal signal delivered to Octave if and only if GraphicsMagick has been initialized.

That's not the only bug here, Octave should also not be able to crash even if the user tries to call a Java function with an array of size 10^20. But this difference in behavior between Java 8 and 9, which is possibly intentional, may help explain why it takes a much smaller array to cause a segfault with Java 9 or newer.

Mike Miller <mtmiller>
Project Administrator
Thu 03 May 2018 07:23:02 AM UTC, comment #62:

Yes it does. I get the following.

followed by an endless supply of SIGSEGV lines.

When N gets to be 10^8, it takes maybe 10 seconds of 100% busy before it prints 1e+09. At 10^9 it is still 100% busy, and after a few seconds starts printing endless screenfuls of SIGSEGV.

From what I have seen, the main difference between Java 8 and Java 9 is that Java 8 always produces a segfault when the argument is of length 10^9, whether a GraphicsMagick function has been called yet or not. That's the same behavior I see with Java 9 when a GraphicsMagick function has not been called. With Java 9, the size to produce a segfault is reduced to 10^5 after a GraphicsMagick function has been called. That's basically what I described in comment #60.

The "endless supply of SIGSEGV lines" symptom only occurs when running with gdb when imformats or some other GraphicsMagick function has been called. If no image functions are called, the segmentation fault always occurs at N=10^9, and it is only a single SIGSEGV message along with Octave's "stopping myself" diagnostic message.

Mike Miller <mtmiller>
Project Administrator
Thu 03 May 2018 06:12:25 AM UTC, comment #61:

@Mike Miller: I'd be curious to know whether the Octave 4.2.2 linked with OpenJDK 8 that you mentioned in comment #3 still produces a segfault in gdb once the SIGSEGV generated by the JVM for stack overflow detection have been made non-fatal in gdb ("handle SIGSEGV nostop print pass" in .gdbinit).

Here's again the modification of the test from comment #59 that probes various argument sizes:

Adrian <adaerr>
Tue 01 May 2018 09:56:48 PM UTC, comment #60:

The GraphicsMagick code isn't in a loadable dldfcn oct file anymore, it's in builtin functions now.

So based on what we've found so far, if I had to summarize this, it's that it has to do with passing large arrays in to Java functions via JNI. There is some size at which Octave goes into a 100% busy state (and never returns? takes an exponentially long time?) but does not crash. And there is some larger size at which Octave crashes immediately with a SIGSEGV.

For some as-yet-unknown reason, calling any GraphicsMagick functions in Octave before the Java function is called drastically reduces these size limits.

Ideally, Octave would know about this limit or be able to detect this condition dynamically and print an error message that some size threshold has been crossed and avoid the segmentation fault.

Mike Miller <mtmiller>
Project Administrator
Mon 30 Apr 2018 11:20:54 PM UTC, comment #59:

All the following is on a box with an Intel Core i5-2500K CPU, on which under gdb the 'javaaddpath (".")' instruction terminates the octave process:

I can confirm that creating a .gdbinit containing "handle SIGSEGV nostop print
pass" makes octave survive the first SIGSEGV, but line 4 is still
fatal with the same signal:

For completeness, outside gdb the script from comment #58 always
produces a segfault in line 4, whether with the .gdbinit "handle
SIGSEGV nostop print pass" configuration or ot, and whether run via
"octave --norc --no-gui comment58.m" or "octave-cli --norc
comment58.m":

A modified script changing the argument size along the lines of
what Dmitri did, namely:

  • outside gdb either segfaults or freezes (CPU load->100%) for k=5 / N=1e5,
  • inside gdb, with "handle SIGSEGV nostop print pass" in .gdbinit (otherwise as usual segfault at first line), shows a curious behaviour:

The last line is the pager offering me as many pages of segfault
messages as I had patience scrolling through, and when I quit it
typing 'q <return>', I return to the gdb prompt.

So according to Rik's explanation a large number of SIGSEGV are just caught inside the JVM for overflow detection. But what causes one signal to be eventually passed on to octave (or so it seems) ?

Adrian <adaerr>
Mon 30 Apr 2018 06:46:56 PM UTC, comment #58:

Well, that is interesting. Of course, executing just imformats is enough to load in the GraphicsMagick dldfcn, as well as execute some pretty intricate code with persistent variables in subfunctions for which mlock has been called.

It would help if the original reporter could verify the new proposed test code still produces a segfault.

Rik <rik5>
Project Administrator
Mon 30 Apr 2018 06:22:25 PM UTC, comment #57:

I don't know. But removing the call to imwrite makes a larger sized argument

work just fine. It takes a while and is busy most of the time, doing Octave-to-Java array type conversion I suspect, but exits without error.

Simply calling 'imformats' to print the image format types supported by GraphicsMagick, instead of imwrite, is enough to allow the 1:1e6 argument to lead to a segfault.

Mike Miller <mtmiller>
Project Administrator
Mon 30 Apr 2018 06:08:10 PM UTC, comment #56:

@Mike: is this just due to a difference in the amount of memory reserved for the JVM between versions 8 and 9? The fact that changing the Java code to use less memory stops the segfault is suggestive.

Rik <rik5>
Project Administrator
Mon 30 Apr 2018 06:01:52 PM UTC, comment #55:

The same test code still causes a segfault in Octave for me.

With Octave 4.2.2 linked with Java 8, I get no segfault.

With Octave 4.2.2 linked with Java 9:

With Octave 4.3.92 linked with Java 9:

If I reduce the size of the array passed in to addNodes, I get no segfault.

My processor is Intel(R) Core(TM) i7-4770HQ.

Mike Miller <mtmiller>
Project Administrator
Mon 30 Apr 2018 05:48:17 PM UTC, comment #54:

I added the following to my local .gdbinit

In this case, I get a printout about the segmentation fault which is used internally by JVM for overflow detection, but it doesn't cause gdb to immediately abort. I am also using the pass option, so a true SIGSEGV would be passed to enclosing program which created the JVM if it were real. But, this doesn't happen indicating that it is a spurious thing.

I ran both the tiny test program, and also Octave, and neither issues more than one SIGSEGV (associated with the startup of the JVM). The test code in Octave I used was

Also, without gdb the code above runs fine multiple times using

Don't know if it was the case that the original reporter was running with the '-f' flag to prevent user customizations in .octaverc which might actually be triggering this.

Rik <rik5>
Project Administrator
Mon 30 Apr 2018 02:04:33 PM UTC, comment #53:

I filled the bug on redhat bugzilla. Please have a look at the discussion. It is above my head:

https://bugzilla.redhat.com/show_bug.cgi?id=1572811

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sun 29 Apr 2018 10:06:10 PM UTC, comment #52:

Sorry, bug in the code of comment #50, second line in main() should allocate space for two options,

This does not change the rest of the observations, and the program still produces a segfault under gdb. As said before, the code is mainly the example from the JNI JVM-invocation documentation at

https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/invocation.html

Adrian <adaerr>
Sun 29 Apr 2018 09:57:03 PM UTC, comment #51:

Submit bug report to OpenJDK ?

The Java issues that were cited here consider the Segfault at JVM creation resolved since basically summer 2017, so the issues are closed. Notably
https://bugs.openjdk.java.net/browse/JDK-8183188
observed the bug on a Xeon X5675 processor, where indeed the simple program now executes flawlessly. Wouldn't it be worthwhile re-opening the bug or submitting a new issue on the fact that the test program still fails under gdb on certain processors (possibly for a different reason)? This seriously hampers debugging of programs using jvm-functionnality, as here, and also indicates that something is still rotten in the land of JNI. However it would probably be better if someone more knowledgeable than me in Java/JNI-matters and generally in programming reported the bug. :-)

[Never mind the question in my previous comment: I imagine that if it were easy to link octave with a non-openjdk JVM you'd have tried that.]

Adrian <adaerr>
Sun 29 Apr 2018 05:57:40 AM UTC, comment #50:

Just to share a 'negative' result, I tried to implement the workaround mentioned on

https://issues.apache.org/jira/browse/DAEMON-363

(linked to by the openjdk bug referenced by John in comment #12 and comment #17)

Namely to increase the Java stack size by the option "-Xss2m". I have no clue how the stack guard works and how it affects JVMs, nor how this option would circumvent the problem, but it seems openjdk does not implement the -Xss option anyway. Maybe it would be worth trying a JRE that implements it ?

Source code to be compiled with

Adrian <adaerr>
Sat 28 Apr 2018 10:52:01 PM UTC, comment #49:

Sorry -- it was cut-n-paste error with the previous code.

Here is the test code:

Running standalone I got:

octave:1> tjava
N = 1000
Elapsed time is 0.00399303 seconds.
N = 2000
Elapsed time is 0.00136709 seconds.
N = 3000
Elapsed time is 0.00197601 seconds.
N = 4000
Elapsed time is 0.00241995 seconds.
N = 5000
Elapsed time is 0.00188708 seconds.
N = 6000
Elapsed time is 0.00217009 seconds.
N = 7000
Elapsed time is 0.00244117 seconds.
N = 8000
Elapsed time is 0.00257707 seconds.
N = 9000
Killed

(It hanged after N=9000 and i had to kill -9 it)

running under gdb I got:

octave:1> tjava
[New Thread 0x7fffb2035700 (LWP 3230)]
[New Thread 0x7fffb1f33700 (LWP 3231)]
[New Thread 0x7fffb1c2f700 (LWP 3232)]
[New Thread 0x7fffb1b2d700 (LWP 3233)]
[New Thread 0x7fffb1a2b700 (LWP 3234)]
[New Thread 0x7fffb09a5700 (LWP 3235)]
[New Thread 0x7fffb08a3700 (LWP 3236)]
[New Thread 0x7fffb0529700 (LWP 3237)]
[New Thread 0x7fffb0427700 (LWP 3238)]
[New Thread 0x7fffb0326700 (LWP 3239)]
[New Thread 0x7fffb0225700 (LWP 3240)]
[New Thread 0x7fffb0124700 (LWP 3241)]
[New Thread 0x7fff91326700 (LWP 3242)]
[New Thread 0x7fff91225700 (LWP 3243)]
[New Thread 0x7fff90cfa700 (LWP 3244)]
[New Thread 0x7fff90bf9700 (LWP 3245)]
[New Thread 0x7fff90af8700 (LWP 3246)]
[New Thread 0x7fff68ae4700 (LWP 3247)]
N = 1000
Elapsed time is 0.00256085 seconds.
N = 2000
Elapsed time is 0.00192499 seconds.
N = 3000
Elapsed time is 0.00288796 seconds.
N = 4000
Elapsed time is 0.00139713 seconds.
N = 5000
Elapsed time is 0.0035069 seconds.
N = 6000

Thread 7 "QThread" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc0e51700 (LWP 3228)]
0x00007fff9b4ff09b in ?? ()
Missing separate debuginfos...

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 28 Apr 2018 09:15:13 PM UTC, comment #48:

So there is no crash but CPU hogging for the original code, right ? This is one of the two behaviours I observe for octave outside gdb. Does it (the original code, including graph.addNodes) also hog CPU when run under gdb ?

In your modified code you omit the method call

graph.addNodes(1:1000000);

which is the only problematic when running freely, and in gdb on systems where one gets past the "javaaddpath('.')". Not sure therfore what your testing of the imwrite timing is supposed to show: it's certainly not nice that imwrite or zeros cause termination instead of some recoverable out of memory error, but I have the impression it has nothing to do with the segfault that this bug-report is about.

Adrian <adaerr>
Sat 28 Apr 2018 08:19:59 PM UTC, comment #47:

It runs pretty much the same under gdb. I noticed that octave RES memory went to ~6GB at around N=20000 and the computer started to swap (it has "only" 8 GB of RAM).

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 28 Apr 2018 08:08:25 PM UTC, comment #46:

No crash on

vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU T9900 @ 3.06GHz
stepping : 10
microcode : 0xa0b
cpu MHz : 1595.627
cache size : 6144 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm pti tpr_shadow vnmi flexpriority dtherm ida
bugs : cpu_meltdown spectre_v1 spectre_v2
bogomips : 6116.57
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual

The original code hogged the CPU and I had to kill -9 it after ~30 minutes.
Then I modified the code as follows:

and I got:

octave:1> tjava
N = 1000
Elapsed time is 0.126127 seconds.
N = 2000
Elapsed time is 0.513371 seconds.
N = 3000
Elapsed time is 1.0897 seconds.
N = 4000
Elapsed time is 1.89885 seconds.
N = 5000
Elapsed time is 3.00135 seconds.
N = 6000
Elapsed time is 4.31079 seconds.
N = 7000
Elapsed time is 5.73707 seconds.
N = 8000
Elapsed time is 7.74923 seconds.
N = 9000
Elapsed time is 9.53925 seconds.
N = 10000
Elapsed time is 11.7024 seconds.
N = 11000
Elapsed time is 14.1665 seconds.
N = 12000
Elapsed time is 17.2648 seconds.
N = 13000
Elapsed time is 20.0761 seconds.
N = 14000
Elapsed time is 22.8917 seconds.
N = 15000
Elapsed time is 26.3692 seconds.
N = 16000
Elapsed time is 36.8092 seconds.
N = 17000
Elapsed time is 34.7174 seconds.
N = 18000
Elapsed time is 48.9227 seconds.
N = 19000
Elapsed time is 62.3376 seconds.
N = 20000
Elapsed time is 73.7005 seconds.
N = 21000
Elapsed time is 72.4562 seconds.
N = 22000
fatal: caught signal Terminated -- stopping myself...
attempting to save variables to 'octave-workspace'...
warning: save: unable to save java objects, skipping
warning: called from
_imwrite_ at line 215 column 3
imwrite at line 119 column 5
tjava at line 6 column 13
save to 'octave-workspace' complete

(I kill it with signal 15 at the end.)
Notice the time increase is not uniform.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 28 Apr 2018 05:44:37 PM UTC, comment #45:

I think it is Java bug in cpu probe code. We have seen this before
with ASAN
https://savannah.gnu.org/bugs/?52061
and that happened before Spectre/Meltdown brouhaha.

I am away from Xeon at the moment, but should be able to test on Core2duo laptop shortly. (The test program runs fine there.)

BTW AMD FX-8350 crashes as well.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 28 Apr 2018 02:30:07 PM UTC, comment #44:

I would also like to know the results of testing a system working with gdb with the OP's code (comment #43).

This may be a red herring, but is this related to Spectre and Meltdown and subsequent microcode patches by Intel?

Using 'cat /proc/cpuinfo' I find that my system is

It is an i7 and crashes, just as Adrian's i5 system crashes.

An additional field that I have never noticed in the output is

I see that there were also reports of HW instability with the first microcode update from Intel for Ubuntu (https://news.softpedia.com/news/canonical-pulls-intel-s-microcode-update-from-ubuntu-repos-due-to-hardware-issue-519494.shtml). Maybe something is still not complely correct.

Rik <rik5>
Project Administrator
Sat 28 Apr 2018 01:42:03 PM UTC, comment #43:

I can also confirm that the small test program crashes on an Intel i7-6600U.

In response to Rik's remark (comment #31), I upgraded gdb to 8.0 (the version currently shipped in Debian-experimental) on both the Xeon and the Intel i5 system: the behaviour remains the same (no error on Xeon, segfault on i5).

@Dmitri A. Sergatskov: I'd be curious like Rik (comment #31) to know whether you see the bug I reported, on your Xeon system where the small standalone program runs fine. That is, if octave (in gdb or not) crashes (and when) if given the commands

As Rik commented, whatever the result does not really tell us if this is related to the same bug or not, but it would give a better idea on the reproductibility of this particular test. Up to here Mike Miller seems to have observed the same bug (from the way I read his comment #3), but both Rik and John W. Eaton see a segfault much earlier under gdb.

Adrian <adaerr>
Sat 28 Apr 2018 12:14:58 AM UTC, comment #42:

Aha, nice observation Dmitri. Concerning my tests the program runs fine (under gdb) on a
model name : Intel(R) Xeon(R) CPU E5520
and crashes on a
model name : Intel(R) Core(TM) i5-2500K CPU

Adrian <adaerr>
Fri 27 Apr 2018 11:35:11 PM UTC, comment #41:

So it looks like it is CPU dependent -- it crashes on i7 and runs fine on an old Nehalem-based Xeon (W3530) and on Core2 duo.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 27 Apr 2018 11:12:42 PM UTC, comment #40:

Now I have the small program to crash. It was a bunch of updates including a new(ish) kernel...

The 530 line of that file is:

get_cpu_info_stub(&_cpuid_info);

This all looks awfully familiar.
https://savannah.gnu.org/bugs/?52061

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Fri 27 Apr 2018 10:07:58 PM UTC, comment #39:

I was finally able to get a different result out of gdb with the small test program.

I created a local .gdbinit file with the following

Now, I can compile and run the test program and the backtrace has a bit more meaning.

Of course, without gdb this runs just fine.

Rik <rik5>
Project Administrator
Fri 27 Apr 2018 09:56:51 PM UTC, comment #38:

@ John W. Eaton / comment 36: On another machine I see the small program behaving as you describe (ok without gdb, segfault in gdb). Now I have to find what distinguishes this computer from the one which runs the same binary just fine.

Adrian <adaerr>
Fri 27 Apr 2018 09:47:58 PM UTC, comment #37:

This is really puzzling, as the small program from comment #12 works well with or without gdb on my current system (octave 4.2.2, kernel 4.9 or 4.15), as I reported in comment #28. I even just upgraded gdb from 7.12-6 to 7.12-6-b1 to match the version of yours (this also draws in libpython3.6-minimal, libpython3.6-stdlib and libpython3.6), but still it does not produce any error and exits normally.

Adrian <adaerr>
Fri 27 Apr 2018 09:23:30 PM UTC, comment #36:

The thing that would shed more light is to understand why the smaller test program also crashes. Show me how to make that work reliably with or without gdb. Since it is much simpler, that should be a far easier task than trying to debug the crash in Octave.

John W. Eaton <jwe>
Project Administrator
Fri 27 Apr 2018 09:23:22 PM UTC, comment #35:

Sorry, comment #34 was missing its footnote, namely the packages affected by the upgrade from octave 4.2.1 to 4.2.2:

[*] The upgrade of octave from 4.2.1 to 4.2.2 implied an upgrade to the following packages. All other packages where held (in particular the kernel and openjdk packages). Citing the relevant part from /var/log/aptitude:

Adrian <adaerr>
Fri 27 Apr 2018 09:16:56 PM UTC, comment #34:

Checking back, the single out of four GNU/Linux systems where the
originally reported bug did not seem to occur (see comment #1) was in
fact not as up-to-date as the others: its kernel was at 4.14 and
octave at 4.2.1. It executed the instructions cited at the start of
comment #1 without error. I did unfortunately not think of running it
from within gdb to see if that changed things. Once I selectively
upgraded octave to 4.2.2[*], still running the 4.14 kernel, this
system also exhibited the bug reported in comment #1.

So with this correction, the bug seems to be totally reproducible with
octave 4.2.2 on four Debian systems (1 stretch, 3 buster). When
running octave under gdb, the bug is a SIGSEGV signal upon execution
of the fourth and last line of the following code:

When octave is run directly without attaching gdb (in GUI or CLI
mode), the bug also occurs. but on two systems leads to an application
crash with the signal 11 / SIGSEGV reported by the controlling
terminal, while on two other systems the interface freezes (the
octave-gui process ramps up to 100% CPU usage until killed).

So it seems that this bug could be (at least on Debian systems) a
regression in octave 4.2.2, not occuring on octave 4.0.3 and 4.2.1.

Let me know if some other test could help shed more light. I'll
upgrade gdb to the version used by Dmitri to see if that changes
things.

Adrian <adaerr>
Fri 27 Apr 2018 09:15:44 PM UTC, comment #33:

@John W. Eaton / comment #30: Yes, octave-cli under gdb fails the test (output below).

octave-cli however executes the instructions and returns to a prompt when not run under gdb, contrary to 'octave --no-gui'. Aren't those equivalent ?

Adrian <adaerr>
Fri 27 Apr 2018 09:11:09 PM UTC, comment #32:

When I run octave-cli from the Debian 4.2.2 on my system using gdb, I see this:

When I compile and run the test program that I showed in comment #12 on the same system, I see

That appears to be exactly the same crash, independent of Octave. Is the smaller test program incorrect? If so, how should it be fixed? If someone will show me how to make that program woek on my system with gdb, then maybe I could fix Octave as well.

Also, without gdb, the smaller test program appears to work properly.

Also, with valgrind --tool=memcheck --leak-check=full, I get the attached set of diagnostics.

(file #44046)

John W. Eaton <jwe>
Project Administrator
Fri 27 Apr 2018 08:58:16 PM UTC, comment #31:

I'm still not convinced that this isn't machine, kernel, and tool-dependent. Even if the stand-alone program works on your machine, and octave-4.2.2 does not, that could be explained by the linker getting lucky with where it has put pages in memory.

I notice that you are using gdb version 7.12-6, while Dmitri is running 8.1-11.

@Dmitri: You reported that the test program worked for you. Can you run a session of Octave under gdb and try the OP's test sequence

Rik <rik5>
Project Administrator
Fri 27 Apr 2018 08:55:29 PM UTC, comment #30:

Using 4.2.2, does this test also fail with octave-cli running under gdb?

John W. Eaton <jwe>
Project Administrator
Fri 27 Apr 2018 08:21:32 PM UTC, comment #29:

I tested several combinations of octave and GNU/Linux kernel versions.
The TLDR; of this comment is that 4.0.3 runs the test fine on kernels
4.9 and 4.15, whereas 4.2.2 produces a segment violation error on all
tested kernels (4.9, 4.14 and 4.15). Combined with the previous
comment #28 this really points to a specific octave bug.

I ran

gdb --args octave --no-gui --norc

then typed 'run' to launch octave

then pasted line by line:


Results

octave 4.0.3-3

GNU/Linux kernel 4.9.0-6-amd64 (Debian 4.9.82-1+deb9u3 (2018-03-02))
libc 2.24
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git

GNU/Linux kernel 4.15.0-3-amd64 (Debian 4.15.17-1 (2018-04-19))
libc 2.27
GNU gdb (Debian 7.12-6+b1) 7.12.0.20161007-git

No error/no segfault under gdb. Only error is from gdb openjdk plugin:

The other instruction execute fine.


octave 4.2.2

GNU/Linux kernel 4.9.0-6-amd64 (Debian 4.9.82-1+deb9u3 (2018-03-02))
libc 2.24
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git

under gdb:

=> I guess this reflects a dependency problem in Debian:
octave 4.2.2 should depend on openjdk-9

once openjdk-9 is installed the 'javaaddpath(".")' instruction indeed passes without error. A segfault is produced by the last instruction 'graph.addNodes(1:1000000);'. Full log:

This being the stock octave from Debian without symbols, the backtrace is useless.

p.s.

The 4.0.9 kernel was a fresh Debian stretch/stable install, where octave is at 4.0.3. Octave 4.2.2 and openjdk-9 where then added by adding the buster/testing sources to apt.

Conversely, the 4.14/4.15 kernel was a testing system with octave at 4.2.2, then downgraded to 4.0.3 to test this combination also.

Adrian <adaerr>
Fri 27 Apr 2018 10:49:56 AM UTC, comment #28:

Just as Dmitri in comment #20, the standalone JNI createJavaVM test program from comment #12 runs fine for me, in gdb or not, while on the same machine the original bug creates a segfault under gdb at the graph.addNodes instruction.

The standalone program works both with openjdk-8 and openjdk-9 (minor difference: error during plugin loading in gdb for openjdk-8, see below).

Here are the details on the test conditions:

GNU/Linux kernel 4.15.0-3-amd64 (Debian 4.15.17-1 (2018-04-19))
libc 2.27
GNU gdb (Debian 7.12-6+b1) 7.12.0.20161007-git

(side-note: the openjdk-8 test runs fine - with the same gdb.error for the plugin - also on a fresh Debian stable install with kernel 4.9; openjdk-9 is not packaged for stable yet)

openjdk-8:

openjdk-9:

Adrian <adaerr>
Wed 25 Apr 2018 03:56:41 PM UTC, comment #27:

Not sure if this is relevant, just FYI -- here is the Fedora's hotspot patch added to 9.0.4.11

Dmitri.
--

(file #44031)

Dmitri A. Sergatskov <dasergatskov>
Wed 25 Apr 2018 03:16:40 PM UTC, comment #26:

I agree. Whatever this is, it is probably upstream and shouldn't delay the 4.4.0 release.

Rik <rik5>
Project Administrator
Wed 25 Apr 2018 03:04:13 PM UTC, comment #25:

Unless someone can show a way to make the standalone program work with gdb or asan, then I have to assume this bug is not a bug in Octave and there's not much we can do about it.

John W. Eaton <jwe>
Project Administrator
Tue 24 Apr 2018 10:43:39 PM UTC, comment #24:

I ran the trivial program from comment #12 under valgrind, and it produces a lot of warnings about invalid writes. I can't say how much of it is real, but the log file is attached.

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 10:31:12 PM UTC, comment #23:

Linux td-119156 4.16.3-300.fc28.x86_64 #1 SMP Thu Apr 19 19:04:56 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

BTW, this is may be of some interest:

https://bugzilla.redhat.com/show_bug.cgi?id=1492175

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 24 Apr 2018 10:30:40 PM UTC, comment #22:

If this is a kernel problem, or a problem with the particular way that the address sanitizer and gdb work, would going back and trying valgrind be a solution? From what I remember, that runs the executable in simulation, which is slow, but it doesn't depend on setting magic bits on memory pages and having the kernel and CPU detect triggering conditions.

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 10:28:11 PM UTC, comment #21:

@Dmitri: what does running 'uname -a' from the shell report?

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 10:18:05 PM UTC, comment #20:

It seems the test program works for me -- no crash:

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 24 Apr 2018 10:10:28 PM UTC, comment #19:

I compiled the test case from comment #12 and it segfaults as well when using ASAN. If this is a kernel issue, it seems to affect multiple kernels.

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 09:58:26 PM UTC, comment #18:

After compiling with ASAN and trying "__java_init__" I get

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 09:45:38 PM UTC, comment #17:

The bug I referenced

https://bugs.openjdk.java.net/browse/JDK-8183188

points to

https://access.redhat.com/security/vulnerabilities/stackguard?sc_cid=70160000000gzltAAA&

I think ASAN uses stack guard pages also. Does gdb? Is that what causes the program to crash when running in those environments?

The OP mentioned that the problem appears on some systems an not others. Could that be due to differences in kernel versions?

John W. Eaton <jwe>
Project Administrator
Tue 24 Apr 2018 09:34:25 PM UTC, comment #16:

I just noticed the same thing

Given that the backtrace contains

maybe this is something related to this plugin interface.

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 09:32:11 PM UTC, comment #15:

Another possibility would be to compile with ASAN. I found that with the address sanitizer installed I got a quick segfault. I'm going to try building without debugging, but with ASAN, so that gdb is not involved.

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 09:28:33 PM UTC, comment #14:

?

John W. Eaton <jwe>
Project Administrator
Tue 24 Apr 2018 09:21:31 PM UTC, comment #13:

Is it possible that gdb is interacting with the JVM in some way to produce this segfault that is not a real symptom? On my system, there are gdb plugins under /usr/share/gdb for each of the versions of libjvm.so that are installed on my system:

Is there a way to prevent these plugins from being loaded to see whether the segfault occurs when gdb is not trying to load the JVM independently of Octave trying to load it?

Mike Miller <mtmiller>
Project Administrator
Tue 24 Apr 2018 09:14:42 PM UTC, comment #12:

It crashes for you inside the dlopen call? That's weird. For me, it crashes inside the call to the create vm function.

This problem appears to be independent of Octave.

Try the following program:

Note that this doesn't use dlopen/dlsym. On my system, I compile with

Your include file and library directories may be different.

Then if I run it with

it appears to succeed and prints the address of the JVM object created by JNI_CreateJavaVM.

However, if I run it with gdb, it segfaults in the same way as I am seeing with Octave:

I found this example here: https://bugs.openjdk.java.net/browse/JDK-8183188

John W. Eaton <jwe>
Project Administrator
Tue 24 Apr 2018 08:59:14 PM UTC, comment #11:

I stepped through all the way through and the issue occurs for me in the dlopen call at line 208 of oct-shlib.cc

The variable file holds

I tried changing the flags sent to dlopen, but that didn't help either. This bug may be upstream in libjvm itself.

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 08:12:26 PM UTC, comment #10:

Stepping through the code in initialize_jvm

I checked and both create_vm and get_vm are 0 so this branch is taken.

The code then jumps down to this

For a Linux system, the path ends in libjvm.so. Finally, the segfault occurs in the next line.

Maybe something bad happens when we call the constructor with a null value?

Or maybe when lib.search fails it leaves the object lib in a bad state?

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 08:02:44 PM UTC, comment #9:

As you will have discovered, you can use the internal function _java_init_ just to start the JVM. That might help with debugging.

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 07:05:57 PM UTC, comment #8:

I'm looking at this now. The failure is inside the JVM JNI_CreateJavaVM function. Maybe we are calling it incorrectly or not managing the memory for the arguments properly.

John W. Eaton <jwe>
Project Administrator
Tue 24 Apr 2018 06:54:07 PM UTC, comment #7:

There is another bug report about lint warnings during compilation of Octave's own java library.

Maybe it is a red herring, but unchecked, unsafe accesses to memory could certainly produce a segfault.

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 06:13:45 PM UTC, comment #6:

Under gdb I can confirm a SEGFAULT. I used

and then attached to the running process with 'gdb -p XXXX'. Within Octave, I ran

Note, I didn't even need to call javaObject.

The backtrace is pretty useless

This was with the release candidate 4.3.91.

Rik <rik5>
Project Administrator
Tue 24 Apr 2018 10:30:15 AM UTC, comment #5:

@Dimitri/comment #4: The ClassNotFoundException seems to indicate
that the class is not instantiated, maybe that is why you see no
segfault (running under gdb I suppose). Do you have Baz.class in
the pwd ?

@Mike/comment #3: Thanks for further narrowing things down. So if
I understand correctly loading the class alone already causes
some bad memory accesses, and the other instructions were just
necessary to reveal the memory corruption when run outside gdb ? That
would explain why this bug for a long time seemed almost non
reproducible to me, striking unexpectedly at different moments
when working interactively.

Just to be sure: you remove the octave-workspace before running
your minimal example ? I found that the presence of a workspace
resulting from a previous crash can change the point at which the
segmentation fault hits (my first attempt at a minimal example
did not crash anymore once I had removed the workspace).

Sorry for the confusion regarding the image package, somehow I
thought it provided imwrite, I should have checked.

Adrian <adaerr>
Tue 24 Apr 2018 12:05:39 AM UTC, comment #4:

hg id faed660f7222 seems to work on Fedora:

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 23 Apr 2018 11:36:58 PM UTC, comment #3:

I can reproduce this crash with Octave 4.2.2 and Octave 4.3.91, both are linked with OpenJDK 9, including the Debian package octave 4.2.2-2.

I have a self-compiled Octave 4.2.2 that is linked with OpenJDK 8. It does not immediately appear to crash in the same way. But when I run it under gdb, it does generate a segmentation fault.

The image package is not needed at all to reproduce this error. The imwrite function is part of Octave, no packages or init files are needed.

In fact the call to 'imwrite' is not needed at all either. When it is not present, octave appears to run and exit normally, but under gdb the segmentation fault can still be seen. The minimal example

still generates a segmentation fault when run under gdb.

Mike Miller <mtmiller>
Project Administrator
Mon 23 Apr 2018 11:00:25 PM UTC, comment #2:

Also forgot references to bugs #52501, #50644 and #50068 that seem to reflect the case of the "immune computer" described in the original submission. That "panic: Aborted" error is however different from the segmentation fault reported here.

Adrian <adaerr>
Mon 23 Apr 2018 10:45:17 PM UTC, comment #1:

Forgot to add that I did take care of moving the .octaverc out of the way. The segmentation fault remains when specifying --no-init-file --no-site-file

Adrian <adaerr>
Mon 23 Apr 2018 10:38:07 PM UTC, original submission:

Panic/segfault when calling Java method after forge:im[read|write]

The following minimal (non-)working example causes octave to
reproducibly panic and abort on two out of four (fairly up-to-date
Debian testing) GNU/Linux boxes I tested, and to freeze on a third
Computer.

Note that there is no error when either the "imwrite" instruction or
the last line is commented.

An additional file "Baz.java" with the content

needs to be compiled ("javac Baz.java") and the class file put in
octave's working directory. A pre-compiled class file for java >= 8 is
attached. Then the code at the top of this message is either pasted
into the command window of an octave GUI, or stored in a file
crashtest.m in the same directory as Baz.class and the run via "rm -f
octave-workspace; octave --no-gui crashtest.m". When run in a terminal
this way, I see this error message on two of four computers:

When pasted into a GUI, the message shown after the crash in the
terminal used to launch octave, is "octave exited with signal 11".

On a third Linux box both the command-line and the GUI do not crash
but start hogging a CPU (becoming unresponsive, including to Ctrl-C)
until manually terminated.

No idea for the moment why one out of A fourth Computers I tested is
immune to this bug in GUI mode: it executes the code and shows a
prompt, as expected. In text-mode octave does panic, but with a
slightly different message

).

Can anybody reproduce this ? I have no clue for the moment why the
behaviour differs among the tested machines.

Here's the list of octave-related packages installed on one of the two
segfaulting systems. For Java I tried openjdk versions 8 and 9 with
the same result.

p.s.
Despite the dependency of my example on the octave-forge 'image'
package I deliberately omit the '[octave-forge]' in the title because
the interaction with the java handling code, and the type of error,
seem to indicate some more general problem with memory corruption.
Feel free to add it back if it seems appropriate.

Adrian <adaerr>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44046:  java-example.log added by jwe (878KiB - text/x-log)
file #44031:  PStack-808293.patch added by dasergatskov (2KiB - text/x-patch)
file #44013:  val.log.lz added by rik5 (20KiB - application/x-lzip)
file #44000:  crashtest.m added by adaerr (121B - text/x-objcsrc)
file #44001:  Baz.java added by adaerr (91B - text/x-java)
file #44002:  Baz.class added by adaerr (240B - application/java-vm)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by hardy (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by adaerr (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-12-23 mtmiller Carbon-CopyRemoved 80942 => -
    2018-05-01 mtmiller Release4.3.91 => 4.4.0
    2018-04-27 jwe Attached File- => Added java-example.log, #44046
    2018-04-25 dasergatskov Attached File- => Added PStack-808293.patch, #44031
    2018-04-24 rik5 Attached File- => Added val.log.lz, #44013
    2018-04-24 rik5 Release4.2.2 => 4.3.91
    2018-04-23 mtmiller CategoryNone => Libraries
        StatusNone => Confirmed
    2018-04-23 adaerr Attached File- => Added crashtest.m, #44000
        Attached File- => Added Baz.java, #44001
        Attached File- => Added Baz.class, #44002

    Back to the top


    Powered by Savane 3.4