bugGNU Octave - Bugs: bug #51436, Segmentation fault when running...

 
 

bug #51436: Segmentation fault when running make check-local on 32-bit (with gcc7)

Submitted by:  Atri Bhattacharya <badshah400>
Submitted on:  Mon 10 Jul 2017 10:06:30 PM UTC  
 
Category: Configuration and Build SystemSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Segfault, Bus Error, etc.
Status: InvalidAssigned to: None
Originator Name: Open/Closed: Closed
Release: 4.2.1Operating System: GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Wed 12 Jul 2017 05:23:44 PM UTC, comment #20:

OK. Once again, many thanks for all the help.

Atri Bhattacharya <badshah400>
Wed 12 Jul 2017 05:17:33 PM UTC, comment #19:

I think 2 GB is a good estimate. I have had build failures from even the compiler not having enough memory on machines with only 1 GB, so I always provision at least 2 GB when building Octave on a VM.

It's up to you what you want to do with OpenBLAS. Building Octave with a specific link for OpenBLAS means that it will only ever run with OpenBLAS. Building Octave against the reference BLAS library means that you can drop in OpenBLAS later at runtime and Octave will use it. I always run configure with the "--with-blas=blas" option to ensure that that is the case. IMHO building and running the test suite with netlib BLAS is safer. If you distribute the binaries you can let the user decide which BLAS library to use.

Closing this bug report.

Mike Miller <mtmiller>
Project Administrator
Wed 12 Jul 2017 05:05:52 PM UTC, comment #18:

Right, I think I figured it out. Apparently setting the RAM requirement to 2 GB for obs machines on which octave is being built solves the issue! Probably 1 GB will do too, but I have not tried that yet. That is why on my VMs which always had at least 1 GB of physical memory, all tests were working out ok. Does this make sense?

Furthermore, I went ahead and disabled the openblas linking, because I got the impression from comment #2 that using openblas is not recommended. I haven't tested it with 2 GB memory and openblas linking allowed to see if that works. Should I?

Atri Bhattacharya <badshah400>
Wed 12 Jul 2017 03:37:37 PM UTC, comment #17:

I'm a bit confused at this point...

When you say,

All of them work just fine, and yet when building on the openSUSE build service the 32-bit builds crash every single time at make check.

do you mean that the make check fails when running on the openSUSE build service?

If so, what kernel version is running there? If it is one of the kernel versions with the bug, then I suspect that is the problem.

Also, can you take the binary that was built there, and run it on some other system that has a different version of the kernel that is not supposed to have the bug and see if it crashes there when doing _run_test_suite_? Also, I would verify that _run_test_suite_ demonstrates the problem on the build server, same as "make check".

John W. Eaton <jwe>
Project Administrator
Wed 12 Jul 2017 02:51:58 PM UTC, comment #16:

Tried over and over to see if any test crashes on my 32-bit VM but nada. All of them work just fine, and yet when building on the openSUSE build service the 32-bit builds crash every single time at make check. The only hope is to see if something from the build log pops out as the explanation. Thanks for all the help, folks!

Atri Bhattacharya <badshah400>
Tue 11 Jul 2017 06:35:19 PM UTC, comment #15:

Yeah that is likely. I am trying to replicate the exact VM setup. Will get back to you.

Atri Bhattacharya <badshah400>
Tue 11 Jul 2017 05:35:47 PM UTC, comment #14:

Which mean it should pass "make check" as well.

So your build environment on VM is probably different (different kernel versions or libraries) than the one on the build server.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Tue 11 Jul 2017 05:33:17 PM UTC, comment #13:

[In response to comment #11]
I see this (no crash):

Atri Bhattacharya <badshah400>
Tue 11 Jul 2017 05:29:21 PM UTC, comment #12:

Sorry, comment #11 seems to have not come out right.

octave:1> javaObject("java.lang.Object");
octave:2> a=javaObject("java.lang.Object");
octave:3> a
a =

<Java object: java.lang.Object>

So, it doesn't crash here.

Atri Bhattacharya <badshah400>
Tue 11 Jul 2017 05:29:09 PM UTC, comment #11:

Does it crash if you do (at octave prompt)
?

Dmitri.

Dmitri A. Sergatskov <dasergatskov>
Tue 11 Jul 2017 05:26:32 PM UTC, comment #10:

That's not the bug on my machine. Sorry for the delay in responding, but I installed a fresh 32-bit VM, then disabled the make check commands and built octave for a 32-bit system. Then launch

Atri Bhattacharya <badshah400>
Tue 11 Jul 2017 02:50:05 PM UTC, comment #9:

Yeah, this is almost definitely due to the kernel bug affecting Java.

I have a Debian 32-bit unstable VM. I have installed the octave package from the official repositories, linked with netlib BLAS and LAPACK, ATLAS and OpenBLAS are not even installed.

Test command is simply: octave --eval 'javaObject("java.lang.Object");'

Under linux 4.11.6-1, which is the most recently packaged version in Debian, there is no segfault.

Under linux 4.9.30-2, which is vulnerable to CVE-2017-1000364, there is no segfault.

Under linux 4.9.30-2+deb9u1, which contained the first fix attempt for CVE-2017-1000364, Octave segfaults.

Under linux 4.9.30-2+deb9u2, which reverted the previous and added a new fix, there is no segfault.

Please do confirm that you are able to run the test suite under a different linux version.

Mike Miller <mtmiller>
Project Administrator
Tue 11 Jul 2017 02:49:22 PM UTC, comment #8:

Removed openblas completely, in favour of lapack. Still triggers the segmentation fault during make check on 32-bit.

Atri Bhattacharya <badshah400>
Tue 11 Jul 2017 02:17:48 PM UTC, comment #7:

I am trying to build on kernel 4.11.8, which does not have this bug any more. Indeed, scilab and libreoffice which were seen to be crashing due to this kernel bug seems to be working fine on my virtual machines.

I am trying to build without linking to openblas to try out Dmitri's suggestion in comment #2.

Thanks for the quick responses.

Atri Bhattacharya <badshah400>
Tue 11 Jul 2017 12:47:02 PM UTC, comment #6:

Actually I think this may be a known bug that has to do with a kernel regression and the Java runtime.

See for example https://bugs.debian.org/865303 and https://bugs.launchpad.net/bugs/1699772. This bug apparently affects many user-space programs that embed or launch a Java VM running on certain versions of the 32-bit Linux kernel.

Mike Miller <mtmiller>
Project Administrator
Tue 11 Jul 2017 04:45:27 AM UTC, comment #5:

From what I could tell, when running 'make check', the segfault almost always occurred in a dgetrf_ function call in libopenblas.so. And only after all the tests in ov-java.cc-tst passed.

But when running run-octave from the command line it was possible to get it to crash immediately on any Java object or method call.

I forgot to link to the openSUSE build system, which is where this report seems to originate from: https://build.opensuse.org/package/show/science/octave.

Mike Miller <mtmiller>
Project Administrator
Tue 11 Jul 2017 03:49:33 AM UTC, comment #4:

Can you find where it crashes? It seems this would be independent of the GUI, so the easiest way to get a stack trace might be to use "./run-octave -g" and then run the test individually from the prompt. Something like

It still could be a problem with the way we initialize the JVM or whether we are properly saving/restoring CPU modes. As I recall, some JVM implementations like to set floating point rounding modes and such. Maybe there is something else we are missing there?

John W. Eaton <jwe>
Project Administrator
Tue 11 Jul 2017 03:41:44 AM UTC, comment #3:

Agree with Dmitri, please try building and running tests without OpenBLAS to see if that makes a difference.

Both Debian-based and Fedora-based distributions are building Octave 4.2.1 and running the test suite without problems. Both build with netlib BLAS and LAPACK, not OpenBLAS.

https://buildd.debian.org/status/fetch.php?pkg=octave&arch=i386&ver=4.2.1-2&stamp=1498828904&raw=0

https://kojipkgs.fedoraproject.org//packages/octave/4.2.1/3.fc27/data/logs/i686/build.log

FWIW I can reproduce this crash or a similar crash during the test suite by building against OpenBLAS on a 32-bit i686 VM. It seems to occur most often in the DGETRF function, which pushes the blame towards OpenBLAS. It also only seems to occur after the Java runtime is loaded. Both OpenBLAS and the JVM start a number of background threads when they are initialized.

For example this does not crash

but this does

And both of these crash regardless of the value of OMP_NUM_THREADS.

Mike Miller <mtmiller>
Project Administrator
Tue 11 Jul 2017 02:00:13 AM UTC, comment #2:

i686 is quite an exotic architecture now-days... Not many
people have it around now days, but nevertheless other distributions seems to build it sucessfully e.g.:

https://kojipkgs.fedoraproject.org//packages/octave/4.2.1/3.fc26/data/logs/i686/build.log

I would suspect blas libraries. I assume you are using eithe atlas or openblas. Try to rerun make check preloading a reference
blas library (or just recompile against reference blas):

(adjust the path to your blas library accordingly)

Sincerely,

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 10 Jul 2017 10:11:50 PM UTC, comment #1:

Just noting here that using the same config, builds on 64-bits proceed through the make check region with no problem. So this is a 32-bit-only issue.

Atri Bhattacharya <badshah400>
Mon 10 Jul 2017 10:06:30 PM UTC, original submission:

During building octave on 32-bit systems (on openSUSE), we see the following segmentation fault during running make check-local. The corresponding build log, trimmed to show only the make check section (full log attached), follows. Any suggestion to fix this would be highly appreciated. Many thanks!

Atri Bhattacharya <badshah400>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #41161:  _log32bit.tar.xz added by badshah400 (64KiB - application/x-xz - Full build log)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by dasergatskov (Posted a comment)
  • -unavailable- added by badshah400 (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 12 Jul 2017 05:17:33 PM UTCmtmillerStatusNone=>Invalid
      Open/ClosedOpen=>Closed
    Mon 10 Jul 2017 10:06:30 PM UTCbadshah400Attached File-=>Added _log32bit.tar.xz, #41161

    Back to the top


    Powered by Savane 3.1-cleanup1