bugGNU Octave - Bugs: bug #39000, Can't override BLAS XERBLA handler...

 
 

bug #39000: Can't override BLAS XERBLA handler on Windows

Submitted by:  Clemens Buchacher <drizzd>
Submitted on:  Wed 15 May 2013 04:07:23 PM UTC  
 
Category: LibrariesSeverity: 4 - Important
Priority: 7 - HighItem Group: Segfault, Bus Error, etc.
Status: Patch SubmittedAssigned to: None
Originator Name: Clemens BuchacherOpen/Closed: Open
Release: devOperating System: Microsoft Windows

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 26 Nov 2017 10:54:58 PM UTC, comment #43:

Apparently this bug report goes on and on. I fixed bug #52516 where a matrix of all zeros was returned when the matrix norm was NaN. I broke the test on matrix norm in to two parts so that a norm of NaN returns a NaN matrix and a norm of Inf returns a zeros matrix.

It would be best if we had some BIST tests for all of the corner case matrices which have caused problems in the past. Probably a file in the test/ directory.

Rik <rik5>
Project Administrator
Thu 23 Nov 2017 10:01:01 PM UTC, comment #42:

I found a few more instances in dMatrix.cc and fMatrix.cc where the new function norm1 could be used. I checked in the changes to stable here (http://hg.savannah.gnu.org/hgweb/octave/rev/117107eed917).

This bug can be closed again, or jwe can leave it open until MXE Octave supports installing Octave's own version of XERBLA.

Rik <rik5>
Project Administrator
Thu 23 Nov 2017 06:27:53 PM UTC, comment #41:

Indeed this patch solves the problem (checked on Win-7 machine).
Running __run_test_suite__ did not find any new error.

Avinoam Kalma <avinoam>
Project Member
Wed 22 Nov 2017 05:57:54 AM UTC, comment #40:

I've attached a patch against stable that works for me. I introduced a new local function norm1 which calculates the 1-norm, but preserves any NaN. The 1-norm was actually being calculated in quite a few places so it makes sense to abstract it to its own function. Please try the patch and review it as I would like to commit it before the 4.2.2 release.

(file #42457)

Rik <rik5>
Project Administrator
Wed 22 Nov 2017 05:19:27 AM UTC, comment #39:

bug #50007 contains a patch that extends max() for arrays to accept a nanflag. Maybe it could be used here.

Avinoam Kalma <avinoam>
Project Member
Tue 21 Nov 2017 08:36:52 PM UTC, comment #38:

In each of the the three bug reports the problem has essentially been operands with Inf or NaN that--through mldivide or mrdivide--end up calling particular Fortran routines in liboctave.

There are a number of ways in, but they only lead to a few routines in dMatrix.cc (and friends). So should something like my suggested code in comment #34 be implemented? Should it reside in place (3 instances in dMatrix.cc) or should a new local function in dMatrix.cc like norm1 be created?

More ambitiously, max() for arrays could be extended to accept a nanflag. There is a bug report about that somewhere, and Matlab has had this for a while. If this were done then the 1-norm would be quite simple to calculate without having to handroll any code.

Rik <rik5>
Project Administrator
Tue 21 Nov 2017 07:11:09 PM UTC, comment #37:

I guess it depends on what kinds of error messages and results you want from calls to BLAS and LAPACK functions for bad inputs.

I see a few different categories.

1. Incorrect parameters that shouldn't happen (we screw up a call, for example). We still don't want Octave to crash. So our version of xerbla should be called and throw an error.

2. Matrix and vector arguments with NaN or Inf values that don't make sense for functions that we call. Should we protect calls that we make and return results? This is a case-by-case thing. If we can identify and return "reasonable" results (NaNs, 0, Inf, whatever) instead of throwing an error, we probably should.

3. Matrix and vector arguments with NaN or Inf values that don't make sense for internal functions. Same as above, if we can prevent the call somehow, we probably should. But these will be harder to detect.

Since it's probably impossible to detect all possible cases, we really have to have our own version of xerbla that doesn't cause Octave to exit.

FWIW, it's clear that the BLAS authors intended for users to replace XERBLA with their own version. Back in the day, that was simple because you were building the library for yourself anyway. Now it seems absurd that there is no standard way for a user of the BLAS to provide a replacement for this function without modifying the library sources.

John W. Eaton <jwe>
Project Administrator
Tue 21 Nov 2017 06:53:56 PM UTC, comment #36:

@jwe: I support that approach. Unfortunately, Windows is difficult to develop for, at least when the code is originally written for a UNIX environment.

Octave can provide a vanilla version of itself with patched BLAS libraries that just works. If others want to experiment with their own systems and libraries, that is fine, but it isn't an Octave responlibility to get all combinations of HW/SW/Octave to work together.

Is there any code that could be removed if we assume the XERBLA handler can always be overwritten? I'm possibly thinking of code for this bug, bug #45577, and bug #46330. We've attempted many workarounds in Octave for bugs which reside in Fortran libraries, and maybe not all of that needs to be kept now.

Rik <rik5>
Project Administrator
Tue 21 Nov 2017 06:04:19 PM UTC, comment #35:

The real problem is that we don't seem to have a way to override the call to xerbla on windows systems.

Well, we do, at least for our own builds. We can modify any BLAS libraries that we build so that they don't have their own xerbla routine, or so that it can call a function we supply.

This doesn't fix the problem for everyone, or for people who want to replace the blas library with their own. Unless they also do the same.

I thought we just had to worry about bad values passed to the BLAS or LAPACK functions that we call directly, but that's not the case. So I don't see how it's possible to check all the calls that we make so that there is no possible way that a bad BLAS or LAPACK call can happen INTERNALLY to BLAS or LAPACK. We can't possibly know all the code paths that could happen.

I think I can fix our Windows build so that the reference BLAS and OpenBLAS do the right thing and also document the issue. That would be good enough for me.

John W. Eaton <jwe>
Project Administrator
Tue 21 Nov 2017 05:50:38 PM UTC, comment #34:

The problem is that the 1-norm of the matrix is being calculated. The code is

The 1-norm is the largest column sum of the absolute value of A, but NaN always fails in comparisons, so the fact that one column sum yields NaN is not enough to make the norm NaN.

This can be fixed by changing the norm to something else, like 2, but this implies a singular value decomposition and a lot, lot more computation.

Maybe this is worth it, but I also hate having to code to a specific operating system. If Windows has a problem then perhaps this section of code should be protected by #ifdefs so that only Windows users have to experience the slow down.

As a possible intermediate solution one could write a routine to calculate the 1-norm, but respect the presence of NaNs.

Possible code

Rik <rik5>
Project Administrator
Tue 21 Nov 2017 07:23:14 AM UTC, comment #33:

I confirm behaviour of comment #21 in linux, both 4.2.1 and dev, with blas, openblas, atlas.

Marco Caliari <caliari>
Project Member
Tue 21 Nov 2017 06:47:50 AM UTC, comment #32:

On current dev it crashes so fast that I do not even see any error messages or Windows pop-ups. (Remarkable as I know that behavior (= application window simply disappearing w/o a trace) only from Linux, Windows usually catches misbehaved programs and complains.) CLI or GUI makes no difference.

Win 7, 64bit, a01c210e4287

Philip Nienhuis <philipnienhuis>
Project Member
Tue 21 Nov 2017 05:19:34 AM UTC, comment #31:

The line

Causes crash in Windows, and the error

In Linux.

The crash is in line 2217 of dMatrix.cc. The check

Does not help.

A solution could something like

But:

  • Is there a better way to find if there is a NaN/Inf in the matrix?
  • Where should we locate this check?
Avinoam Kalma <avinoam>
Project Member
Sun 28 May 2017 06:18:06 PM UTC, comment #30:

I am reopening this bug since there is an example in bug #51092
that makes Octave crash in Windows:

The error message before the crash is:

Avinoam Kalma <avinoam>
Project Member
Thu 06 Oct 2016 03:59:02 PM UTC, comment #29:

I pushed this changeset:

http://hg.savannah.gnu.org/hgweb/octave/rev/5aa8f199e328

Closing this rerport as fixed. If more cases pop up, we need to fix them individually. I don't see any generally reliable way to replace xerbla with our own function.

John W. Eaton <jwe>
Project Administrator
Wed 05 Oct 2016 08:20:02 PM UTC, comment #28:

At least for the reference BLAS and LAPACK, I think that the only way that xerbla can be called is if we call a BLAS or LAPACK function incorrectly. If we can avoid doing that, then we shouldn't even need to override the default version of XERBLA.

It is still nice to replace that function on systems where it is possible (and really, it should be possible on all systems) so that we can avoid causing Octave to exit if there is an accidentally incorrect call to a BLAS or LAPACK function.

John W. Eaton <jwe>
Project Administrator
Wed 05 Oct 2016 07:48:18 PM UTC, comment #27:

Sorry, I meant:

Without this patch every one of the examples ends with a crash (on Windows).

Maybe there is a better solution, I do not know....

Avinoam Kalma <avinoam>
Project Member
Wed 05 Oct 2016 07:44:28 PM UTC, comment #26:

@Avinoam:
Your answer is a bit ambiguous to me, sorry.

"... ends with a crash" - with or without the patch ?

In my case patch file #38631 fixed the issue AFAICS.

Philip Nienhuis <philipnienhuis>
Project Member
Tue 04 Oct 2016 09:06:19 PM UTC, comment #25:

>> "does this patch really help the windows users?"


In Windows, every one of the examples ends with a crash.

Avinoam Kalma <avinoam>
Project Member
Tue 04 Oct 2016 09:00:04 PM UTC, comment #24:

Windows seems to be more important, but for Linux there is no regression with file #38631

So does this patch really help the windows users and should this patch go in the 4.2 release? Then it should happen very soon.

Kai Torben Ohlhus <siko1056>
Project Member
Sun 02 Oct 2016 03:52:23 PM UTC, comment #23:

That latest patch (file #38631) seems to work fine in a --enable-windows-64 cross build:

I'm no C++ style expert but it looks good enough to me.

Should the "[ inf 2345 ] / 5" case be included as a test to be able to detect regressions?

Philip Nienhuis <philipnienhuis>
Project Member
Fri 30 Sep 2016 04:17:39 PM UTC, comment #22:

Please review the attached patch

(file #38631)

Avinoam Kalma <avinoam>
Project Member
Tue 30 Aug 2016 06:41:27 AM UTC, comment #21:

@Avinoam: just nasty because I find it tedious to criticize coding style, if the patch itself is good ;-) sorry a bit off topic

Kai Torben Ohlhus <siko1056>
Project Member
Mon 29 Aug 2016 07:38:39 PM UTC, comment #20:

@Kai: Thanks for your remarks (Why nasty?)

Actually, this patch does solve some other bug reports, like bug #46330, (test/bug-46330.tst can be tested on Windows too).

About the coding stile - I'll try to improve the patch. Thanks :-)

Avinoam Kalma <avinoam>
Project Member
Mon 29 Aug 2016 08:37:52 AM UTC, comment #19:

I have tested the patch of comment #18 on Linux without problems. Calls like in comment #17 result in a nice warning, like Matlab does.

Before:

After:

But as said in comment #17, only one of many potential cases is fixed by this patch.

Another nasty remark is the code formatting. Please do not introduce multiple empty lines and lines with only white space characters. See http://wiki.octave.org/C%2B%2B_style_guide.

Kai Torben Ohlhus <siko1056>
Project Member
Mon 29 Aug 2016 04:23:39 AM UTC, comment #18:

Another patch included. Please review.

This patch calculates the matrix norm, and if it is Inf or NaN,
the result is set to zero, rcon is also set to zero, and a singular matrix warning is shown. I have used
octave::warn_singular_matrix (), but maybe another warning should be used.

(file #38375)

Avinoam Kalma <avinoam>
Project Member
Tue 23 Aug 2016 09:51:49 AM UTC, comment #17:

Checking the example

The error (and the crash after it) is produced in calling

(line 2130 in liboctave/array/dMatrix.cc).

A workaround can be something like:

Checking the matrix norm (like in http://hg.savannah.gnu.org/hgweb/octave/rev/bf5fbf347aaf), and avoiding calling dgelsd in case of problematic norn.
But this changeset does not set an error or warning, as it should.

Such modification should also apply to other *Matrix.cc files.

Avinoam Kalma <avinoam>
Project Member
Fri 29 Jul 2016 03:22:57 PM UTC, comment #16:

Unfortunately, not all BLAS libraries make it easy to override the default error handler even though that was the original intent of having xerbla...

I suppose we could fix this for our mxe-octave builds for openblas and the reference blas. But that doesn't solve the problem generally. To do that requires cooperation from the people who are writing and packaging BLAS libraries.

Also, as I understand it, XERBLA should only be invoked when a BLAS or LAPACK routine is called incorrectly and we really shouldn't be making calls with invalid parameters.

For the example in comment 13, isn't that just a variation of bug #46330? But even though that bug is closed, I don't think it is really fixed yet. I'll comment there as well.

John W. Eaton <jwe>
Project Administrator
Thu 05 Feb 2015 01:25:19 AM UTC, comment #15:

I just got a Windows VM set up, same segfault here.

Mike Miller <mtmiller>
Project Administrator
Thu 05 Feb 2015 01:16:43 AM UTC, comment #14:

The simpler test case

produces the error you cited

The Windows OS then catches the error and says that the application tried to read memory at 0x010 (clearly a segfault) and then the program exits.

Re-titling the bug to reflect the real issue of being unable to override the XERBLA handler on Windows.

Rik <rik5>
Project Administrator
Wed 04 Feb 2015 03:27:08 PM UTC, comment #13:

Yes, this particular test case looks solved.

However, you're not seeing the xerbla error message that's supposed to show up if this test case were still triggering the underlying problem.

I closed bug #36211 as a duplicate of this one. The simpler test case in that report still shows the xerbla message on Linux for me, what does

do for you on Windows? On Linux, it prints "On entry to DLASCL parameter number 4 had an illegal value" and then catches the error and returns to the Octave prompt. On Windows, last time I checked, this would cause Octave to abort, which is what this bug is really about.

Mike Miller <mtmiller>
Project Administrator
Wed 04 Feb 2015 04:15:30 AM UTC, comment #12:

@Mike: I used a very recent tip (2/3/15) and I don't experience any problem with the test data. I used the NSIS installer route and tried both reference BLAS and OpenBLAS. I think whatever gremlin was there has been excised. If you're fine with that level of testing you can close the report.

Rik <rik5>
Project Administrator
Wed 04 Feb 2015 01:53:23 AM UTC, comment #11:

@Mike: I have a Windows XP VM. I'll test tonight. Going through the MXE compile though is going to take 1-2 hours.

Rik <rik5>
Project Administrator
Fri 30 Jan 2015 04:06:00 PM UTC, comment #10:

Is the BLAS/LAPACK error handler still not being correctly overridden in the current MXE build? I was able to reproduce this a year ago, but I don't have my WinXP VM at the moment to test a current build.

Mike Miller <mtmiller>
Project Administrator
Sat 11 Jan 2014 08:27:02 PM UTC, comment #9:

I get the same error as the original report with an MXE build of 3.8.0 in a Windows XP VM.

Mike Miller <mtmiller>
Project Administrator
Sat 18 May 2013 08:25:07 AM UTC, comment #8:

The system is a Core i5-2540M @ 2.60 GHz with Windows 7 64-bit.

I have changed MXE to build gcc 4.8.0 as a cross-compiler, which includes gfortran. So that may also make a difference in terms of floating point arithmetic.

Clemens Buchacher <drizzd>
Fri 17 May 2013 03:17:56 PM UTC, comment #7:

I'm not able to reproduce the problem with the MXE build either.

I do see a message about the matrix being singular, but no error from lapack.

What hardware are you using? I'm running in a virtual box system with Windows 7 installed.

In any case, I guess we should fix the MXE build so that we can overrride XERBLA with our own version so that it doesn't exit.

John W. Eaton <jwe>
Project Administrator
Fri 17 May 2013 01:12:12 PM UTC, comment #6:

I also cannot reproduce on Linux with octave 3.7.3+, lapack 3.4.2.

Clemens Buchacher <drizzd>
Fri 17 May 2013 01:01:00 PM UTC, comment #5:

I did not modify the lapack version in my MXE build. The version is 3.4.2.

In SRC/dlasd8.f there is this code:

CALL DLASD4( K, J, DSIGMA, Z, WORK( IWK1 ), RHO, D( J ),
$ WORK( IWK2 ), INFO )
*

  • If the root finder fails, the computation is terminated.

*
IF( INFO.NE.0 ) THEN
CALL XERBLA( 'DLASD4', -INFO )
RETURN
END IF

And XERBLA seems to print the error message that I am getting. DLASD8 in turn gets called via functions such as zgelsd.

Clemens Buchacher <drizzd>
Wed 15 May 2013 10:17:23 PM UTC, comment #4:

I'm unable to duplicate the problem. Are you using the reference version fo lapack that is included in the MXE-octave build, or replacing it with some other version?

The version of DLASD4 that is used by my MXE-octave build does not appear to do any argument checking. So I don't see where that message is coming from.

Also, Octave doesn't call DLASD4 directly, so if there is an invalid call, it seems that it would be a bug in LAPACK.

John W. Eaton <jwe>
Project Administrator
Wed 15 May 2013 04:47:01 PM UTC, comment #3:

That's because you can't do it in Win32 (at least not in the way you do it under Linux/UNIX). The functions in the BLAS DLL will always resolve xerbla to the symbol present in the BLAS DLL (I believe this is done at link time, when the DLL is created).

Michael Goffioul <goffioul>
Project Member
Wed 15 May 2013 04:32:17 PM UTC, comment #2:

This sounds like the same symptom as bug #38575. The xerbla error handler in liboctave is not overriding the xerbla error handler in the BLAS/LAPACK libraries.

Mike Miller <mtmiller>
Project Administrator
Wed 15 May 2013 04:09:52 PM UTC, comment #1:

Just to be clear. This crashes Octave, closing the command line window or returning back the cmd shell.

Clemens Buchacher <drizzd>
Wed 15 May 2013 04:07:23 PM UTC, original submission:

With my MXE build of the development branch I get this crash with the attached file.

% crashme.m
xy = load('xy.txt');
x = xy.x;
y = xy.y;
z = x\y;

> crashme

warning: matrix singular to machine precision, rcond = 1.57068e-020

    • On entry to DLASD4 parameter number -1 had an illegal value

Where:

> size(xy.x)

ans =

350 350

> size(xy.y)

ans =

350 1

Does not happen with the 3.6.4 mingw build from the wiki:

> crashme

warning: matrix singular to machine precision, rcond = 8.43218e-021

Clemens Buchacher <drizzd>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #42457:  bug39000.cset added by rik5 (15KiB - application/octet-stream)
file #38631:  bug39000_v2.diff added by avinoam (7KiB - application/octet-stream)
file #38375:  bug39000.patch added by avinoam (2KiB - application/octet-stream)
file #28088:  xy.dat.tar.gz added by drizzd (12KiB - application/x-gzip)

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -unavailable- added by caliari (Posted a comment)
  • -unavailable- added by avinoam
  • -unavailable- added by siko1056 (Posted a comment)
  • -unavailable- added by avinoam
  • -unavailable- added by avinoam (Posted a comment)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by goffioul (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by drizzd (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 24 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 22 Nov 2017 05:57:54 AM UTCrik5Attached File-=>Added bug39000.cset, #42457
      StatusConfirmed=>Patch Submitted
    Tue 21 Nov 2017 05:50:38 PM UTCrik5StatusNone=>Confirmed
    Tue 21 Nov 2017 05:19:34 AM UTCavinoamCarbon-Copy-=>Added -unavailable-
    Sun 28 May 2017 06:18:06 PM UTCavinoamStatusFixed=>None
      Open/ClosedClosed=>Open
    Thu 06 Oct 2016 03:59:02 PM UTCjweStatusPatch Submitted=>Fixed
      Open/ClosedOpen=>Closed
    Tue 04 Oct 2016 09:00:04 PM UTCsiko1056Priority5 - Normal=>7 - High
    Fri 30 Sep 2016 04:17:39 PM UTCavinoamAttached File-=>Added bug39000_v2.diff, #38631
    Mon 29 Aug 2016 04:23:39 AM UTCavinoamAttached File-=>Added bug39000.patch, #38375
      StatusConfirmed=>Patch Submitted
      Carbon-Copy-=>Added siko1056
    Sat 12 Sep 2015 09:54:22 PM UTCmtmillerDependencies-=>bugs #45484 is dependent
    Tue 01 Sep 2015 03:25:35 PM UTCrik5Dependencies-=>bugs #45849 is dependent
    Thu 05 Feb 2015 01:25:19 AM UTCmtmillerSeverity3 - Normal=>4 - Important
    Thu 05 Feb 2015 01:16:43 AM UTCrik5StatusFixed=>Confirmed
      Summarymldivide crashes with DLASD4 parameter error=>Can't override BLAS XERBLA handler on Windows
    Wed 04 Feb 2015 04:15:30 AM UTCrik5StatusConfirmed=>Fixed
    Fri 30 Jan 2015 04:04:18 PM UTCmtmillerDependencies-=>bugs #43610 is dependent
    Sat 11 Jan 2014 08:31:42 PM UTCmtmillerDependencies-=>bugs #36211 is dependent
    Sat 11 Jan 2014 08:27:02 PM UTCmtmillerCategoryNone=>Libraries
      StatusNone=>Confirmed
    Wed 15 May 2013 04:07:23 PM UTCdrizzdAttached File-=>Added xy.dat.tar.gz, #28088

    Back to the top


    Powered by Savane 3.1-cleanup1