bugGNU Octave - Bugs: bug #61246, matrix inversions give wrong...

 
 

bug #61246: matrix inversions give wrong results (inv, lu, mldivide)

Submitted by:  Joohwa Lee <starry>
Submitted on:  Thu 30 Sep 2021 05:04:09 AM UTC  
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Need Info Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  6.3.0 Operating System:  Microsoft Windows

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Wed 27 Oct 2021 06:38:48 AM UTC, comment #28: 

@Joohwa Lee: A release candidate for Octave 6.4.0 was made that was built on updated hardware. Could you please check if these operations still fail on your affected machines?

You can download the release candidate from here:
https://alpha.gnu.org/gnu/octave/?C=M;O=D

Markus Mützel <mmuetzel>
Project Member
Wed 20 Oct 2021 09:30:00 PM UTC, comment #27: 

I just checked octave-6.3.90-w64-installer.exe from alpha.gnu.org
and I do not see any problems on i9-9880 (Win10).

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Mon 11 Oct 2021 12:46:04 PM UTC, comment #26: 

Given that OpenBLAS consists mostly of C and assembler code, dropping it in as a replacement for "libblas.dll" in the Octave installation might just work.

However, since 0.3.13 built on octave.space seems to work but 0.3.13 built on jwe's build system does not, I don't know how much we would learn from that test...

Markus Mützel <mmuetzel>
Project Member
Sat 09 Oct 2021 05:37:02 PM UTC, comment #25: 

Openblas 0.3.18 has been just released.

I am reading https://www.openblas.net/Changelog.txt and there are quite a few fixes for bugs in AVX512 (Xeon(R) W-2133 has AVX512).

I do not know if it is possible to drop-in replace openblas dll
in the octave distribution with the "official" one from https://www.openblas.net/ to check if that fixes the bug?

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 09 Oct 2021 05:22:06 PM UTC, comment #24: 

My mistake, it is openblas-openmp. And OPM rather than MPI
later on.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Sat 09 Oct 2021 05:15:11 PM UTC, comment #23: 

@Dmitri: I'm not sure what openblas-mpi is. Is it the same or similar to openblas-openmp?

If I understand their documentation correctly, they advice to not use multithreaded OpenBLAS in a multithreaded application iff OpenBLAS is called from multiple threads of that application. Alternatively, they advice to use OpenMP for multithreading the main application and OpenBLAS to be able to set a reasonable limit for the total number of threads.

Afaict, that doesn't apply to us because we use OpenBLAS only in the interpreter thread.

I still don't understand the difference in results between the official installer and the builds from octave.space. If it is a bug in OpenBLAS, why does it only manifest depending on the build system?

It's probably best if jwe could build a release candidate and affected people could check if they still see the errors.

Markus Mützel <mmuetzel>
Project Member
Fri 08 Oct 2021 03:48:08 PM UTC, comment #22: 

I saw the problem not only on windows but on linux as well
(though not at the moment with openblas 0.3.17-2).
Fedora's buildbot is using serial openblas, so there is no visible problem there. So I do not think it is a problem with openblas build on Windows, most likely a bug in openblas.
I read somewhere (may be in the openblas docs) that one should not use threaded openblas library in multithreaded application, but
rather an MPI interface. Fedora currently uses "flixiblas"
wrapper library for blas that can be switched dynamically to different libs. By default it set to openblas-mpi. Openblas-mpi does not show this problem.

WRT comment #17 I would like to see more details. Disabling HT in bios may not be enough, what is important is to make sure that number of threads is reduced and does not exceed the number of CPUs.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 08 Oct 2021 03:07:11 PM UTC, comment #21: 

There is an un-applied patch in bug #61208 that would probably set the number of OpenBLAS threads to the actual number of physical cores. But I'm unsure if it will work before Windows 7.

Octave 7 will be the first version to require Windows 7 or newer. So, we should probably wait with that change until after the last Octave 6.x release.

Limiting the number of OpenBLAS threads didn't help for Joohwa Lee anyway. See comment #13.
So, the bugs that Dmitri and Joohwa are seeing might not be the same.
But IIUC, Dmitri didn't need to limit the number of OpenBLAS threads with a build from octave.space. The "only" difference was that the performance was worse when the threads weren't limited.
With Octave from the "official" installer, he got wrong result when he didn't limit the number of threads...

I also don't really understand the OpenBLAS build system very well. From what I gather, code specific to multiple target CPUs is built when we set DYNAMIC_ARCH=1. Additionally, there seems to be some common "base" code that is shared between all architectures. The minimum architecture that is required to run that common code can be selected with "TARGET=".

My understanding of that might be flawed. From what I gather, we are doing it right. But obviously, we are missing something...

Markus Mützel <mmuetzel>
Project Member
Fri 08 Oct 2021 02:40:34 PM UTC, comment #20: 

I don't really understand the openblas build system but even though we set TARGET=PRESCOTT I see the following in the log file:

sed -n 's/.*\(-DTS=[^ ]*\) .*/\1/p' openblas  | sort | uniq -c | sort -n
      3 -DTS=
    471 -DTS=_BARCELONA
    471 -DTS=_BOBCAT
    471 -DTS=_CORE2
    471 -DTS=_DUNNINGTON
    471 -DTS=_NANO
    471 -DTS=_OPTERON
    471 -DTS=_OPTERON_SSE3
    471 -DTS=_PENRYN
    471 -DTS=_PRESCOTT
    473 -DTS=_BULLDOZER
    473 -DTS=_EXCAVATOR
    473 -DTS=_PILEDRIVER
    473 -DTS=_STEAMROLLER
    475 -DTS=_ATOM
    475 -DTS=_COOPERLAKE
    475 -DTS=_HASWELL
    475 -DTS=_NEHALEM
    475 -DTS=_SANDYBRIDGE
    475 -DTS=_SKYLAKEX
    475 -DTS=_ZEN

so it looks like the sources are built for a variety of CPUs.

If the problem is solved by setting max number of threads, then I believe we already made that change in the Windows startup script, correct?

I can still make a test release if you think that's the best option.

John W. Eaton <jwe>
Project Administrator
Fri 08 Oct 2021 02:25:24 PM UTC, comment #19: 

AFAIK "prescott" was the first 64-bit intel cpu.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 08 Oct 2021 02:02:38 PM UTC, comment #18: 

Yes, I can check pre-release. Note that I hve problem with some
ocve.space binaries as well.
What fixes it for me is setting number of openblas threads to no more than the the number of actual CPU cores.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 08 Oct 2021 01:53:30 PM UTC, comment #17: 

Afaict, the buildbots on octave.space are using the same build rules you were using on your AMD FX 8350 systems when Octave 6.3 was released. Yet still, OpenBLAS seems to be working correctly if people are using the builds from octave.space. But they seem to get wrong results if they use the installer that was built on your system.

We had an issue in the past where somehow the processor architecture of the build system influenced the built binaries. We presumably fixed that issue by setting "DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 TARGET=PRESCOTT" when building OpenBLAS.
Maybe something similar is still happening?
Maybe "TARGET=PRESCOTT" wasn't a good choice?
See also: https://github.com/xianyi/OpenBLAS#support-for-multiple-targets-in-a-single-library

The error seems to depend on the processor of the host system. It seems to be working for me no matter which version of Octave I install. Both the "official" installer and builds from octave.space work fine for me (Intel(R) Core(TM) i5-4690 CPU @ 3.50GHz).

You proposed on discourse to build a release candidate:
https://octave.discourse.group/t/start-release-process-for-octave-6-4-0/1652/19

@Dmitri and Joohwa: You seem to have access to systems that are affected by this bug. Would you be willing to check if the error still occurs for you with a build from jwe's new build system?

Markus Mützel <mmuetzel>
Project Member
Fri 08 Oct 2021 01:01:32 PM UTC, comment #16: 

Yes, the 6.3.0 Windows releases were built on AMD FX 8350 systems and now I have AMD Ryzen 9 5950X systems to use for the 6.4.0 release.  But I don't know whether that should make a difference.  As far as I can tell, the openblas build rules haven't changed in the mxe-octave repo.

John W. Eaton <jwe>
Project Administrator
Fri 08 Oct 2021 07:36:28 AM UTC, comment #15: 

I was thinking that we updated OpenBLAS in MXE Octave since the release of Octave 6.3.0. But looking back on the repository log, that update happened before the release of Octave 6.3.0.

If the error occurs with the "official" installer built by jwe on his systems, but it works correctly with the installers built on octave.space, it might be that this is again a case where the issue depends on the  processor on the build system vs. host system.

IIUC, jwe upgraded to newer systems. Maybe that issue won't occur with the newer build systems. But as long as we don't understand the actual cause, it's hard to predict.

Ideally, we would get our build rules for OpenBLAS correct such that the resulting binaries don't depend on the processor used by the build system.

Markus Mützel <mmuetzel>
Project Member
Thu 07 Oct 2021 05:15:35 PM UTC, comment #14: 

This does not appear to be an Octave bug per se, rather it is an issue manifested for certain combinations of hardware, software (OpenBLAS vs. Reference BLAS), and operating system (seems to be only MS Windows).

Is that a correct statement or has this been fixed either for Octave 6.4 or Octave 7.0?

Rik <rik5>
Project Administrator
Sun 03 Oct 2021 03:48:57 AM UTC, comment #13: 

I tested another computer by Dell

  Model: Dell Precision 5820
  Specs: Intel(R) Xeon(R) W-2133 CPU @ 3.60GHz   3.60 GHz

1. Version 6.3.0

   OpenBLAS      + Hyperthreading on:   wrong  results
   OpenBLAS      + Hyperthreading off:  wrong  results
   ReferenceBLAS + Hyperthreading off: correct results

2. Version 6.3.2 (octave-2021-10-02-00-22-w64-installer)
   OpenBLAS      + Hyperthreading off: correct results
   ReferenceBLAS + Hyperthreading off: correct results

Conclusions
   Hyperthreading: no effects
   Version 6.3.0 vs. 6.3.1: effective

Joohwa Lee <starry>
Fri 01 Oct 2021 09:37:05 AM UTC, comment #12: 

comment #10:

> Could you show the LU test case that does not work?
> On Xeon or both?
>
> Dmitri.
> --
>


I was wrong.
In Ver 6.3.1, "lu" works fine.
I was not aware of permutation ([L,U,P]=lu(...))

Joohwa Lee <starry>
Fri 01 Oct 2021 09:17:14 AM UTC, comment #11: 
Dmitri A. Sergatskov <dasergatskov>
Fri 01 Oct 2021 09:09:59 AM UTC, comment #10: 

Could you show the LU test case that does not work?
On Xeon or both?

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Fri 01 Oct 2021 08:32:37 AM UTC, comment #9: 

Additional evaluation showed that

In Open BLAS, "lu" does not work in Ver. 6.3.1
In Reference BLAS, lu works fine in Ver. 6.3.0

comment #8:

> I have tested several Octave builds on three computer systems.
>
> 1. Intel(R) Xeon(R) Gold 5118 CPU @ 2.30 GHz 2.29 GHz (Windows 10 Pro 21H1): Workstations (PC1, PC2)
>
> 2. Intel(R) Core(TM) i5-8250U CPU @ 1.60 GHz 1.80 GHz (Windows 10 Pro 21H1): Notebook computer (PC3)
>
>
> Results:
>    PC1: Ver. 6.3.0 (Open BLAS)      gives wrong   results
>    PC1: Ver. 6.3.0 (Reference BLAS) gives correct results
>    PC2: Ver. 6.3.0 (Open BLAS)      gives wrong   results
>    PC2: Ver. 6.3.1 (Open BLAS)      gives correct results
>    PC3: Ver. 6.3.0 (Open BLAS)      gives correct results
>
> I concluded that Reference BLAS or 6.3.1 will help.
> Thanks all.

Joohwa Lee <starry>
Fri 01 Oct 2021 04:23:28 AM UTC, comment #8: 

I have tested several Octave builds on three computer systems.

1. Intel(R) Xeon(R) Gold 5118 CPU @ 2.30 GHz 2.29 GHz (Windows 10 Pro 21H1): Workstations (PC1, PC2)

2. Intel(R) Core(TM) i5-8250U CPU @ 1.60 GHz 1.80 GHz (Windows 10 Pro 21H1): Notebook computer (PC3)

Results:
   PC1: Ver. 6.3.0 (Open BLAS)      gives wrong   results
   PC1: Ver. 6.3.0 (Reference BLAS) gives correct results
   PC2: Ver. 6.3.0 (Open BLAS)      gives wrong   results
   PC2: Ver. 6.3.1 (Open BLAS)      gives correct results
   PC3: Ver. 6.3.0 (Open BLAS)      gives correct results

I concluded that Reference BLAS or 6.3.1 will help.
Thanks all.

Joohwa Lee <starry>
Thu 30 Sep 2021 12:15:13 PM UTC, comment #7: 

Additionally: We've updated OpenBLAS in our build system for Windows since the release of Octave 6.3.0. Could you please check if you still see these large deviations with a nightly build that you can download from octave.space?
https://buildbot.octave.space/#/download

Markus Mützel <mmuetzel>
Project Member
Thu 30 Sep 2021 12:14:49 PM UTC, comment #6: 

I suspect it might be again issue with hyperthreading.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 30 Sep 2021 12:11:18 PM UTC, comment #5: 

For me on Windows 10 with Octave 6.3.0 (32bit) using reference BLAS:

>> version -blas
ans = unknown or reference BLAS
>> n = 7;
>> randn('seed',1);
>> a = randn(n);
>> x = (1:n)';
>> b = a * x;
>> r = x - a \ b
r =

   1.7764e-15
   3.5527e-15
  -1.3323e-15
  -3.5527e-15
  -8.8818e-16
   2.6645e-15
  -2.6645e-15

And the same with OpenBLAS:

>> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.13 NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Nehalem MAX_THREADS=24)
>> n = 7;
>> randn('seed',1);
>> a = randn(n);
>> x = (1:n)';
>> b = a * x;
>> r = x - a \ b
r =

   4.6629e-15
   1.0214e-14
   2.6645e-15
  -7.9936e-15
  -1.7764e-15
   1.7764e-15
  -5.3291e-15

I don't see the large deviations you are using.

My CPU:

>> system('wmic CPU GET Name')
Name
Intel(R) Core(TM) i5-4690 CPU @ 3.50GHz

Markus Mützel <mmuetzel>
Project Member
Thu 30 Sep 2021 12:10:09 PM UTC, comment #4: 

What CPU do you have? Can you switch Openblas to the reference blas in 6.3.0 and repeat the test?

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 30 Sep 2021 11:12:33 AM UTC, comment #3: 

I installed 5.2.0 version and ran the same code.
I got correct results.

[code]
n = 7;
randn('seed',1);
a = randn(n);
x = (1:n)';
b = a * x;
r = x - a \ b;

[r in Ver. 5.2.0]

   3.1086e-15
   6.6613e-15
   2.6645e-15
  -4.4409e-15
  -1.7764e-15
   8.8818e-16
  -3.5527e-15

[r in Ver. 6.3.0]

  -0.1859
   4.6442
   3.4121
  -1.8404
  -1.4111
  -0.4142
  -1.2549

Joohwa Lee <starry>
Thu 30 Sep 2021 05:55:43 AM UTC, comment #2: 

These are the limitations of practical implementations of floating-point numerical linear algebra. Yes, the problem a\b would have an exact solution, namely x, but for practical efficiency the matrix computations are done in finite precision, and so the result is not exact, but off by some amount (which however is very small you have to admit). For better precision, you would have to do the computations with higher internal precision, which is possible, but would be vastly less efficient. There is software to do that, but octave rather has those users who value efficiency over analytical exactness as target group.

Michael Leitner <mleitner>
Thu 30 Sep 2021 05:09:56 AM UTC, comment #1: 

I tried to solve matrix system equations with several methods, i.e., inv, mldivide, lu. They all give wrong results

n = 7;
a = randn(n);
x = (1:n)';
b = a * x;
r = x - a\b;

The last line can be changed to r = x - inv(a)*b;
The results should be all zeros. However, they are not zeros.
When n = 6 (or less), the code gives correct answers.

Joohwa Lee <starry>
Thu 30 Sep 2021 05:04:09 AM UTC, original submission:  
Joohwa Lee <starry>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by mmuetzel
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by mleitner (Posted a comment)
  • -email is unavailable- added by starry (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-10-08 mmuetzel Carbon-Copy- => Added jwe
    2021-09-30 mmuetzel StatusNone => Need Info

    Back to the top


    Powered by Savane 3.7