bugGNU Octave - Bugs: bug #57110, `version -blas` reports...

 
 

bug #57110: `version -blas` reports "unknown or reference BLAS"

Submitter:  Andrew Janke <apjanke>
Submitted:  Wed 23 Oct 2019 11:37:30 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature Request
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * 5.1.0 Operating System:  * Mac OS
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Thu 24 Oct 2019 05:14:33 PM UTC, comment #12: 

Even though it doesn't affect me, I went ahead and filed bug #57112 for the other problem, so this bug can get back on track.

Mike Miller <mtmiller>
Group Member
Thu 24 Oct 2019 03:01:26 AM UTC, comment #11: 

I like using the Debian way and we use a similar thing with the Windows builds for Octave (but just by renaming the OpenBLAS library to be libblas.dll).  But I don't see that we can change the default BLAS library detection in the configure script to always use -lblas because the library might not even be installed on the build system with that name.  Another option is to dynamically load the library at run time, but that brings up a whole other set of issues.

It would be sweet if there were a standard BLAS library function that returned a version string.

John W. Eaton <jwe>
Group administrator
Thu 24 Oct 2019 01:05:19 AM UTC, comment #10: 


> In another thread, it sounded like you were saying Homebrew is moving towards OpenBLAS for everything, in which case I would use that and not worry about alternatives.


That is indeed the case. I'll do that.

Andrew Janke <apjanke>
Thu 24 Oct 2019 12:55:37 AM UTC, comment #9: 

In another thread, it sounded like you were saying Homebrew is moving towards OpenBLAS for everything, in which case I would use that and not worry about alternatives.

When I build Octave for other more controlled packaging systems (Docker, Flatpak, and Snap), I usually build with OpenBLAS and don't bake in any choices.

This is one of those things were I almost think that Debian packaging is giving users too much choice.

Mike Miller <mtmiller>
Group Member
Thu 24 Oct 2019 12:39:02 AM UTC, comment #8: 

Yep, that Atlas behavior is consistent: Octave's BLAS implementation probe is seeing the distinctive non-standard functions for both OpenBLAS and ATLAS in your run-time environment.

So, Dmitri, sounds like you could build the way Mike says and get better results here. How did you install your Octave? Did you build it yourself, or install it with Debian apt? What OS/distro are your running?

I have no idea how to fix this "for real", though. I suppose Octave could take a couple of the standard BLAS functions, introspect yourself to see where the actual run-time linkage for your current process goes, follow symlinks, and use some heuristics to decide which implementation that actually is? Yuck. And that would probably break under the Debian Alternatives system if you swapped the BLAS alternative after launching your octave process.

Question for Mike: Should Mac Octave builds be doing something similar, where it builds against the RI "--with-blas=blas", and then shunts in a particular implementation at run time? Instead of building directly against OpenBLAS or Accelerate like we do now?

Andrew Janke <apjanke>
Thu 24 Oct 2019 12:26:58 AM UTC, comment #7: 

Here  is with atlas:



$ LD_PRELOAD=/usr/lib64/atlas/libtatlas.so octave -q -f
octave:1> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.7 DYNAMIC_ARCH NO_AFFINITY Sandybridge SINGLE_THREADED)
ATLAS
octave:2>


Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Thu 24 Oct 2019 12:17:50 AM UTC, comment #6: 

Ok, I agree, that is a side effect of building Octave with "-lopenblas", but forcing a different libblas.so at runtime with LD_PRELOAD.

On Debian, the typical way this is managed instead is to build Octave with "--with-blas=blas", and use the alternatives system to manage symbolic links to various libblas.so.3 library implementations under directories like /usr/lib/.../blas, /usr/lib/.../openblas, /usr/lib/.../mkl, etc.

I think many of our developers work on Debian and Ubuntu based systems, so we are not as familiar with this setup of building against OpenBLAS, but using LD_PRELOAD to force an additional library to be loaded first.

Mike Miller <mtmiller>
Group Member
Thu 24 Oct 2019 12:16:48 AM UTC, comment #5: 

But even though it's the same code, it's a different aspect of it, so I think @Dmitri should open a separate bug. This bug #57110 is about "Octave's BLAS probe doesn't know about Apple Accelerate". Dmitri's problem is "Octave's BLAS probe detects which BLAS Octave was built against instead of which one it's running against".

Andrew Janke <apjanke>
Thu 24 Oct 2019 12:14:19 AM UTC, comment #4: 

@mtmiller: No, Dmitri's right. Look at those `LD_PRELOAD` lines; he's intentionally plugging in alternate BLAS implementations at run time. This is a supported feature of BLAS. It's not exactly the same problem I'm having, but it's an aspect of the same code in liboctave/lo-sysinfo.cc.

Andrew Janke <apjanke>
Thu 24 Oct 2019 12:11:15 AM UTC, comment #3: 

Aw, crap. Our whole version detection mechanism here might have a problem: it checks for - (sorry if I got this wrong, I'm no linker expert) - the presence of certain linked BLAS implementation DLLs and distinctive functions within them. As it stands now, this linkage happens at compile time, when Octave is built against a particular BLAS implementation.

Interposing a different BLAS DLL using LD_PRELOAD (or DYLD_INSERT_LIBRARIES on macOS) will change the BLAS implementation that actually gets loaded and called for work, but not undo any previous compile-time linkage, so the probes for other BLAS libraries will still succeed, because some of them check for idiosyncratic aspects of those BLAS implementation libraries, which are not part of the BLAS standard interface, and thus will not be masked by run-time DLL interposition.

So in your case, you interposed the Reference Implementation DLL, masking the build-time-linked OpenBLAS implementation, and that's what was called to do the work. But the OpenBLAS implementation was still linked, including the distinctive implementation-specific `openblas_get_config` function. So when lo-sysinfo's blas_version found openblas_get_config linked, it decided that you're just running OpenBLAS, queried `openblas_get_config` for the OpenBLAS metadata, and didn't know about the RI DLL interposition.

And this is not a cosmetic issue: as you demonstrate, the BLAS you're using significantly affects performance. And I'm working on this problem in the first place because Apple Accelerate/vecLib has some issues that cause Octave test failures or odd numeric results.

It is a bummer that the original designers of the BLAS library didn't include a standard blas_info() function that let you query the implementation metadata. Oh well, live and learn; BLAS was an innovator and library/API design has evolved a lot since then.

I do not know enough about linkers and loaders to know how to do a probe that can detect run-time interposition.

Andrew Janke <apjanke>
Thu 24 Oct 2019 12:05:33 AM UTC, comment #2: 

I think something else is going on on your system, Dmitri, does not seem related to this feature request. Octave does not misidentify reference BLAS on my Debian system:


$ octave-cli-6.0.0 --eval "version -blas"
ans = OpenBLAS (config: OpenBLAS 0.3.7 NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=64)

$ LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/blas:/usr/lib/x86_64-linux-gnu/lapack octave-cli-6.0.0 --eval "version -blas"
ans = unknown or reference BLAS


Mike Miller <mtmiller>
Group Member
Wed 23 Oct 2019 11:57:48 PM UTC, comment #1: 

perhaps the following is related (dev version on Fedora):


$ LD_PRELOAD=/usr/lib64/libopenblas.so octave -q -f
octave:1> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.7 DYNAMIC_ARCH NO_AFFINITY Sandybridge SINGLE_THREADED)
octave:2> a=randn(3000);
octave:3> tic; inv(a)*a; toc
Elapsed time is 4.74749 seconds.
octave:4>

$ LD_PRELOAD=/usr/lib64/libopenblasp.so octave -q -f
octave:1> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.7 DYNAMIC_ARCH NO_AFFINITY Sandybridge MAX_THREADS=128)
octave:2> a=randn(3000);
octave:3> tic; inv(a)*a; toc
Elapsed time is 2.03826 seconds.
octave:4>

$ LD_PRELOAD=/usr/lib64/libblas.so.3 octave -q -f
octave:1> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.7 DYNAMIC_ARCH NO_AFFINITY Sandybridge SINGLE_THREADED)
octave:2> a=randn(3000);
octave:3> tic; inv(a)*a; toc
Elapsed time is 37.4247 seconds.
octave:4>


it misidentifies reference blas as openblas.

Dmitri.
--

Dmitri A. Sergatskov <dasergatskov>
Wed 23 Oct 2019 11:37:30 PM UTC, original submission:  

I'm running Octave 4.4.1 and 5.1.0 on macOS 10.14.6, using Octave.app, which has been built against Apple Accelerate (instead of OpenBLAS or whatever).

`version -blas` is not reporting Apple Accelerate.


>> version -blas
ans = unknown or reference BLAS
>> version -lapack
ans = Linear Algebra PACKage Version 3.2.1
>>


Looks like there's just no check for Apple Accelerate in lo-sysinfo. I'll see if I can hack one together.

Downstream bug report: https://github.com/octave-app/octave-app/issues/172

Andrew Janke <apjanke>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by dasergatskov (Posted a comment)
  • -email is unavailable- added by apjanke (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code