bugGNU Octave - Bugs: bug #50463, gsvd implementation uses functions...

 
 

bug #50463: gsvd implementation uses functions removed from lapack >= 3.6.0

Submitted by:  Carlo de Falco <cdf>
Submitted on:  Sun 05 Mar 2017 06:50:25 PM UTC  
 
Category: Configuration and Build SystemSeverity: 5 - Blocker
Priority: 5 - NormalItem Group: Build Failure
Status: Patch SubmittedAssigned to: None
Originator Name: Open/Closed: Open
Release: devOperating System: Any

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 07 Jan 2018 09:35:37 PM UTC, comment #31:

The last patch no longer compiles after the changes from: http://hg.savannah.gnu.org/hgweb/octave/rev/c5c11b07598a
I followed that changeset and also removed the interrupt handling.

I would appreciate if anyone affected by this bug could try the attached updated patch and report if it worked for them.

(file #42867)

Markus Mützel <mmuetzel>
Project Member
Wed 03 Jan 2018 08:02:58 PM UTC, comment #30:

The attached patch changes the dynamic_library class to search all loaded functions if the library name is empty.
I refrained from deleting all other interfaces apart from dlopen and LoadLibrary just yet. Maybe someone else can do that in the future...

I don't know whether the patch breaks gsvd on Mac and I can't test.
It should be working on Linux and Windows.

As jwe said on IRC, the check in configure.ac should probably be made conditional for Windows only. Any help appreciated.

(file #42812)

Markus Mützel <mmuetzel>
Project Member
Wed 03 Jan 2018 06:00:49 PM UTC, comment #29:

@Carlo: I might have misread your question: A test on build-time would probably be not too difficult. I thought the agreed goal was to have a check on run-time (or load-time).

Markus Mützel <mmuetzel>
Project Member
Wed 03 Jan 2018 04:25:52 PM UTC, comment #28:

@Markus, no problem at all! I was just thinking about it in theory, I would never have had time to actually do anything in practice so you are VERY welcome to work on this.

Anyway, if the approach I had in mind requires GCC it is also
not practical on macos as the default system compiler is now clang++.

thanks for answering my curiosity!
c.

Carlo de Falco <cdf>
Project Member
Wed 03 Jan 2018 04:14:21 PM UTC, comment #27:

@Carlo: Sorry. I didn't realize you were also working on a solution. I should have asked earlier. Sorry for the double work.

From what I've read during the last few days, your approach could work using weak links to these functions (a gcc addition to the standard). Unfortunately, I haven't found an equivalent to weak links on Windows.

Markus Mützel <mmuetzel>
Project Member
Wed 03 Jan 2018 03:45:36 PM UTC, comment #26:

Hi,

It seems Markus has an almost finished solution so this
comment should be now out of date, but I had been thinking
of an alternative solution and, even though it will not be used,
I still wonder whether that would have been feasible.

I was thinking we could ship with Octave sources a shared lib
containing xGGSVD3 from the reference LAPACK implementation and
tweak the build system so that it is overridden by the implementation in the linked lapack lib if there is one provided at build time or via LD_PRELOAD.

Would that be possible somehow?

c.

Carlo de Falco <cdf>
Project Member
Wed 03 Jan 2018 03:28:11 PM UTC, comment #25:

A quick check seems to indicate that this actually might work. After compiling the attached file with "mkoctfile -lpsapi check_lib_fcn3.cc", I see the following with the liblapack.dll from Octave 4.2.1:

Most of the build system is still a black box to me: How would psapi be added to the linked libraries? Or is it already?

(file #42808)

Markus Mützel <mmuetzel>
Project Member
Wed 03 Jan 2018 03:24:13 PM UTC, comment #24:

It looks like that could work for Windows. Some care will apparently be needed to ensure that the output array is sized correctly, but I guess it is possible. Weird interface with the length being specified in bytes..

This page https://developer.apple.com/library/content/qa/qa1180/_index.html says that dlopen has been part of Mac OS since 10.3 (released in 2003) and based on dyld calls since 10.4 (released in 2005) so I think it's OK to remove direct support for the dyld interface from Octave now.

A quick search shows that HP/UX did eventually get a dlopen interface so it might be OK to remove support for shl_load even if we still cared about support HP/UX. But I don't recall anyone reporting a problem with HP/UX in many years, so I suspect no one has built current versions of Octave on HP systems in a decade or more. Indeed, this page https://gcc.gnu.org/install/specific.html#hppa-hp-hpux says that although GCC does still support HP/UX, it requires explicit template instantiation, so it would be virtually impossible at this point to build Octave. So I'd say we just declare HP/UX dead to Octave and sing songs of great joy on this glorious day.

and there are only a few places in Octave that

John W. Eaton <jwe>
Project Administrator
Wed 03 Jan 2018 02:53:11 PM UTC, comment #23:

jwe: That is what I thought as well.
To change the dlsym-part of Octave's dynamic_library class shouldn't be too hard.
Unfortunately, "GetModuleHandle (0)" only works for the "main" process. Subsequent calls to "GetProcAddress" don't find functions linked in from loaded libraries. I was unsuccessful finding an equivalent to RTLD_GLOBAL for LoadLibrary (or GetModuleHandle).
Searching the loaded function list seems to be possible on HP/UX (but I have no way of testing this). Removing support for it would also simplify the code.
I haven't researched the DYLD_API (again, I have no way to test any changes). If this could be removed as well even better (from a developer's point of view).

I haven't tried. But for the Windows platform, it might be possible to query a list of all loaded libraries (which are called "modules" in Microsoft's speech from what I gather) with "EnumProcessModules" [1] on the current process and then check each module iteratively with "GetProcAdress" for an entry point to the queried function.

Should I give that a try?

[1] https://msdn.microsoft.com/de-de/library/windows/desktop/ms682631(v=vs.85).aspx

Markus Mützel <mmuetzel>
Project Member
Wed 03 Jan 2018 01:42:08 PM UTC, comment #22:

It appears that

will return a handle to the current process that may then be passed to GetProcAddress to search for a function. Does that also work to search any DLLs that are already linked or loaded as dlopen (0) and dlsym does? If so, then we could easily fix Octave's dynamic_library class to search for functions that are available in the current process.

I found this info here:

https://stackoverflow.com/questions/11741580/dlopen-loadlibrary-on-same-application

John W. Eaton <jwe>
Project Administrator
Wed 03 Jan 2018 01:16:14 PM UTC, comment #21:

Markus: about comment #19, I think we should extend the dynamic_library class to do what is needed instead of including dlfcn.h and calling dlsym directly.

Is there a way for the LoadLibrary API to check for functions that are already linked in the way that dlsym does?

Does HP/UX still exist anywhere in the wild? If we no longer care about that system, then the SHL_LOAD_API bits could be removed.

What about the DYLD_API part? How long has Mac OS had the dlopen API? Maybe we can also remove that now?

John W. Eaton <jwe>
Project Administrator
Tue 02 Jan 2018 11:14:22 PM UTC, comment #20:

Thank you, Carlo, for providing a patch that implements how the new *ggsvd3 functions of LAPACK are used.
The attached patch checks on run-time whether these functions are available in the used LAPACK library and falls back to the *ggsvd functions if they are not.
It works for me on Linux and Windows where I tried with a recent liblapack.dll from MXE-Octave (that includes the *ggsvd3 functions) and one from 4.2.1 (that doesn't provide them).
Can someone please check that it also works as expected with a LAPACK library without the *ggsvd functions?

(file #42792)

Markus Mützel <mmuetzel>
Project Member
Sat 23 Dec 2017 02:12:38 PM UTC, comment #19:

With a cross build of the development branch (I don't remember which hg id of mxe-octave that is from, but it was built on 16 Dec 2017), I see:

On another note: the "dynamic_library" class (in oct-shlib.cc) differentiates between four different mechanism to load shared libraries:

Is it save to unconditionally use the dlsym magic on Linux and Mac, and the "dynamic_library" class on Windows only?

OT: Does MXE Octave insert its hg id somewhere into the build? A text file somewhere in the packed folders would probably be enough to keep track of it.

Markus Mützel <mmuetzel>
Project Member
Sat 23 Dec 2017 12:33:17 PM UTC, comment #18:

Windows is using lapack 3.7.1 (at least in the repo), released June, 2017.

It has depreciated functions turned on that was allowing it to work ok with octave.

John Donoghue <lostbard>
Project Member
Fri 22 Dec 2017 08:55:24 PM UTC, comment #17:

Yes, that would do it. Is the answer correct on Windows using a development version? I'm surprised we don't ship an lapack that is modern.

Rik <rik5>
Project Administrator
Fri 22 Dec 2017 08:19:51 PM UTC, comment #16:

Compiling the attached file with "mkoctfile check_lib_fcn.cc", I see the following on Octave 4.2.1 on Windows 10:

And this on Ubuntu 17.10:

@Rik: Is that the kind of portable check you were referring to in comment #13?
At least on Windows we should know the name of the library that we are packing. Not so sure on Linux or Mac.

(file #42706)

Markus Mützel <mmuetzel>
Project Member
Sat 25 Nov 2017 09:42:12 AM UTC, comment #15:

The attached patch applies smoothly to the current development branch.

(file #42478)

Carlo de Falco <cdf>
Project Member
Fri 24 Nov 2017 09:25:35 AM UTC, comment #14:

Hi, I can help with updating the patch and even provide runtime check for unix but if something windows specific needs to bee done I'm not really sure I can help.

Carlo de Falco <cdf>
Project Member
Thu 23 Nov 2017 11:57:08 PM UTC, comment #13:

It appears that a runtime check can be implemented, but there will need to be separate code for Windows and UNIX-like operating systems.

For the UNIX side, the code below should work to create a persistent variable that is evaluated only once and can then be used as a conditional around blocks of code.

For Windows, it appears that a combination of LoadLibrary/GetProcAddress is necessary. See StackOverflow article: https://stackoverflow.com/questions/42703922/windows-api-call-to-check-if-a-dll-contains-a-particular-procedure#42704262.

Finally, the patch from comment #10 would need to be updated.

Probably the prototypes for both DGGSVD and DGGSVD3 needs to be at the start of the file just so it will actually compile. It also probably means compiling and linking the version of Octave that is exported, say through MXE, with lapack >= 3.6.0.

Rik <rik5>
Project Administrator
Mon 05 Jun 2017 02:28:26 PM UTC, comment #12:

Ping!

I hit this also when building in Mageia-6. The patch in comment #10 helps.

Philip Nienhuis <philipnienhuis>
Project Member
Wed 03 May 2017 05:09:01 PM UTC, comment #11:

Great. I'll review whenever it is available.

Rik <rik5>
Project Administrator
Wed 03 May 2017 01:33:10 PM UTC, comment #10:

Here's the updated patch.

It was so simple to update it that I wonder why mercurial
could not work it out automatically ...

This still replaces dggsvd by dggsvd3 unconditionally,
I'll now try the approach suggested by Rik for checking
whether dggsvd3 is actually available before invoking it.

c.

(file #40572)

Carlo de Falco <cdf>
Project Member
Fri 28 Apr 2017 11:25:47 PM UTC, comment #9:

What was the resolution on whether this should be a runtime or compile time check for DGGSVD3?

I do see Mike's point about people switching BLAS libraries in and out. The MXE installer for Windows, for example, gives the user the option to use either OpenBLAS or refBLAS.

If it is a runtime test, then maybe promote the test to a static variable that is initialized just once at startup?

For example,

Also, this looks good, but is there a way to make this portable to other OS like Windows which might not have dlsym? Could you do a try/catch block?

Rik <rik5>
Project Administrator
Fri 28 Apr 2017 03:59:40 PM UTC, comment #8:

as of hg id 855122b993da the patch below does not apply anymore.

I'll be looking at how to adapt the patch, unless anyone else knows straight away how to fix it ...

Carlo de Falco <cdf>
Project Member
Sun 26 Mar 2017 02:16:13 PM UTC, comment #7:

Mike,

I do see the point in your proposed approach.

Please notice, though, that, as noted in comment #1 , the change
is indeed in netlib lapack API and not only in OpenBlas:

http://www.netlib.org/lapack/lapack-3.6.0.html#_3_lapack_3_6_0_what_8217_s_new

It also seems the problem had already been noticed by Nir and Carnë who were supervising a related GSOC project :

https://bitbucket.org/BarbaraLocsi/octave
http://octave.1599824.n4.nabble.com/Deprecated-LAPACK-routines-GSVD-tp4679040.html

Also it appears that scipy have been discussing the same issue:

https://github.com/scipy/scipy/issues/5266

So assuming the BLAS/LAPACK API to be stable is not really such a safe choice anyway ...

c.

Carlo de Falco <cdf>
Project Member
Sat 25 Mar 2017 09:27:34 PM UTC, comment #6:

I don't have a patch, but a safer alternative might be to detect whether the LAPACK library that Octave has already loaded has the newer function or not.

Something like

and of course need to adapt this to other operating systems.

Mike Miller <mtmiller>
Project Administrator
Sat 25 Mar 2017 09:20:41 PM UTC, comment #5:

Not saying this is a bad change, but this has a possibility of unintended consequences. Until now, Octave was built against the BLAS / LAPACK stable API that hasn't changed in decades. One could build Octave against standard BLAS and LAPACK, and then switch to either ATLAS or OpenBLAS at run time.

If this change is applied, it's possible to build Octave against an old BLAS / LAPACK, and then try to use OpenBLAS at run time, which may exhibit the same run time error. Or it may be possible to build Octave against a LAPACK that does have the xGGSVD3 functions, and switch to a different one at run time that doesn't, resulting in a similar run time error.

On my system, NETLIB, ATLAS, and OpenBLAS all have the xGGSVD3 functions, and I don't know how likely it is to find a system that has a mix of libraries with and without this function, but it's possible.

Mike Miller <mtmiller>
Project Administrator
Sun 19 Mar 2017 01:37:23 PM UTC, comment #4:

Thanks Rik,
I'll combine the two patches tomorrow on the train to Geneve,
hope we can push it during the code sprints.
c.

Carlo de Falco <cdf>
Project Member
Sun 19 Mar 2017 05:47:54 AM UTC, comment #3:

@Carlo: Attached is a small piece of detection code for configure.ac which will set the #define HAVE_DDGSVD3 if it is available in the LAPACK library on your system.

This should be combined with your existing patch. Everywhere in the code where there is a difference between DDGSVD3 and DDGSVD use this construct:

(file #40023)

Rik <rik5>
Project Administrator
Tue 07 Mar 2017 10:29:55 PM UTC, comment #2:

Actually the reason why the previous patch did not
work was just a stupid mistake from my side.

The new patch build and passes all tests on my system.

Still needs the autoconf part to be complete,
I could really use some help for that.

(file #39931)

Carlo de Falco <cdf>
Project Member
Tue 07 Mar 2017 10:38:16 AM UTC, comment #1:

I changed the title to include the minimum version of lapack needed to show the issue, which does not occur with the openblas implementation only.

I am still investigating why my patch does not work, even
if it did work it would still need to include autoconf tests
to determine the available version of ggsvd functions in
order to preserve backwards compatibility.

Carlo de Falco <cdf>
Project Member
Sun 05 Mar 2017 06:50:25 PM UTC, original submission:

Hi,

I'm building the default branch on macOS 10.12.3
using maports for dependencies and openblas for
blas/lapack.

The build fails at the linking stage with the error

Looking at openblas sources, it appears that cggsvd is
deprecated and replaced by cggsvd3.

The attached patch solves the build problem for me but fails
at runtime with error:

Carlo de Falco <cdf>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #42867:  bug50463_lapack_ggsvd3_v3.patch added by mmuetzel (32KiB - application/octet-stream)
file #42812:  bug50463_lapack_ggsvd3_v2.patch added by mmuetzel (34KiB - application/octet-stream)
file #42808:  check_lib_fcn3.cc added by mmuetzel (2KiB - application/octet-stream)
file #42792:  bug50463_lapack_ggsvd3.patch added by mmuetzel (29KiB - application/octet-stream)
file #42706:  check_lib_fcn.cc added by mmuetzel (1KiB - application/octet-stream)
file #42478:  open_OazyWCsr.txt added by cdf (13KiB - text/plain)
file #40572:  open_sJ86Ow6X.txt added by cdf (13KiB - text/plain)
file #40023:  config.ac.diff added by rik5 (948B - application/x-download)
file #39931:  open_1VYMi0ZP.txt added by cdf (13KiB - text/plain)
file #39896:  open_F5jqLbmL.txt added by cdf (13KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by lostbard (Posted a comment)
  • -unavailable- added by mmuetzel (Updated the item)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by rik5 (Updated the item)
  • -unavailable- added by cdf (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 16 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 07 Jan 2018 09:35:37 PM UTCmmuetzelAttached File-=>Added bug50463_lapack_ggsvd3_v3.patch, #42867
    Wed 03 Jan 2018 08:02:58 PM UTCmmuetzelAttached File-=>Added bug50463_lapack_ggsvd3_v2.patch, #42812
    Wed 03 Jan 2018 03:28:11 PM UTCmmuetzelAttached File-=>Added check_lib_fcn3.cc, #42808
    Tue 02 Jan 2018 11:14:22 PM UTCmmuetzelAttached File-=>Added bug50463_lapack_ggsvd3.patch, #42792
      StatusIn Progress=>Patch Submitted
    Fri 22 Dec 2017 08:19:51 PM UTCmmuetzelAttached File-=>Added check_lib_fcn.cc, #42706
    Sat 25 Nov 2017 09:42:12 AM UTCcdfAttached File-=>Added open_OazyWCsr.txt, #42478
    Wed 03 May 2017 01:33:10 PM UTCcdfAttached File-=>Added open_sJ86Ow6X.txt, #40572
    Fri 28 Apr 2017 11:25:47 PM UTCrik5Operating SystemMac OS=>Any
    Sun 19 Mar 2017 05:48:58 AM UTCrik5StatusNone=>In Progress
    Sun 19 Mar 2017 05:47:54 AM UTCrik5Attached File-=>Added config.ac.diff, #40023
    Tue 07 Mar 2017 10:30:23 PM UTCcdfSummarycggsvd implementation uses functions removed from lapack >= 3.6.0=>gsvd implementation uses functions removed from lapack >= 3.6.0
    Tue 07 Mar 2017 10:29:55 PM UTCcdfAttached File-=>Added open_1VYMi0ZP.txt, #39931
    Tue 07 Mar 2017 10:38:16 AM UTCcdfSeverity3 - Normal=>5 - Blocker
      Summarylinking fails as cggsvd deprecated in -lopenblas =>cggsvd implementation uses functions removed from lapack >= 3.6.0
    Sun 05 Mar 2017 06:50:25 PM UTCcdfAttached File-=>Added open_F5jqLbmL.txt, #39896

    Back to the top


    Powered by Savane 3.1-cleanup1