bugGNU Octave - Bugs: bug #53627, build: oct files should not need...

 
 

bug #53627: build: oct files should not need to link with -loctinterp -loctave

Submitted by:  Mike Miller <mtmiller>
Submitted on:  Thu 12 Apr 2018 06:52:15 PM UTC  
 
Category:  Configuration and Build System Severity:  2 - Minor
Priority:  5 - Normal Item Group:  Feature Request
Status:  Ready For Test Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  dev Operating System:  GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Fri 03 Aug 2018 07:17:37 PM UTC, comment #24:

Here's a revised attempt at this, which sets -no-undefined conditionally based on which OS it's running on, with it on by default only on Windows/BeOS/OS2/AIX. file #44700: macos-bundle-undefined-dynamic_lookup-v3.patch

I've added "-bind_on_load" because I think that will produce better fail-fast behavior when loading octfiles with undefined symbols that cannot be resolved.

> What happens if you omit that option and do have undefined symbols? Does the program fail to load and run, or fail if it attempts to call an undefined function?


On macOS, with -no-undefined omitted, mkoctfile can compile files that have undefined symbols, but when they're called, it will produce an unresolved-symbol error (without crashing Octave). Also, you can build Octave without passing --enable-link-all-dependencies to configure.

I expect stuff in libinterp/dldfcn would behave similarly, but I'm not sure how to test that.

Andrew Janke <apjanke>
Fri 03 Aug 2018 07:05:59 PM UTC, comment #23:

> Have you looked at the work that was done for bug #41574?


I looked through #41574, and I don't think anything from there needs to be pulled in here. Commented over there on the current state of things.

Andrew Janke <apjanke>
Thu 19 Jul 2018 10:04:15 PM UTC, comment #22:

This is all very confusing and very operating system dependent.

I would think we want to follow the libtool default behavior, at least on as many operating systems as it is possible to do so. On Windows clearly it is not, DLLs need to be built against the stub library that contains the definitions for them to use. But on most other operating systems, it is a choice, and I think the choice should be to default to allowing undefined symbols.

Glancing through the libtool code, there is a set of linker flags for allowing undefined symbols, another set of flags for no undefined symbols, as well as whether one behavior or the other is unsupported.

On GNU/Linux, I think all of the flags are empty, so the linker is called the same way with or without the -no-undefined option.

I think on Unix operating systems we should not be using -no-undefined. It seems to only be required on Windows/MinGW/Cygwin, BeOS, OS/2, and AIX v3.

I think libtool will either error if you try to build for Windows without this option, or maybe build a static library instead of a shared object and print a warning.

Mike Miller <mtmiller>
Project Administrator
Thu 19 Jul 2018 08:26:15 PM UTC, comment #21:

The -no-undefined option is needed when building Windows DLL files.

What happens if you omit that option and do have undefined symbols? Does the program fail to load and run, or fail if it attempts to call an undefined function?

John W. Eaton <jwe>
Project Administrator
Thu 19 Jul 2018 08:02:33 PM UTC, comment #20:

I never realized that Octave was linking with -no-undefined by default, that was added before my time here. Is there a reason for that? I would think that option shouldn't normally be used on any system.

Mike Miller <mtmiller>
Project Administrator
Thu 19 Jul 2018 01:52:01 PM UTC, comment #19:

Have you looked at the work that was done for bug #41574?

John W. Eaton <jwe>
Project Administrator
Thu 19 Jul 2018 03:52:57 AM UTC, comment #18:

I've attached file #44589: macos-bundle-undefined-dynamic_lookup-v2.patch, which is my latest attempt to get `-undefined dynamic_lookup` working. It's getting closer, but not there yet.

With this, the build succeeds. Though I have to do an explicit `--disable-no-undefined` with `configure`.

The oct files it produces do not have the liboct/libinterp linkage.

And they work when run.

However, the test suite crashes, and early.

Mike, I'll give that "-Xlinker -undefined -Xlinker dynamic_lookup" a try if I have some time in the next couple days. And getting that libtool.m4 logic to activate may be the best way, but I'll need to do some work to figure out how to do that. But it's time to call it a night for now.

Andrew Janke <apjanke>
Thu 19 Jul 2018 03:37:17 AM UTC, comment #17:

Yes, try "-Xlinker -undefined -Xlinker dynamic_lookup".

I see some logic in GNU libtool to automatically use these options, or other variants of them, depending on the name and the version of the operating system:

Mike Miller <mtmiller>
Project Administrator
Thu 19 Jul 2018 02:38:23 AM UTC, comment #16:

I've attached a changeset, file #44587: macos-bundle-undefined-dynamic_lookup.patch, that attempts to enable `-undefined dynamic_lookup` for octfiles and dynamic libraries by adding it to DL_LDFLAGS and MKOCTFILE_DL_LDFLAGS in acinclude.m4.

This isn't sufficient, though. When I do the build with this, it's still erroring out with undefined symbols.

The `-undefined dynamic_lookup` option is absent from the `g++` call there. Sounds like we need to tell `libtool` to use that option. Does anybody know how to do that?

Andrew Janke <apjanke>
Wed 18 Jul 2018 11:59:43 PM UTC, comment #15:

And here's the Apple documentation for it:

See "Loading Plug-in Code With Bundles" under:
https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/MachOTopics/1-Articles/loading_code.html#//apple_ref/doc/uid/TP40001830

And this mirror of the "OS X ABI Dynamic Loader Reference":
https://github.com/jeremysears/osx-abi-dyamic-loader-reference

It doesn't really document the `-bundle` and `-undefined` `ld` options.

The O'Reilly book "Mac OS X for Unix Geeks" also discusses this in section 5.3. I don't see a legit publicly-available copy of that, though.

Andrew Janke <apjanke>
Wed 18 Jul 2018 11:29:53 PM UTC, comment #14:

I don't really know how Python does things, but I think the macOS equivalent for plugin-style loadable modules is building them as "dynamically loaded modules" or "Mach-O bundles" (totally different from "app bundles") instead of a "shared library".

The `-bundle` option to macOS's `ld` looks relevant, along with `-undefined dynamic_lookup`.

This is hard to Google because "bundle" seems to mean two different things in the context of macOS: a Mach-O plugin-style loadable module like we want here, and an "app bundle" directory structure, which is the more common usage.

In Python distutils' tests, there's a reference to using `-bundle -undefined dynamic_lookup`, but I can't find it in the main code. From `tests/test_unixccompiler.py`:

It looks like octave is already making some use of this. In `configure`:

and in the output from my broken Octave build below:

I can see that it's passing `-bundle`. It's also passing `-no-undefined`. Maybe that needs to be `-undefined dynamic_lookup` instead of `-no-undefined`? `-undefined dynamic_lookup` seems to be the magic sauce for having the linker allow unresolved symbols at compilation/initial link time. That's what the Tim Smith blog post links below indicates, for Python modules. It also mentions crashes when not using this, which sounds like the behavior we're seeing with octfiles and Homebrewed Octave on macOS.

> The best solution is to provide -bundle -undefined dynamic_lookup,
> which has the same behavior as the Linux linker.


Based on my reading of that and other Google results for Python modules, then yes, a single compiled extension is portable between two different instances of the same version of Python.

Some references:

https://stackoverflow.com/questions/2339679/what-are-the-differences-between-so-and-dylib-on-osx

https://blog.tim-smith.us/2015/09/python-extension-modules-os-x/

https://cmake.org/pipermail/cmake/2005-March/006249.html

https://books.google.com/books?id=H3JmBgAAQBAJ&pg=PA197&lpg=PA197

From `man ld` on macOS 10.13:

Looks like Lua does the same thing for its dynamic modules:
http://lua-users.org/lists/lua-l/2005-03/msg00349.html

Andrew Janke <apjanke>
Wed 18 Jul 2018 09:51:30 PM UTC, comment #13:

The linker errors here

seem to indicate that the macOS linker requires some resolution of undefined symbols even in a plugin-type shared object.

On GNU/Linux, the advice for building plugin-style loadable modules is to avoid linking them with libraries that the executable will already have opened in the application process space.

For example, when building Python extensions in C, the .so files are not linked against libpython, even though the extension will be calling Python C API functions. At build time, these are unresolved symbols, but the linker does not care.

I guess we need to know what the equivalent build process should be on macOS and we'll do it.

How are C Python extensions built on macOS? Is a single compiled extension portable between two different instances of the same version of Python?

Mike Miller <mtmiller>
Project Administrator
Wed 18 Jul 2018 09:19:14 PM UTC, comment #12:

While we're on this subject: it would be nice if oct files also did not link to liboctinterp and liboctave on macOS. The presence of those dylib links means (I think) that a given octfile is only compatible with an Octave that is installed in the same location as the original Octave that produced those octfiles.

For example, this is what I see for the octfiles produced by the control package:

This has shown up as an issue for a user who's interested in using both a Mac-style Octave.app and a Homebrew-installed octave on the same system: http://lists.gnu.org/archive/html/octave-maintainers/2018-07/msg00101.html. It's of interest to me also, because I would like to be able to install multiple versions and variants of Octave side-by-side on the same machine for testing.

Is there anything I can do to help figure out how to do this on macOS?

Andrew Janke <apjanke>
Wed 18 Jul 2018 09:14:51 PM UTC, comment #11:

That other build failure is fixed now.

I have updated to the latest default branch in the Hg repo, and am still getting a link-related build failure.

I'm attaching bug53627_example01.tgz with details of the files produced in my build.

(file #44584)

Andrew Janke <apjanke>
Sat 14 Jul 2018 06:24:30 AM UTC, comment #10:

I'm getting a different, apparently unrelated build failure on macOS now, so I can't confirm the fix.

Will look in to this when I have time.

Andrew Janke <apjanke>
Wed 11 Jul 2018 11:51:23 PM UTC, comment #9:

Is this still a problem for MacOS builds? Or can we close this report?

Rik <rik5>
Project Administrator
Sun 01 Jul 2018 08:13:20 AM UTC, comment #8:

I think this may have broken the build on Mac, too.

I am building Octave on macOS 10.13.5. I can build the 4.4.0 release okay, but building from the tip of the Hg repo is broken with the following build error.

I'm configuring using the following:

Switching to shared libs doesn't fix it; and doing --enable-link-all-dependencies breaks the build with a different error.

Maybe this needs to be conditionalized for macOS as well as Windows?

Andrew Janke <apjanke>
Thu 14 Jun 2018 08:33:40 PM UTC, comment #7:

I pushed this changeset:

http://hg.savannah.gnu.org/hgweb/octave/rev/245de4f12ca3

John W. Eaton <jwe>
Project Administrator
Wed 13 Jun 2018 08:31:49 PM UTC, comment #6:

Same issue is seen when cross compiling any of the octave forge packages .oct files using mkoctfile in windows, so I guess it does need be conditional for windows.

John Donoghue <lostbard>
Project Member
Wed 13 Jun 2018 08:14:45 PM UTC, comment #5:

Yes, I assume so. If there is no better solution, then I can make the change conditional so it works as before on Windows systems.

John W. Eaton <jwe>
Project Administrator
Wed 13 Jun 2018 11:43:10 AM UTC, comment #4:

Im assing this is directlt related to this change.

In mxe with latest dev (1c4ffd4f9404: create valid tmp scope for who -file )

John Donoghue <lostbard>
Project Member
Mon 11 Jun 2018 07:14:48 PM UTC, comment #3:

I pushed the following changeset:

http://hg.savannah.gnu.org/hgweb/octave/rev/fa66d81d0956

If this causes trouble on Mac or Windows systems, then we can undo whatever is needed and make it conditional. It would also be unfortunate, since it will make building and testing MEX files in the build directory a little more complicated.

John W. Eaton <jwe>
Project Administrator
Mon 11 Jun 2018 06:10:19 PM UTC, comment #2:

If we remove "-loctinterp -loctave" from the link command, then I think we can also eliminate the DEFAULT_LFLAGS variable. We can still accept LFLAGS from the environment, but there is no need for a default.

Since I'm looking at a way to build MEX files for testing and it would be most convenient if we didn't have to worry about linking with liboctave and liboctinterp, I can look at making these changes now.

John W. Eaton <jwe>
Project Administrator
Wed 23 May 2018 09:20:58 PM UTC, comment #1:

What is the minimum unit of change here?

It seems like we could start with modifying the mkoctfile command to remave '-loctinterp -loctave'. Is that expected to be safe on all platforms, or just Linux?

The next step would be to remove LIBADD from the .oct files, and then test on Linux, macOS, and Windows.

Final step would be to eliminate DLD_LIBOCTINTERP_LIBADD as well.

Does that sound right?

Rik <rik5>
Project Administrator
Thu 12 Apr 2018 06:52:15 PM UTC, original submission:

This applies equally to the .oct files that are built as part of Octave, as well as the mkoctfile rules for building external .oct files.

On GNU/Linux, a shared object that is a plugin or loadable module doesn't need to link with the base libraries of the program that is loading it. Since Octave is doing the loading, all .oct files will have access to the functions in Octave's executable image.

I propose that we drop DLD_LIBOCTINTERP_LIBADD and remove both liboctinterp.la and liboctave.la from the LIBADD of each of the .oct files built in libinterp/dldfcn. I also propose that "-loctinterp -loctave" be removed from the mkoctfile build command.

But this will definitely need some testing on macOS and Windows to see whether this is safe or needs to be made conditional.

Mike Miller <mtmiller>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44700:  macos-bundle-undefined-dynamic_lookup-v3.patch added by apjanke (3KiB - application/octet-stream - Revised version with -no-undefined conditionally set based on OS)
file #44587:  macos-bundle-undefined-dynamic_lookup.patch added by apjanke (1KiB - application/octet-stream - patch to enable -undefined dynamic_lookup on macOS)
file #44584:  bug53627_example01.tgz added by apjanke (237KiB - application/gzip - files from build failure)

 

Digest:
   bug dependencies.

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mtmiller (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-08-03 apjanke Attached File- => Added macos-bundle-undefined-dynamic_lookup-v3.patch, #44700
    2018-07-19 apjanke Attached File- => Added macos-bundle-undefined-dynamic_lookup-v2.patch, #44589
    2018-07-19 apjanke Attached File- => Added macos-bundle-undefined-dynamic_lookup.patch, #44587
    2018-07-18 apjanke Attached File- => Added bug53627_example01.tgz, #44584
    2018-07-17 rik5 Dependencies- => Depends on bugs #54324
    2018-06-14 jwe StatusPostponed => Ready For Test

    Back to the top


    Powered by Savane 3.3