bugGNU Octave - Bugs: bug #55363, global variable in multiple mex or...

 
 

bug #55363: global variable in multiple mex or oct files collapse to one definition with -Bsymbolic-functions

Submitted by:  Guillaume <gyom>
Submitted on:  Wed 02 Jan 2019 01:02:32 PM UTC  
 
Category:  Configuration and Build System Severity:  1 - Wish
Priority:  5 - Normal Item Group:  Feature Request
Status:  Confirmed Assigned to:  None
Originator Name:  Guillaume Open/Closed:  Open
Release:  dev Operating System:  GNU/Linux

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Tue 08 Jan 2019 11:12:04 AM UTC, comment #12:

I agree. A version script should be used when compiling MEX files, as this is expected for Matlab compatibility and can otherwise create very obscure bugs (as I experienced). Is this gcc specific or is there a similar flag compatible with, eg, clang?
For oct files, do all of the necessary symbol have the same prefix, as it seems that one can use 'DEFUN*' in the version script?

Guillaume <gyom>
Mon 07 Jan 2019 08:37:17 PM UTC, comment #11:

I retitled this to more accurately descripe the situation. I was also able to confirm this with an oct file demo, attached as foo1.cc and foo2.cc. So this is entirely a linker phenomenon with the -Bsymbolic-functions option, which allows multiple definitions of the same symbol to collapse to the same definition in the process space at run time.

The feature request at this point looks like making mkoctfile automatically pass a version script to the linker when building a mex file, so that it filters out all symbols and only the 'mexFunction' symbol is exported.

Should the same magic be done on oct files? That's a little more complicated, as mkoctfile would have to essentially parse the symbol tables and look for all DEFUN-related symbols and add them to a dynamic version script. But it would make oct files more robust as well, only exporting the minimum that is required for the oct file to be loaded by Octave.

Mike Miller <mtmiller>
Project Administrator
Mon 07 Jan 2019 08:25:46 PM UTC, comment #10:

So Matlab provides a boilerplate version script file that it passes to the linker to ensure that only the mexFunction symbol is exported in the dynamic symbol table? Octave can do that as well if there is no other way to deal with this. I do not think Octave should be in the business of parsing LDFLAGS and trying to remove individual options that the user or distribution intentionally added.

Mike Miller <mtmiller>
Project Administrator
Mon 07 Jan 2019 11:25:38 AM UTC, comment #9:

Adding '-Wl,-Bsymbolic-functions' manually to the Matlab mex compilation command line does not change the behavior. Matlab seems to use the '-Wl,--version-script=' flag, which probably takes precedence.

On Ubuntu, the '-Wl,-Bsymbolic-functions' link option comes from $OCT_LINK_OPTS. If I redefine it to contain '-Wl,-z,relro' only before calling mkoctfile, then everything works fine again. $OCT_LINK_OPTS seems to be populated from $LDFLAGS.

I could find a number of related discussions, where that flag created similar issues:

https://lists.ubuntu.com/archives/ubuntu-devel/2018-June/040369.html
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637184
https://bugs.launchpad.net/ubuntu/+source/tango/+bug/1250534
https://github.com/grpc/grpc/issues/11195

Would there be a way to automatically strip '-Wl,-Bsymbolic-functions' from $LDFLAGS if present? As mentioned in comments #4 and #5, it cannot be assumed to be able to declare all functions and variables to be static in a MEX file. For example, I could find this being described here:

http://anadoxin.org/blog/control-over-symbol-exports-in-gcc.html

with the solution to use '-Wl,--version-script='.

Guillaume <gyom>
Fri 04 Jan 2019 07:59:41 PM UTC, comment #8:

Yeah I fully agree with comment #7.

I can confirm that building the mex files with '-Wl,-Bsymbolic-functions' results in the variable 'myglobalvar' referring to the same single location in Octave's process space when both functions are loaded.

Can you build the same mex files with Matlab using the '-Wl,-Bsymbolic-functions' link option and see whether the behavior follows there as well?

Mike Miller <mtmiller>
Project Administrator
Fri 04 Jan 2019 07:45:28 PM UTC, comment #7:

Is this then an issue best left to the programmer? If they want they can add "-Bsymbolic-functions" to their link flags or they can subtract it. It doesn't seem like an issue that Octave should be involved in (code in moctfile.cc to try and find and strip options we don't approve of).

Rik <rik5>
Project Administrator
Fri 04 Jan 2019 11:11:40 AM UTC, comment #6:

mexVersion seems to be an old symbol that does not exist any more. The nm output in comment #1 was from a MEX file compiled with an old version of Matlab:

A recent build gives:

so it seems that we could only export the mexFunction symbol and match Matlab behavior.

Using the static keyword would solve the issue here but, to my understanding, there would still be a potential issue for MEX files corresponding to, e.g., wrappers to other libraries as they would necessarily contain non-static functions (unless there is a way to specify that these functions shouldn't be exported further from the file containing the mexFunction definition). That's the point made in comment #5?

Looking further on Ubuntu, it seems that the difference in behavior is due to the link flag -Bsymbolic-functions as I then manage to reproduce the issue on openSUSE after adding it manually.

https://stackoverflow.com/questions/7216973/

Guillaume <gyom>
Thu 03 Jan 2019 09:38:13 PM UTC, comment #5:

OTOH, maybe you want a global variable that is shared among multiple files that are used to make a MEX or .oct function but you don't want it to be visible outside of the MEX or .oct function.

In any case, making the default behavior to only export the mexFunction symbol seems reasonable. From the nm output shown in comment #1 it appears that mexVersion is also exported. Are there any other symbols that Matlab MEX files may export?

John W. Eaton <jwe>
Project Administrator
Thu 03 Jan 2019 07:38:01 PM UTC, comment #4:

It's not purely the version of gcc, because I cannot reproduce this on Debian with gcc 8.2.0.

It's possible that Ubuntu is applying some different default compile or link options with their gcc 8.2.0.

I think it goes without saying that all of this is moot if you declare 'myglobalvar' to actually be a static variable so that it is properly file scoped.

Mike Miller <mtmiller>
Project Administrator
Thu 03 Jan 2019 06:14:40 PM UTC, comment #3:

Thanks for the feedback. I don't think it's a regression though: one of the tests is with Octave 4.4.0 because that's the latest version available as a package on that platform. The fact that it works for you is interesting as it could then be due to a difference in compiler, between gcc 7.3 and 8.2, even if I cannot understand how yet.

Reading more about exporting symbols in shared libraries, I wonder if mkoctfile should use an export map:

https://accu.org/index.php/journals/1372

Guillaume <gyom>
Thu 03 Jan 2019 05:02:10 PM UTC, comment #2:

I'm changing the category to Regression since this appears to have been Matlab compatible in version 4.4.0. Given the limited time before the 5.0 release I am marking this as Won't Fix on the 5.0 Bug Fix List (https://wiki.octave.org/Bug_Fix_List_-_5.0_Release#Bugs_marked_Configuration_and_Build_System).

I wonder if this is all down to different versions of compilers?

Anyways, I tested with gcc 7.30 on Kubuntu 18.04 LTS and the 5.0.1 pre-release code and it behaves the same as Matlab. Marking as "Works For Me".

Rik <rik5>
Project Administrator
Wed 02 Jan 2019 01:30:24 PM UTC, comment #1:

And here is the output of nm on the compiled MEX file.

On OpenSUSE 15.0, gcc 7.3.1:

On Ubuntu 18.10, gcc 8.2.0:

and with Matlab:

So, interestingly, the variable myglobalvar is 'd: initialized data, local' with Matlab while it is 'D: initialized data, global' with Octave.

Guillaume <gyom>
Wed 02 Jan 2019 01:02:32 PM UTC, original submission:

I notice a difference in behavior concerning global variables defined in MEX files between Matlab or Octave (openSUSE 15.0, gcc 7.3.1) on one side and Octave (Ubuntu 18.10, gcc 8.2.0) on the other side.

Using the two attached files after compilation, on Matlab or Octave 4.4.0 (openSUSE 15.0, gcc 7.3.1), one gets:

while on Octave 4.4.1 (Ubuntu 18.10, gcc 8.2.0), one gets:

This seems to indicate that globally defined variables are not exported from the (MEX) shared library with Matlab (ie behave as static) while this does not seem to be the case on the Ubuntu platform.

The compilation flags on the two platforms are copied below.

On OpenSUSE 15.0, gcc 7.3.1:

On Ubuntu 18.10, gcc 8.2.0:

Guillaume <gyom>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #45894:  foo1.cc added by mtmiller (940B - text/x-c++src)
file #45895:  foo2.cc added by mtmiller (940B - text/x-c++src)
file #45840:  mytest1.c added by gyom (239B - text/x-csrc)
file #45841:  mytest2.c added by gyom (239B - text/x-csrc)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by gyom (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-01-07 mtmiller Attached File- => Added foo1.cc, #45894
        Attached File- => Added foo2.cc, #45895
    2019-01-07 mtmiller Severity3 - Normal => 1 - Wish
        Item GroupRegression => Feature Request
        StatusWorks For Me => Confirmed
        SummaryGlobal variable in a MEX file => global variable in multiple mex or oct files collapse to one definition with -Bsymbolic-functions
    2019-01-03 rik5 Item GroupIncorrect Result => Regression
        StatusNone => Works For Me
    2019-01-02 gyom Attached File- => Added mytest1.c, #45840
        Attached File- => Added mytest2.c, #45841

    Back to the top


    Powered by Savane 3.4