bugGNU Octave - Bugs: bug #54260, mkoctfile doesn't export some...

 
 

bug #54260: mkoctfile doesn't export some Octave features that packages need at build time

Submitter:  Mike Miller <mtmiller>
Submitted:  Fri 06 Jul 2018 05:47:35 PM UTC
   
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * GNU/Linux
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 10 Nov 2018 03:43:22 PM UTC, comment #20: 

@Mike: I assume you have completed the activities for comment #19 about opening any new bugs for remaining issues. 

The bug in this report was resolved in July, 2018.  Closing report.

Rik <rik5>
Group administrator
Thu 12 Jul 2018 07:37:47 PM UTC, comment #19: 

I will take responsibility to go through the discussion here and open new bug reports for the remaining issues.

Mike Miller <mtmiller>
Group Member
Thu 12 Jul 2018 04:42:44 PM UTC, comment #18: 

Can this bug be closed?  The issue in the title has been addressed.  Perhaps additional bug reports should be opened if individual packages like strings and communications need updates.

Rik <rik5>
Group administrator
Sat 07 Jul 2018 02:39:54 AM UTC, comment #17: 

This is good, I think almost all of the items in my original list can be resolved now by using mkoctfile instead of octave-config or octave --eval. We can update strings and communications to detect PCRE and HDF5 on their own.

I think some more small documentation updates and organization can still be done, and we can look at what can be removed to simplify, and then figure out when and whether to deprecate the octave-config program.

Mike Miller <mtmiller>
Group Member
Fri 06 Jul 2018 11:23:24 PM UTC, comment #16: 

I pushed this change for mkoctfile and the scripts used to insert the configuration variables in it:

http://hg.savannah.gnu.org/hgweb/octave/rev/cbef9fc53fef

John W. Eaton <jwe>
Group administrator
Fri 06 Jul 2018 08:30:53 PM UTC, comment #15: 

I was mistaken, we have pkg-config files for Octave 5, not Octave 4.4.

Mike Miller <mtmiller>
Group Member
Fri 06 Jul 2018 08:26:22 PM UTC, comment #14: 

The major reason to keep BLAS_LIBS and LAPACK_LIBS is because of the whole BLAS alternatives issue, where a system may have MKL and OpenBLAS and ATLAS, but a package building against Octave should use exactly the same link flags that Octave was built with to ensure that there are no conflicts.

Mike Miller <mtmiller>
Group Member
Fri 06 Jul 2018 08:13:23 PM UTC, comment #13: 

A few external (non-Forge) projects do use the FFTW symbols from mkoctfile, so to be safe I would keep those for now.

Mike Miller <mtmiller>
Group Member
Fri 06 Jul 2018 08:12:03 PM UTC, comment #12: 

I think AR and RANLIB are obsolete static library build tools and are good candidates for removal. The 'ann' Forge package uses 'AR' and 'ARFLAGS', but this package is unmaintained, not a problem. Notice that the scripts/pkg/private/configure_make.m function file actually sets both AR and RANLIB in the environment using mkoctfile.

BLAS_LIBS, LAPACK_LIBS, and FLIBS are used by several packages. Since BLAS and LAPACK are required core dependencies, as opposed to optional dependencies like FFTW and HDF5, I think it makes sense to keep these three.

LIBS, READLINE_LIBS, and SED are probably safe to remove.

DEPEND_EXTRA_SED_PATTERN is only defined on Solaris, I don't even see it used in the build system at all, maybe it can be deleted from the build completely.

Mike Miller <mtmiller>
Group Member
Fri 06 Jul 2018 07:52:05 PM UTC, comment #11: 

Agree with documenting which are used internally by mkoctfile and may be set by the user.

I started by looking at what variables are actually used in mkoctfile, but I can move variables from octave-config first and split into groups of "used internally and may be set by the user", "unused internally, but may be set in the environment", "fixed internal Octave configuration variables".  For the second category, I'm thinking of things like AR, RANLIB, SED, etc.  I see these as candidates for removal.

Here are the variables I found that are not used in mkoctfile:


AR
BLAS_LIBS
DEPEND_EXTRA_SED_PATTERN
FFTW3F_LDFLAGS
FFTW3F_LIBS
FFTW3_LDFLAGS
FFTW3_LIBS
FFTW_LIBS
FLIBS
LAPACK_LIBS
LIBS
RANLIB
READLINE_LIBS
SED


John W. Eaton <jwe>
Group administrator
Fri 06 Jul 2018 07:35:28 PM UTC, comment #10: 

I definitely agree about making it clear which variables can be overridden from the environment, but we can still do all of that in mkoctfile, and just document that certain variables are intended to be defaults with an override available and others are not.

I would rather look at adding things to mkoctfile first, then we can look at which things can be removed. I will have to do a wider search to see if any other external projects are using FFTW3_LIBS.

I just did a quick search and found that a project called shogun definitely does use 'octave-config -p BINDIR' for example.

Mike Miller <mtmiller>
Group Member
Fri 06 Jul 2018 07:28:00 PM UTC, comment #9: 

I think the build-time logic of 32-bit vs 64-bit indexing integers might also be solved by simply using OCTAVE_ENABLE_64 from the public API as a replacement for the old USE_64_BIT_IDX_T, does that look right? As far as I can tell sparsersb is the only package that has conditional code that depends on this at build time.

Mike Miller <mtmiller>
Group Member
Fri 06 Jul 2018 07:27:10 PM UTC, comment #8: 

Looking at it now, mkoctfile has variables like RANLIB, AR, CXX, CXXFLAGS, etc., that it uses and that the user could override if needed.  Having "mkoctfile -p CXX" display that variable is useful as a way of saying "this is the default for the C++ compiler that mkoctfile will attempt to use".  But there is, of course, no guarantee that these things will exist on the system where mkoctfile runs.

It seems to me that the other variables like the version and installation directories also really only make sense as read-only configuration variables.  Most of these are currently only in octave-config.  Probably we should make some distinction between variables that users may override and which are for informational purposes only.

The FFTW3 variables should probably just be removed.  Is there any justification for keeping them?

Should we keep system-wide variables like BINDIR, EXEEXT, MANDIR, MAN1DIR, etc.?  I don't see why, if we are not providing info about things like FFTW, HDF5 or PCRE.

John W. Eaton <jwe>
Group administrator
Fri 06 Jul 2018 07:11:53 PM UTC, comment #7: 

I can make the changes for mkoctfile.

John W. Eaton <jwe>
Group administrator
Fri 06 Jul 2018 07:11:30 PM UTC, comment #6: 

"Oh and we already do have pkg-config files starting with Octave 4.4."

Shows you what I know!  :-)

John W. Eaton <jwe>
Group administrator
Fri 06 Jul 2018 07:05:10 PM UTC, comment #5: 

Oh and we already do have pkg-config files starting with Octave 4.4.

Mike Miller <mtmiller>
Group Member
Fri 06 Jul 2018 07:04:20 PM UTC, comment #4: 

I don't think Octave needs to provide a .m4 file as a service to other packages. I think it's fine if we say that packages that use HDF5 to interface with Octave just need to detect it on their own. Both PCRE and HDF5 have pkg-config files these days, there is no need for Octave to provide autoconf logic for those libraries.

I'm fine with changing the communications package to find HDF5 on its own, and PCRE should be easy as well for the strings package.

Let's just add the bare minimum that we need to mkoctfile to make it equivalent to what people actually use from octave-config. That does include the various "DIR" variables like LOCALFCNFILEDIR and LOCALOCTFILEDIR. VERSION and CANONICAL_HOST_TYPE would also be nice to add. OCTINCLUDEDIR is already available with mkoctfile, so gsl can start using that today.

And something to easily indicate whether octave_idx_type is a 32 or 64 bit integer.

Mike Miller <mtmiller>
Group Member
Fri 06 Jul 2018 06:55:53 PM UTC, comment #3: 

Yes, octave-config was meant to be like those other -config programs.  That was a popular thing to do before pkg-config emerged as a crowd favorite.  We could also provide a pkg-config file for Octave.  That can also be used for cross compiling, correct?

Either way, as a first step we could certainly eliminate the duplication by deprecating octave-config and having mkoctfile do everything that octave-config currently does.

For Octave dependencies, there is no way to guarantee that the dev libraries and tools are available just because they were available on the system where Octave was built.

But we could put our configure checks in separate .m4 files and distribute them so that people can easily use the same tests we do in Octave.

John W. Eaton <jwe>
Group administrator
Fri 06 Jul 2018 06:43:51 PM UTC, comment #2: 

Agree with all of your points, especially about not using Octave to get features needed. But do you mean that for HDF5 for example, should a package use its own configure logic, or should HDF5 flags be added to mkoctfile for packages to use?

I think octave-config was meant to be analogous to other -config programs, (fltk-config, pcre-config, mysql_config, etc) to just report any information that might be useful to someone who wants to build against Octave. mkoctfile started out as just a compiler wrapper, but over time it has added more variables that might be useful, so there is some overlap now.

My preference would be to move more into mkoctfile as needed, and deprecate octave-config, so there is one tool that does everything needed for building something against Octave. But I don't have a strong opinion about whether mkoctfile should report PCRE_LIBS and HDF5_LIBS, or whether packages should write their own configuration logic using autoconf for those.

Mike Miller <mtmiller>
Group Member
Fri 06 Jul 2018 06:10:47 PM UTC, comment #1: 

LOCALVERFCNFILEDIR, LOCALVEROCTFILEDIR, LOCALVERARCHLIBDIR are artifacts of the old configure/make file that felt the need to show where things were gooing to be installed, but arent needed.


gsl is doing a temporaty set of CPPFLAGS=-I so that the configure's AC_CHECK_HEADER will work


just out of interest, what is/was the intent of octave-config? or is that tool going to disappear over time ?

or do some things that are direct build settings need be in mkoctfile, others that are more of just how octave was configured?

Perhaps there needs to also be more spearation between octave-config and mkoctfile ?


My vote at the very least is not to use octave for compile options

John Donoghue <lostbard>
Group Member
Fri 06 Jul 2018 05:47:35 PM UTC, original submission:  

Several Octave Forge packages use some equivalent of "octave --eval", or the octave-config program, to get information about how Octave was configured or built. I'm trying to summarize the features that all packages need (for now), so they can potentially be added to mkoctfile to have one place where all packages can get the information they need.

  • CANONICAL_HOST_TYPE is only exported by 'octave-config -p', not 'mkoctfile -p'. Both database and fl-core (unmaintained) check CANONICAL_HOST_TYPE to see if it contains 'mingw' to add certain libraries to the link stage. The gsl package uses 'octave --eval' to get the same variable for similar purposes. The windows package uses it for OS-dependent logic (how to copy files, whether to strip object files).


  • VERSION is also only exported by 'octave-config -p', not 'mkoctfile -p'. is used by both the optiminterp and windows package to define a HAVE_OCTAVE_440 preprocessor symbol, for example, but I don't see that those symbols are actually used in any code.


  • LOCALVERFCNFILEDIR, LOCALVEROCTFILEDIR, LOCALVERARCHLIBDIR are all again from 'octave-config -p', used only by the windows package, and I'm not sure why it needs them. I see that they are AC_SUBST'd, but I don't think the build uses them now, maybe once upon a time.


  • PCRE_CPPFLAGS and PCRE_LIBS are used by the strings package, and they are queried with 'octave --eval'. Should a package rely on how Octave determined it was to be built against an external library, or should it use its own logic?


  • HDF5_CPPFLAGS, HDF5_LDFLAGS, and HDF5_LIBS are used by the communications package and they are queried with 'octave --eval'. Same question as above, should a package rely on Octave to tell it how to link with external libraries?


  • octincludedir is used by the gsl package to add a -I option to CPPFLAGS. Shouldn't this be implicit in mkoctfile? I'm not sure why this is needed at all.


  • Whether the index integer is 32-bit or 64-bit can only be derived with 'octave --eval' at the moment. The sparsersb package uses that at build time to add a preprocessor symbol to conditionally compile some functions one way or another. Is there or can we add an easier way for a package to know the size of the octave_idx_type at build time?


A lot of these are questions of the form "Do we need this?"

Anyone may add more to this list if I missed any features that Octave Forge packages currently use at build time that can't be derived from mkoctfile alone.

I'm leaving out a big category of packages, which are packages that ask Octave whether a function or several functions are defined on the core load path, which it then uses to conditionally install its own m-files. This happens when a function is moved from Octave core into a package, or vice versa, and the package wants to be compatible with more than one release while not abiding annoying 'shadows a core library function' warning messages. I think that requires a different general treatment for cross-compiling than the goal of this bug.

Mike Miller <mtmiller>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by mtmiller (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-11-10 rik5 StatusConfirmed Fixed
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code