bugGNU Octave - Bugs: bug #38928, configure -> hdf5 doesn't...

 
 

bug #38928: configure -> hdf5 doesn't handle path through

Submitted by:  None
Submitted on:  Tue 07 May 2013 01:57:51 PM UTC  
 
Category: Configuration and Build SystemSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Build Failure
Status: ConfirmedAssigned to: None
Originator Name: Clemens1509Originator Email: -unavailable-
Open/Closed: OpenRelease: dev
Operating System: GNU/Linux

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 21 Oct 2014 11:08:58 PM UTC, comment #26:

Rik:

I agree. I was being a bit lazy. I'll take another look.

John W. Eaton <jwe>
Project Administrator
Tue 21 Oct 2014 10:30:49 PM UTC, comment #25:

I think the change from FT2_CFLAGS to FT2_CPPFLAGS needs to be reverted. The CPPFLAGS variable is undefined, while the FT2_CFLAGS correctly has the -I options for the library.

The issue is that configure.ac uses the PKG_CHECK_MODULES macro as the interface between Autotools and pkg-config and the output of the macro is only XXX_CFLAGS and XXX_LIBS. Documentation on the macro is at https://autotools.io/pkgconfig/pkg_check_modules.html. Alternatively, we could spin up our own m4 macro to use the --cflags-only-I option of pkg-config, but this seems like more work.

I also don't think we need to add all of the CPPFLAGS (FT2_CPPFLAGS, HDF5_CPPFLAGS, Z_CPPFLAGS, FFTW_XCPPFLAGS, LLVM_CPPFLAGS) to AM_CPPFLAGS in libinterp/Makefile.am. The module.mk files in the subdirectories go to the trouble to apply the CPPFLAGS only to the files that need them. The convenience libraries in the subdirectories are then linked together into libinterp, but CPPFLAGS isn't required for that step. For instance, the parse-tree directory has no need to see the FFTW_XCPPFLAGS during compilation. Another example comes from corefcn/module.mk where there is this comment:

HDF5_CPPFLAGS probably does need to be snuck into AM_CPPFLAGS because it gets so widely included through ov.h that it is essentially used everywhere in libinterp.

I would say that the same considerations also apply to liboctave/Makefile.am. Most of liboctave does not need FFTW_XCPPFLAGS or ARPACK_CPPFLAGS. Only a few files in the numeric directory need those, and so only that convenience library is built with those CPPFLAGS. QRUPDATE_CPPFLAGS could be added to the file numeric/module.mk since that is currently missing.

CURL_CPPFLAGS should probably just go into util/module.mk since there is only one file there that uses them.

Rik <rik5>
Project Administrator
Tue 21 Oct 2014 07:36:50 PM UTC, comment #24:

Also, following up on Rik's comment #18, I started to deprecate the --with-PKG-includedir and --with-PKG-libdir options, but maybe that's not such a good idea. One nice thing about the current setup is that these individual options and the corresponding variables allow us to separate out the libraries and CPPFLAGS and LDFLAGS options that are used when building .oct files that depend on external libraries that are not linked with the Octave libraries. So keeping the individual variables makes sense to me, and if we are going to keep the individual variables, we might as well keep the configure options. Instead of removing them I tried to fix the liboctave, libinterp, and libgui Makefile.am files so that they use the PKG_CPPFLAGS variables that are needed. I checked in the following chageset:

http://hg.savannah.gnu.org/hgweb/octave/rev/18a3eaf7bdf0

John W. Eaton <jwe>
Project Administrator
Tue 21 Oct 2014 06:30:25 PM UTC, comment #23:

Yes, the Debian packages do provide pkg-config files now.

Mike Miller <mtmiller>
Project Member
Tue 21 Oct 2014 06:28:22 PM UTC, comment #22:

I followed some of the discussion when the Debian maintainers were overhauling the HDF5 packaging, but got lost somewhere along the way (it's a confusing library).

Here is the end of the conversation from back in June:

https://lists.debian.org/debian-science/2014/06/msg00014.html
https://lists.debian.org/debian-science/2014/06/msg00018.html

The first message says that since all users of HDF5 will have to rebuild against libhdf5.so.8 anyway, might as well require them to use the new -I and -L flags.

The second message (and last word on the thread) says yes, but this is still a work in progress and some things could still be done. One possibility raised was having yet another library called hdf5 that would redirect to the proper serial/parallel library, and restore the ability of users to link with just -lhdf5.

I think one of the motivators for not using alternatives is that these libraries really are not drop-in replacements of each other. If I build Octave using the serial library, it will probably work with one of the parallel libraries, but definitely not the reverse.

Mike Miller <mtmiller>
Project Member
Tue 21 Oct 2014 06:24:12 PM UTC, comment #21:

Any chance the hdf5 package on debian provides support for pkg-config?

Michael Goffioul <goffioul>
Project Member
Tue 21 Oct 2014 05:52:25 PM UTC, comment #20:

I started working on a fix for this problem. Handling the possibility of various locations for hdf5.h is not hard but I'm not sure what to do about the location of the library. Presumably it should match the header file that is chosen, but I'm not sure how to do that in a reliable way.

Again, I'm curious to know why this change was made on Debian systems without providing some alternatives-style mechanism to provide a system-wide default that just works.

John W. Eaton <jwe>
Project Administrator
Tue 21 Oct 2014 05:22:59 PM UTC, comment #19:

I'm now hitting the problem with hdf5.h being installed in /usr/include/hdf5/serial on Debian systems. Why was this change made? Should we accomodate it in the Octave configure script? It seems odd to have to put

in Octave's files, but I can do that if needed. Should we also be searching for hdf5/mpich/hdf5.h? What about hdf5/openmpi/hdf5.h? Or maybe hdf5/hdf5.h? How should we determine which one to use?

John W. Eaton <jwe>
Project Administrator
Tue 23 Sep 2014 04:30:16 AM UTC, comment #18:

Per the discussion at OctConf 2014, the --with-XXX-includedir and --with-XXX-libdir are going to be removed. To build, follow Mike's suggestion and add any necessary directories to either CPPFLAGS or LDFLAGS before compiling.

Rik <rik5>
Project Administrator
Tue 09 Sep 2014 05:31:00 PM UTC, comment #17:

As a workaround, here is how I build Octave with HDF5 in Debian unstable (which has been moved to non-standard include and lib directories):

This is similar to what I originally suggested in comment #2,
essentially using the global CPPFLAGS and LDFLAGS instead of the --with-hdf5-* options.

I think the best approach is to try to remove HDF5 from the Octave public API as much as possible, which in turn means we wouldn't have to add HDF5 flags all over the place. Since that's a big separate issue on its own, I'll open a separate bug report about that.

Mike Miller <mtmiller>
Project Member
Wed 06 Aug 2014 06:07:01 PM UTC, comment #16:

If you see a way to hide the hid_t type, then yeah, I'm all for that.

I'm also in favor of cleaning up the set of installed header files.

And cleaning up the config.h that we install (I think there is another bug report about that mess).

John W. Eaton <jwe>
Project Administrator
Wed 06 Aug 2014 06:02:30 PM UTC, comment #15:

Yes, adding either HDF5_CPPFLAGS or ALL_CPPFLAGS gets past that one rule. There are other specific .df rules in libinterp//module.mk that also need HDF5_CPPFLAGS added. And there are sources, e.g. builtins.cc, octave.cc, libgui/src, libgui/graphics, that also need it. I can try to iteratively go through and add it to the minimal number of rules that need it.

At the extreme opposite end, would it be a terrible thing to define our own "hid_t" type, it's only an int? Or to be really safe, our own opaque handle type that would map to an hid_t type? This would potentially remove the need for HAVE_HDF5 and hdf5.h to be in any public header files. From an encapsulation point of view this would be a win, it would be far better for users to not even have to see this conditional and depend on these third-party header files.

Of course now that I'm chasing which files include other files, this implies massively cleaning up the set of files installed in the include directory, I believe there are many files installed that don't need to be, for example ls-.h, ov-.cc. The ls-hdf5.h header file contains calls to HDF5 functions, but it is never needed by any external user of Octave.

Mike Miller <mtmiller>
Project Member
Wed 06 Aug 2014 05:48:08 PM UTC, comment #14:

Hopefully Mike can try some of the permutations. If you look at the .deps/.Plo files though you see that we are generating dependencies on hdf5.h for huge numbers of files which inherit through the ov.h header files.

For example, libinterp/operators/.deps/*.Plo has 127 files that depend on hdf5.h, but these don't really "depend" in the sense of actually using function prototypes from hdf5.h.

Rik <rik5>
Project Administrator
Wed 06 Aug 2014 05:39:54 PM UTC, comment #13:

Or, those rules should be using ALL_CPPFLAGS instead of just CPPFLAGS.

John W. Eaton <jwe>
Project Administrator
Wed 06 Aug 2014 05:36:44 PM UTC, comment #12:

I don't think the default pattern rule needs all the extra CPPFLAGS variables.

There are lines like

in the dldfcn/module.mk file. Can't something similar be used for those files that need the extra LLVM_CPPFLAGS or HDF5_CPPFLAGS?

John W. Eaton <jwe>
Project Administrator
Wed 06 Aug 2014 04:59:16 PM UTC, comment #11:

Oy. This looks like it might be a bigger problem than I thought. It appears that maybe the rule for generating .df files from .cc files should have every single configurable CPPFLAGS variable in it.

The code in libinterp/Makefile.am is:

How far does the build proceed if you add $(HDF5_CPPFLAGS) after $(LLVM_CPPFLAGS)?

Rik <rik5>
Project Administrator
Wed 06 Aug 2014 01:43:51 AM UTC, comment #10:

Sure, a clean build with

fails in many, many places.

I get the same missing header file error with generation of the .df files in corefcn, octave-value, with libinterp/octave.cc, libinterp/builtins.cc, libinterp/corefcn/oct-errno.cc, just to name a few.

I think the problem is basically because hdf5.h is included in the public API and leaks into all ov*.h header files. After a successful (patched) build, I count almost 400 files that depend on hdf5.h, looking in the generated .Plo and .Po files. So it may have to be added to essentially all CPPFLAGS in libinterp and libgui.

Mike Miller <mtmiller>
Project Member
Tue 05 Aug 2014 10:58:38 PM UTC, comment #9:

@Mike: Could you post the error log for the build around the problem in libinterp/corefcn? Also, what is the value of HDF5_CPPFLAGS in libinterp/Makefile? Did you re-run bootstrap and configure after updating to the new Debian locations for HDF5 files?

Rik <rik5>
Project Administrator
Tue 05 Aug 2014 10:26:37 PM UTC, comment #8:

I am now able to test this and it is not fully resolved with the changes on default.

Debian has new HDF5 packages coming down the pike that move the header files and libraries into namespaced subdirectories. Building default against these new packages fails in libinterp/corefcn with "hdf5.h: No such file or directory" so we are still missing some places where the flags are needed.

The Debian HDF5 maintainer has proposed a couple of patches to Octave to solve this issue, attached here. The first patch adds HDF5_CPPFLAGS to everything in libinterp and libgui/src, I had to add also to libgui/graphics (original patch was based on 3.8.1). With this I get a successful build. The second patch adds HDF5 flags to mkoctfile, I'm not sure if this is necessary for building Forge packages with mkoctfile or what, I haven't fully tested that path yet.

(file #31844, file #31845)

Mike Miller <mtmiller>
Project Member
Tue 14 Jan 2014 06:19:40 PM UTC, comment #7:

No response so I'll assume the patch worked. Closing report.

Rik <rik5>
Project Administrator
Wed 06 Nov 2013 12:08:34 AM UTC, comment #6:

I added the HDF5_CPPFLAGS where I think they were necessary. Hopefully the original reporter, Clemens, can verify that everything builds correctly now. Use the development branch code with a changeset >= 17858:25601fe8c6f7.

Rik <rik5>
Project Administrator
Wed 30 Oct 2013 12:41:19 AM UTC, comment #5:

Yes, the HDF5_CPPFLAGS and possibly HDF5_LDFLAGS need to be added to more modules, probably both corefcn and one or more dldfcns, possibly also the GUI, not sure how many modules touch HDF5.

Unfortunately I don't have a way to test this at the moment, but hopefully Clemens still can to verify a fix.

Mike Miller <mtmiller>
Project Member
Tue 29 Oct 2013 11:53:45 PM UTC, comment #4:

Mike, was there something we needed to change in the build system for this?

It looks like maybe the module.mk for libinterp/corefcn needs to be changed to add HDF5_CPPFLAGS.

This line is adding some special CPPFLAGS, maybe it needs more?

Rik <rik5>
Project Administrator
Mon 13 May 2013 12:05:16 PM UTC, comment #3:

ok - that workaround works. Maybe let the bug open until fixed.

Clemens <clemens1509>
Thu 09 May 2013 02:00:32 AM UTC, comment #2:

Ok, your 2nd config.log attachment looks correct to me. I think there might be a bug in the way HDF5_CPPFLAGS is not used everywhere it is needed.

Can you run configure the way I suggested and report back whether that builds successfully, in order to confirm that the problem is with the Makefile rules and not with your particular setup?

Mike Miller <mtmiller>
Project Member
Tue 07 May 2013 04:49:15 PM UTC, comment #1:

The attachment doesn't look like a complete config.log, it was interrupted. Can you attach one that completed when you got a successful configure where you were able to run make?

Can you also try configuring Octave the following way instead, this is the technique I use and it works for me regularly:

If this works and the --with-hdf5 options didn't, then there may indeed be a minor bug in one of the make files.

Mike Miller <mtmiller>
Project Member
Tue 07 May 2013 01:57:51 PM UTC, original submission:

configure looks like:
./configure --prefix=/home//dev/octave-3.6.4/install --with-hdf5-includedir=/home//dev/hdf5-1.8.10/install/include/ --with-hdf5-libdir=/home//dev/octave-3.6.4/../hdf5-1.8.10/install/lib/

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #31844:  hdf5-flags.patch added by mtmiller (1kB - text/x-patch)
file #31845:  hdf5-mkoctfile.patch added by mtmiller (2kB - text/x-patch)
file #28045:  config2.log.gz added by None (116kB - application/x-gzip-compressed)
file #28039:  config.log added by None (30kB - text/x-log)

 

Digest:
   bug dependencies.

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -unavailable- added by goffioul (Posted a comment)
  • -unavailable- added by jwe (Posted a comment)
  • -unavailable- added by rik5 (Posted a comment)
  • -unavailable- added by clemens1509 (Posted a comment)
  • -unavailable- added by mtmiller (Posted a comment)
  • -unavailable- added by None (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 15 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Mon 15 Sep 2014 03:07:01 PM UTCmtmillerDependencies-=>bugs #43226 is dependent
    Tue 09 Sep 2014 05:40:28 PM UTCmtmillerDependencies-=>Depends on bugs #43180
    Tue 05 Aug 2014 10:26:37 PM UTCmtmillerAttached File-=>Added hdf5-flags.patch, #31844
      Attached File-=>Added hdf5-mkoctfile.patch, #31845
      StatusFixed=>Confirmed
      Open/ClosedClosed=>Open
      Release3.6.4=>dev
    Tue 14 Jan 2014 06:19:40 PM UTCrik5StatusReady For Test=>Fixed
      Open/ClosedOpen=>Closed
    Mon 11 Nov 2013 07:29:51 PM UTCrik5Summaryconfugure -> hdf5 doesn\'t handle path through=>configure -> hdf5 doesn't handle path through
    Wed 06 Nov 2013 12:08:34 AM UTCrik5StatusConfirmed=>Ready For Test
    Mon 13 May 2013 12:38:06 PM UTCmtmillerStatusNeed Info=>Confirmed
    Wed 08 May 2013 06:06:55 AM UTCNoneAttached File-=>Added config2.log.gz, #28045
    Tue 07 May 2013 04:49:15 PM UTCmtmillerStatusNone=>Need Info
    Tue 07 May 2013 01:57:51 PM UTCNoneAttached File-=>Added config.log, #28039

    Back to the top


    Powered by Savane 3.1-cleanup