bugGNU Octave - Bugs: bug #49503, [mxe-octave] cross-building...

 
 

bug #49503: [mxe-octave] cross-building packages with configure script calling octave itself have issues

Submitted by:  Avinoam Kalma <avinoam>
Submitted on:  Tue 01 Nov 2016 11:53:28 AM UTC  
 
Category:  Configuration and Build System Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Build Failure
Status:  Ready For Test Assigned to:  None
Originator Name:  Open/Closed:  Open
Release:  other Operating System:  GNU/Linux

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Tue 15 Sep 2020 06:30:34 AM UTC, comment #42: 

@Markus: The configure option "--disable-system-octave" works for my BuildBot configuration. ­čĹŹ

https://buildbot.octave.space/#/builders/2/builds/1/steps/6/logs/stdio

334 [build]    build-octave
335 [done]     build-octave

Kai Torben Ohlhus <siko1056>
Project Member
Sun 13 Sep 2020 11:02:07 AM UTC, comment #41: 

The attached changeset makes the buildbot adjustments outlined in comment #40.
I hope I got this right as I have no means of testing.

If jwe agrees that it would be ok to make those or similar changes on his buildbots, we could make the change John D proposed in comment #38 without breaking them.

(file #49793)

Markus M├╝tzel <mmuetzel>
Project Member
Sat 12 Sep 2020 10:12:00 AM UTC, comment #40: 

That should have read:
It would be necessary to configure the cross-building buildbots that create a tarball from the hg repository with "--enable-system-octave" to prevent that.
Those are:
w32-on-debian
w32-stable-on-debian
w64-32-on-debian
w64-32-stable-on-debian
w64-64-on-debian
w64-64-stable-on-debian

It would probably be ok for these buildbots to build the "build-octave" package (i.e. configure with "--disable-system-octave") or make sure the current release version is installed on the workers:
w32-release-on-debian
w64-32-release-on-debian
w64-64-release-on-debian

Not affected would be:
mxe-native-all-on-debian
mxe-native-on-debian
and all other buildbot builders that don't use MXE Octave anyway.

Markus M├╝tzel <mmuetzel>
Project Member
Sat 12 Sep 2020 10:04:20 AM UTC, comment #39: 

Re comment #38: We could do that. But then the buildbots would build and install two probably differently configured native Octave versions.
It would be necessary to configure the cross-building buildbots with "--enable-system-octave" to prevent that.
@jwe: What is your opinion on that?

Markus M├╝tzel <mmuetzel>
Project Member
Fri 11 Sep 2020 11:13:35 PM UTC, comment #38: 

Should USE_SYSTEM_OCTAVE default to the value of MXE_NATIVE_BUILD, so that its defaults to no when cross building ?

John Donoghue <lostbard>
Project Member
Fri 11 Sep 2020 05:00:36 PM UTC, comment #37: 

I added some more comments, a dependency on the "actual" octave package and pushed the patch here:
https://hg.octave.org/mxe-octave/rev/eae508c12529

Marking as ready for test.

Markus M├╝tzel <mmuetzel>
Project Member
Fri 11 Sep 2020 03:45:11 PM UTC, comment #36: 

Re comment #33, yeah it doesnt handle namespaces
I added some additional checks so that it should now - I added it to https://savannah.gnu.org/patch/?9972

John Donoghue <lostbard>
Project Member
Fri 11 Sep 2020 09:37:10 AM UTC, comment #35: 

I'm seeing recurring errors with parallel install. So attached yet another updated patch.
I hope we are getting closer to completion now.

(file #49784)

Markus M├╝tzel <mmuetzel>
Project Member
Fri 11 Sep 2020 06:42:42 AM UTC, comment #34: 

Forgot to attach the updated patch.

(file #49782)

Markus M├╝tzel <mmuetzel>
Project Member
Fri 11 Sep 2020 06:41:33 AM UTC, comment #33: 

The way of-sparsersb uses Octave in its configure script is probably one of the cases jwe was referring to in comment #19.
That kind of "mis-usage" should probably best be solved on a per-case basis.

Anyhow, attached is an updated patch that sets the Octave index size of build-octave to match the configuration for the target Octave.

@John D: That m4 script looks promising. Although it might be difficult to exactly match Octave's behavior (e.g., taking dispatching class methods and functions in +namespaces into account). Maybe you could ask for feedback by other package developers on the maintainers mailing list.

Markus M├╝tzel <mmuetzel>
Project Member
Fri 11 Sep 2020 02:56:34 AM UTC, comment #32: 

@Markus (comment #28): No more objections from my side.  I cannot find an easier solution at the moment.

Kai Torben Ohlhus <siko1056>
Project Member
Thu 10 Sep 2020 03:41:30 PM UTC, comment #31: 

Somewhat related, I had been looking at creating a script use in the windows package for configure to detect functions without needing to use octave, that looks like may works

script attached in case anyone else is interested in using/modifying it

(file #49779)

John Donoghue <lostbard>
Project Member
Thu 10 Sep 2020 01:34:34 PM UTC, comment #30: 

Thanks.

The configure script of of-sparsersb seems to execute ┬┤sizemax() > intmax ("int32")┬┤ in Octave to check if octave_idx_type is 64bit:
https://sourceforge.net/p/octave/sparsersb/ci/default/tree/src/configure.ac (currently in line 129)
There is probably a way to do that without the need to run Octave.

Anyway, the attached patch adds the conditional dependency to those packages, too.

(file #49776)

Markus M├╝tzel <mmuetzel>
Project Member
Thu 10 Sep 2020 11:51:43 AM UTC, comment #29: 

At a look at packages I have locally on my computer, others that currently use it:

windows package uses it to check for the winqueryreg function

sparsersb

optiminterp checks for it but doesnt use it

mapping checks for deg2rad functions

communications detects it but doesnt appear to use it

John Donoghue <lostbard>
Project Member
Thu 10 Sep 2020 09:08:31 AM UTC, comment #28: 

The attached patch is basically the same as the one in comment #26. It contains only some cosmetic changes.

At the moment, only the of-image and of-statistics package (conditionally) depend on build-octave. Are these the only packages that use Octave in their build process?

@Kai: Do I understand correctly that you are no longer opposed to have such a package in MXE Octave?

(file #49775)

Markus M├╝tzel <mmuetzel>
Project Member
Thu 10 Sep 2020 05:40:45 AM UTC, comment #27: 

@Markus: Thanks, this sounds good.  I'll observe when the change arrives at the mxe-repo ­čśë

Kai Torben Ohlhus <siko1056>
Project Member
Wed 09 Sep 2020 05:10:16 PM UTC, comment #26: 

The attached patch should be complete with the logic to use or not use the build-octave package.

@Kai: If we should opt to include that change, you could configure MXE Octave with "--disable-system-octave" and your buildbots should be able to build the binary of-packages correctly.
All others could just configure like before and nothing would change for them.

I don't think it is necessary to include any logic that overrides that option in case the "hg-octave-dist" target was build. That is a separate target that afaict is not pulled in by any other target. If someone builds that target, they should take care to not configure with "--disable-system-octave".

(file #49771)

Markus M├╝tzel <mmuetzel>
Project Member
Wed 09 Sep 2020 11:45:48 AM UTC, comment #25: 

For statistics, yes its only checking for existing functions using the conditional install script and creates a index file of the installed files using the update_index_file script.

John Donoghue <lostbard>
Project Member
Wed 09 Sep 2020 06:42:36 AM UTC, comment #24: 

Thinking about it, the "hg-octave-dist" target could probably override the variable that will control whether "build-octave" is built or not. That way it wouldn't be necessary to explicitly disable building "build-octave" with a configure flag if it isn't needed.

Markus M├╝tzel <mmuetzel>
Project Member
Wed 09 Sep 2020 06:27:22 AM UTC, comment #23: 

The "build-octave" package uses the tarball MXE Octave was configured to use. AFAICS, there's no need to have different flavors of that package.

Wrt the "hg-octave-dist" target, I was thinking about treating that the same as an Octave installed on the build system. The same configure flags would apply (either "--enable-system-octave" or "--disable-system-octave").

Adding some paths to the end of Octave's search path is a good idea. Unfortunately, that wouldn't work for built-in functions. AFAICT, .m-file functions (and .oct or .mex files) always shadow built-in functions no matter where they occur in the search path.

If I understand the logic in the statistics of-package correctly, it uses Octave only to check for functions existing in core Octave. IIUC, the only Octave function that seems to execute in the build process is "conditional_installation":
https://sourceforge.net/p/octave/statistics/ci/default/tree/utils/conditional_installation.m
Skimming through it, I don't see if it does anything else. Maybe John D could add some insider info.

Markus M├╝tzel <mmuetzel>
Project Member
Wed 09 Sep 2020 02:27:09 AM UTC, comment #22: 

@Markus (comment #16): Yes, I also saw this warning

https://buildbot.octave.space/#/builders/3/builds/5/steps/9/logs/stdio

and now I know about it's meaning.  Maybe "broken" is a too harsh term.  I still have a bad feeling about it, if one cannot determine which Octave is detected, by specifying an option in of-image's configure, for example, or (jwe comment #19): the Octave used for testing differs from the version run later on MS Windows.

Regarding comment #20 and comment #21:

Thanks for starting working on this. I assume the approach from file #49770 (comment #21) has to come in the three mxe flavors (default, stable, release)?  Just thinking about it, when somebody runs

make hg-octave-dist
make

https://hg.octave.org/mxe-octave/file/f55351ed06fe/hg-octave-dist.mk

Octave has to be built three times, right? ­čśô

Wasn't it easier to leave it for this release now as it is.  And kindly ask the package maintainers to add a folder containing the backup implementations to the end of the path to avoid shadowing core Octave functions (if exist?).

addpath ("backup_implementations", "-end")

https://www.octave.org/doc/v5.2.0/XREFaddpath.html

I am not sure, what does the of-statistics package really needs.  Looking at the various config files

https://sourceforge.net/p/octave/statistics/ci/default/tree/

it is a mixture of MS Windows detection, Octave version detection, runtime & install time branches... maybe currently no one knows, as the package lacks of a maintainer.

Kai Torben Ohlhus <siko1056>
Project Member
Tue 08 Sep 2020 09:31:50 PM UTC, comment #21: 

The patch adds a dependency on "build-octave" to the of-image and of-statistics that currently need it according to https://savannah.gnu.org/bugs/?59081#comment12 . (Hopefully, the right ones in the attached updated patch.)
That way, the new build-octave target can "die out" as soon as the Octave Forge packages are "fixed" and no longer depend on Octave in their build process.

(file #49770)

Markus M├╝tzel <mmuetzel>
Project Member
Tue 08 Sep 2020 09:14:32 PM UTC, comment #20: 

You are right. And I agree that binary (Octave Forge) packages should avoid using Octave in their build scripts if that is possible.

If it is reliable or not, depends on for what Octave is used in the build process: It's probably not a good idea to rely on it for configure options or similar things.
However, the image package - as an example - uses the "exist" function to check whether some functions are provided by core Octave. John D suggested to use "mkoctfile -p FCNFILEDIR" as an alternative. But that won't work for built-in functions and it relies on internal knowledge where which file is stored (which could change between versions).
AFAICT there is no alternative way how that information can be reliably queried atm.
That kind of information would also be reliable with a minimal Octave. (If a feature is disabled, the corresponding functions are still present but they throw an error or warning afaict.)

I'm attaching a work in process patch that adds a build-octave target to MXE Octave. With it, of-image builds successfully for me.
No logic yet to alternatively use an Octave that is installed on the build system. And the warning message about a missing native build of Octave is still displayed. And probably a lot of other missing parts...

(file #49769)

Markus M├╝tzel <mmuetzel>
Project Member
Tue 08 Sep 2020 08:17:10 PM UTC, comment #19: 

Even if you build a native version, the problem remains that any tests you do running that native version don't necessarily match what the cross-compiled version will do.  That would especially be true if you built it using all the options to disable external dependencies as we do with the no-extras builbot configurations.

John W. Eaton <jwe>
Project Administrator
Tue 08 Sep 2020 07:13:45 PM UTC, comment #18: 

If a system is set up to create an Octave tarball, can't we assume that the dependencies for a minimal Octave version (alike jwe's no-extras-debian buildbot) are present?

Markus M├╝tzel <mmuetzel>
Project Member
Tue 08 Sep 2020 07:10:10 PM UTC, comment #17: 

The issue with a build-octave target is that either you have to assume that all the dependancies of octave are on the native system before building, or you have to build native versions of whatever is needed as build-xxxx rules.

We could check for files existing using mkoctfile -p FCNFILEDIR to get the path that could eliminate the need to use octave

Not as easy as calling octave to do it though.

John Donoghue <lostbard>
Project Member
Tue 08 Sep 2020 06:40:10 PM UTC, comment #16: 

Are you sure the detection mechanism is broken?

I haven't checked the other packages. But the impact for the image package would have been minor. Either core functions would have been shadowed (but probably still work). Or functions would be missing.

There is also a warning during "make" if the correct version of Octave wasn't found. So I wouldn't say "it was pure luck". E.g.:

Warning - could not find native build version 6.0.91 of octave - some packages may fail, but continuing

I'd argue that detecting features is generally better than detecting versions. But it is probably difficult to come up with a tool that checks which functions exist in a certain version of Octave without installing that version. (That is why building of-image depends on Octave.) Writing such a tool is probably more complicated than just building Octave.

I'd also argue that it wouldn't make using MXE Octave more complicated if it built one more build tool (i.e. Octave itself in that case). It does that for (most of) the other build tools already.
We could also have a configure switch "--enable-system-octave" like we do for other build tools or dependencies. With that option, "build-octave" wouldn't be built. (Or maybe turn it around and only build "build-octave" with "--disable-system-octave".)

If no one opposes, I'll see if I can put something together.

Markus M├╝tzel <mmuetzel>
Project Member
Tue 08 Sep 2020 04:34:51 PM UTC, comment #15: 

The octave detection mechanism by the OF-packages is broken for the mxe-crossbuilds.  It was pure luck, that the maintainers building the mxe-crossbuilds had the "correct" Octave naively installed.  If the maintainer had Octave 4.2 or older installed, all mxe-crossbuilds were flawed.

In bug #59081 John reported, that only two packages are affected.  I agree to jwe (comment #11) to solve this on per package basis, than making mxe-octave even more complicated.

Maybe one can work with

octave-config -p VERSION

which is available during the mxe-octave package build.

Kai Torben Ohlhus <siko1056>
Project Member
Tue 08 Sep 2020 04:28:31 PM UTC, comment #14: 

One issue I haven't seen discussed is that the package builds that are using Octave for configuration info aren't really properly set up for cross compiling.  Running Octave on the build system to determine what features will be available in the cross compiled version won't always provide the correct answer.

John W. Eaton <jwe>
Project Administrator
Tue 08 Sep 2020 03:57:00 PM UTC, comment #13: 

We now have the "hg-octave-dist" target in MXE Octave which probably solved the original issue for the buildbots(?). That target builds and installs a native Octave from the hg repo and also creates a tarball.

Would it still be interesting to have a "build-octave" or "build-octave-no-extras" target? That target would build and install a minimal Octave that natively runs on the build platform (see comment #10). In contrast to "hg-octave-dist", it would use an existing tarball instead of the hg repository.

We came across this in bug #59081.

Markus M├╝tzel <mmuetzel>
Project Member
Sat 12 Nov 2016 12:04:36 AM UTC, comment #12: 

I think we could talk about what the right solution for a future release of Octave could be. I know this was touched on at least a bit when we made the move to deprecate octave_config_info. There are certain feature tests that some Forge packages need to perform in their configure scripts.

The image package needs to run a native octave process to query whether the "im2double" function is implemented in Octave itself, so that it can either provide its own implementation or not. This kind of thing is a temporary measure until it can do a hard dependency on the Octave 4.2 release for example.

The communications package runs a native octave to query the options that Octave has for building against HDF5 (HDF5_CPPFLAGS, LDFLAGS, etc), which are not available via a cross-mkoctfile tool. This could be done differently, by querying for HDF5 itself rather than relying on Octave for how to (and whether or not to) build against HDF5.

Mike Miller <mtmiller>
Project Administrator
Fri 11 Nov 2016 09:50:03 PM UTC, comment #11: 

Also, is it really necessary?  Is there no way around this for the individual packages that currently require it?  I admit that I haven't tried to look at the problem.

John W. Eaton <jwe>
Project Administrator
Fri 11 Nov 2016 09:49:10 PM UTC, comment #10: 

Hmm, good question.  For my purposes, it would just be useful to have the build-octave target to allow building packages when cross compiling and I would be able to ensure that the dependencies for that are present.

For now, I guess it doesn't matter.  I can build my own before creating the windows binaries that we distribute.

John W. Eaton <jwe>
Project Administrator
Fri 11 Nov 2016 09:28:46 PM UTC, comment #9: 

in some ways it makes sense to have a build-octave - although then do we also need build-xxxx for all the octave dependencies as well?

John Donoghue <lostbard>
Project Member
Fri 11 Nov 2016 09:04:20 PM UTC, comment #8: 

I can always install a copy of the current Octave before beginning to build the binary distribution for Windows, but why not just have a "build-octave" package that builds it for us and installs it in the mxe "usr" directory tree, same as the other build tools?  Then we can ensure that we have the proper version available.  If you agree, I can add the package.  But yes, we can also close this report.

John W. Eaton <jwe>
Project Administrator
Fri 11 Nov 2016 08:31:16 PM UTC, comment #7: 

Can we call this fixed now?

John Donoghue <lostbard>
Project Member
Mon 07 Nov 2016 07:56:25 PM UTC, comment #6: 

I had a patch in mxe-octave to ignore that issue, however it should work ok if we assume that a current native octave is installed.

John Donoghue <lostbard>
Project Member
Mon 07 Nov 2016 12:42:39 PM UTC, comment #5: 

I think the mapping pkg build has similar issues in mxe (rad2deg and deg2rad functions). Carn├ź's configure script didn't work there either.

Philip Nienhuis <philipnienhuis>
Project Member
Thu 03 Nov 2016 04:54:57 PM UTC, comment #4: 

And check and warn in cross build if don't have the correct version of octave

http://hg.octave.org/mxe-octave/rev/724749afb8b5

John Donoghue <lostbard>
Project Member
Thu 03 Nov 2016 04:36:26 PM UTC, comment #3: 

Pushed to mxe this change to fail pkg build if configure failed:

http://hg.octave.org/mxe-octave/rev/bdcbb82d57e2

John Donoghue <lostbard>
Project Member
Thu 03 Nov 2016 11:59:33 AM UTC, comment #2: 

There was a discussion on the maintainers list (I think) where it was decided it was resonalable to expect whoever was packaging the binary packages would have a native octave installed.

Having said that, currently mxe doesn't check that a native veersion is installed, and I see from doing a build myself, that mxe silently fails on the configure script if you don't have octave and packages just the install the inst files

So perhaps a check for octave and at the very least a warning given during the mxe build would be good.

Theres at least one other package in mxe that would check the octave version, however I think I patched it in mxe to just not install the file make would conditionally not install.

John Donoghue <lostbard>
Project Member
Wed 02 Nov 2016 06:02:24 PM UTC, comment #1: 

This works for me. I believe you are using the mxe octave releases and I thought there was already a bug for this but apparently not.

The issue is that during package installation, the image package checks if the function im2double already exists. If it does not, then it creates that function. I'm unsure what mxe is doing when running configure but I guess during that phase, it runs an older Octave version without im2double instead of the one that it just built.

I'm unsure if this is an issue with the image package (and there's at least the mapping package with a similar approach) or with mxe-Octave. Can anyone working on mxe comment?

Carn├ź Draug <carandraug>
Project Member
Tue 01 Nov 2016 11:53:28 AM UTC, original submission:  

>> which im2double
'im2double' is a function from the file C:\Octave\Octave-4.2.0-rc3\share\octave\4.2.0-rc3\m\image\im2double.m
>> pkg load image
warning: function C:\Octave\Octave-4.2.0-rc3\share\octave\packages\image-2.6.1\im2double.m shadows a core library function
warning: called from
    load_packages_and_dependencies at line 48 column 5
    load_packages at line 47 column 3
    pkg at line 411 column 7
>> which im2double
'im2double' is a function from the file C:\Octave\Octave-4.2.0-rc3\share\octave\packages\image-2.6.1\im2double.m

Avinoam Kalma <avinoam>
Project Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49784:  bug49503_build_octave_v5.patch added by mmuetzel (13KiB - application/octet-stream)
file #49782:  bug49503_build_octave_v4.patch added by mmuetzel (13KiB - application/octet-stream)
file #49779:  octave-funcs.m4 added by lostbard (3KiB - text/plain)
file #49776:  bug49503_build_octave_v3.patch added by mmuetzel (13KiB - application/octet-stream)
file #49775:  bug49503_build_octave_v2.patch added by mmuetzel (8KiB - application/octet-stream)
file #49771:  bug49503_build_octave.patch added by mmuetzel (8KiB - application/octet-stream)
file #49770:  bug49503_build_octave_wip_v2.patch added by mmuetzel (5KiB - application/octet-stream)
file #49769:  bug49503_build_octave_wip.patch added by mmuetzel (5KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by siko1056 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Posted a comment)
  • -email is unavailable- added by mtmiller (Posted a comment)
  • -email is unavailable- added by jwe (Posted a comment)
  • -email is unavailable- added by lostbard (Posted a comment)
  • -email is unavailable- added by avinoam (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only project members can vote.

     

     

     

    Follow 19 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-09-13 mmuetzel Attached File- => Added bug49503_buildbot_settings.patch, #49793
    2020-09-11 mmuetzel StatusPatch Submitted => Ready For Test
    2020-09-11 mmuetzel Attached File- => Added bug49503_build_octave_v5.patch, #49784
        StatusIn Progress => Patch Submitted
    2020-09-11 mmuetzel Attached File- => Added bug49503_build_octave_v4.patch, #49782
    2020-09-10 lostbard Attached File- => Added octave-funcs.m4, #49779
    2020-09-10 mmuetzel Attached File- => Added bug49503_build_octave_v3.patch, #49776
    2020-09-10 mmuetzel Attached File- => Added bug49503_build_octave_v2.patch, #49775
    2020-09-09 mmuetzel Attached File- => Added bug49503_build_octave.patch, #49771
    2020-09-08 mmuetzel Attached File- => Added bug49503_build_octave_wip_v2.patch, #49770
    2020-09-08 mmuetzel Attached File- => Added bug49503_build_octave_wip.patch, #49769
        StatusNone => In Progress
    2020-09-08 mmuetzel Operating SystemMicrosoft Windows => GNU/Linux
    2017-09-06 mtmiller Summarymxe-octave: cross-building packages with configure script calling octave itself have issues => [mxe-octave] cross-building packages with configure script calling octave itself have issues
    2016-11-17 mtmiller Item GroupRegression => Build Failure
        Release4.2.0-rc3 => other
        Summarymxe-octave: packages configure script calling octave itself have issues => mxe-octave: cross-building packages with configure script calling octave itself have issues
    2016-11-02 carandraug Summarywarning: function im2double.m shadows a core library function => mxe-octave: packages configure script calling octave itself have issues
    2016-11-02 carandraug CategoryOctave Forge Package => Configuration and Build System

    Back to the top


    Powered by Savane 3.5