Fri 01 Jul 2016 04:24:13 PM UTC, comment #28:
I think that even if we integrate libgnu into the non-recursive build, there will still be one recursive call to Make because of BUILT_SOURCES.
It is possible to use GNU Make's MAKELEVEL variable to set the variable value only once when the top-level Makefile is executed. See the attached makefile for an idea of how we might do that. But the only way I see to make this work requires using "ifeq" in the Makefile and automake doesn't like that.
(file #37653)
|
Fri 01 Jul 2016 03:44:32 PM UTC, comment #27:
Okay, this is a small issue. I'm going to close this report.
|
Fri 01 Jul 2016 03:35:29 PM UTC, comment #26:
I think the reason for the HG-ID messages is
1. The script to create or recreate the HG-ID file now only runs once as a side effect of it being sourced by Makefile.
2. However, our build sources Makefile multiple times still, because of the recursion into libgnu. Makefile is evaluated once for "make all", then again for "make all-recursive", and a third time for "make all-am".
If we can successfully integrate libgnu into our non-recursive make setup (I worked on that some, but got blocked by some problems in gnulib), there will only be one make invocation and only one of those messages.
|
Fri 01 Jul 2016 03:27:51 PM UTC, comment #25:
Testing again today (7/1/16), I see it has gotten much better. The only issue now is multiple warnings about HG-ID. I'm a developer, so I can deal with it, but that would also be nice to reduce down to a single warning.
Steps to Reproduce:
The second invocation of make on a newly built tree produces the following log.
|
Fri 01 Jul 2016 02:52:14 AM UTC, comment #24:
I think this has gotten much better with recent massive changes to the build system. There are still a lot of files that need to be rebuilt to see if the contents have changed, but there is not much of a way around that with make. I think the problems that this report originally mentioned have all been solved. I frequently rebuild in the same directory while pulling in new changes and I haven't noticed any need to run make more than once (other than once this morning that jwe addressed with http://hg.savannah.gnu.org/hgweb/octave/rev/5b521e104f2a)
|
Sat 05 Mar 2016 01:34:20 AM UTC, comment #23:
Oh, it worked for me because none of my options had embedded spaces. I checked in this change as a quick fix, but I'm afraid there could be other quoting problems lurking.
http://hg.savannah.gnu.org/hgweb/octave/rev/9429c39994f8
I also just noticed that I forgot to handle OCTAVE_LINK_DEPS, OCTAVE_LINK_OPTS, OCT_LINK_DEPS, and OCT_LINK_OPTS. These definitions come from link-deps.mk files. I have to either extract that info from those files at configure time or move the info into the configure script and I'm not sure what to do about that. I will try to fix it over the weekend.
|
Fri 04 Mar 2016 10:04:07 PM UTC, comment #22:
I'm getting a new message during compilation
When I look at the offending line in subst-config-vals.sh I see
The problems seems to be the environment variables CFLAGS, CXXFLAGS, and FFLAGS which are all set in the shell before the call to ./configure.
|
Fri 04 Mar 2016 08:57:17 PM UTC, comment #21:
I checked in the following chanagesets:
http://hg.savannah.gnu.org/hgweb/octave/rev/df859661e31e
http://hg.savannah.gnu.org/hgweb/octave/rev/7e8a410e6124
In the past, the build system was trying to handle things like
but I don't think this was ever really working correctly. Now it won't work at all as the new setting will be ignored, so files that have the value of "libdir" inserted into them won't be updated, and if you try to do make install with a modified variable like this, even if the installation step succeeded, Octave will not have matching information compiled in so it will not run properly
I don't think we've lost much feature-wise, but setting variables like this on the make command line doesn't currently generate any warning or error, so that's probably not good.
I'm not sure of the best way to go about generating an error. I mean, there are a lot of variables that could be set, and I don't know that we have a good way to maintain the entire list. If we did, maybe we could have make check the definitions of all the variables against what was set at configure time and issue an error if there is a mismatch? Is it worth the trouble?
You should see some improvement if you have to run configure (or config.status) again for an existing build because generated files do not depend on Makefile. And just adding a new file to a module.mk file should not trigger a complete rebuild of everything since the timestamps of config.h, oct-conf-post.h, and other generated files should not change just because some module.mk file changed.
|
Wed 02 Mar 2016 08:09:26 PM UTC, comment #20:
Sorry, I was missing the part where it would be generated up front by configure. That sounds reasonable to me.
|
Wed 02 Mar 2016 06:50:44 PM UTC, comment #19:
We already do what I suggested for the cruft.def file using the mkf77def script. I don't see why we shouldn't do this kind of thing for other generated files. I'll experiment a little and see if it helps.
|
Wed 02 Mar 2016 06:41:07 PM UTC, comment #18:
Another possibility is to have configure generate scripts containing subsets of configuration variables that are needed to generate things like defaults.h. Then defaults.h could depend on the script instead of Makefile and changing Makefile in some way that is unrelated to the configuration variables that go into defaults.h wouldn't cause defaults.h to be rebuilt.
If configure uses move-if-change when generating the script, then simply running configure again would not cause defaults.h to be touched unless there was some change that actually affects defaults.h.
|
Wed 02 Mar 2016 06:35:34 PM UTC, comment #17:
The problem with the current setup is that the configuration info is stored in the Makefile but Makefile often changes in other ways that don't affect the generated files. That's why move-if-change is used, to avoid a cascade of building other things when no real change has happened.
One solution is to generate the files at configure time and break the dependency on Makefile. But then we need to prevent inconsistencies if configuration variables are modified on the Make command line.
Regarding your version.h example, that file would be generated by configure, so it would be up to date in the sequence of events that you describe.
|
Wed 02 Mar 2016 06:31:28 PM UTC, comment #16:
Mike, I don't understand your comment about using stamp files. For single file targets we shouldn't need stamp files. We would just use mv instead of move-if-change and then use the timestamp on the target file.
|
Wed 02 Mar 2016 05:50:45 PM UTC, comment #15:
I didn't think that any rule that used move-if-change was silent before. I thought they always wrote out "TARGET is unchanged".
|
Wed 02 Mar 2016 05:43:51 PM UTC, comment #14:
But honestly I don't have a problem with a few extra "GEN" lines that make it look like there is still something being built. I think the difference is those rules used to be completely silent but make noise now that we've gone to the Automake silent-rules setup.
|
Wed 02 Mar 2016 05:42:07 PM UTC, comment #13:
I think removing the Makefile from the dependency chain is the opposite of what you guys are talking about, unless I'm completely confused. Let's say libinterp/version.h does not depend on Makefile. Then if I re-run configure and build from a dirty build tree, version.h will not be regenerated if it already exists and I'll have old definitions in the header file. Depending on Makefile is exactly what you want if you don't care about the possibility of someone entering variables on the command line and forcing that to rewrite a file. The only problem here is the files that are rewritten to the same contents and then the timestamp is not updated to avoid excessive recompiles.
A way around this would be stamp files, dirtying up the build tree with a half a dozen or so extra files, but relying on proper system timestamps to ensure make calculates build order correctly.
|
Wed 02 Mar 2016 04:47:14 PM UTC, comment #12:
I think that solution would be quite acceptable. Better yet, this appears to already be baked in to the Autotools suite when the the cache is enabled for configure.
I just tried the following
This correctly raises an error, points outs what variables are different, and suggests two resolutions.
The only wrinkle here is that you have to throw the -C option when running configure the first time. Maybe there is a way to automatically specify that in configure.ac?
|
Wed 02 Mar 2016 04:11:47 PM UTC, comment #11:
Yes, the current build system tries to do the right thing if someone attempts something like
because the configuration variables are stored in the Makefile and many are substituted into source files.
I don't really care about changing the configuration variables on the make command line, so simply removing the Makefile from the dependency chain wouldn't bother me much. But is there a reliable way we can detect this situation and refuse to continue building? I guess I'd like to see something like
The reason for making this an error rather than just a warning is that things will not work if Octave has one value for something like "fcnfiledir" built in to it but the Makefile used another value as the location for installing .m files.
|
Wed 02 Mar 2016 06:02:41 AM UTC, comment #10:
I think it should be possible to decouple some of the dependencies on the Makefile. If I remember correctly, this was partly a result of the structure of our old build system and no longer applies.
Most programs use configure to change variables that need to be modified rarely. I think the Octave system may be trying too hard to customizable and granting the ability to change things on a per make invocation rather than just a per configure invocation.
|
Wed 02 Mar 2016 05:37:46 AM UTC, comment #9:
OK, I think I understand now. It's not just that they are listed in both BUILT_SOURCES and as dependencies of programs, but that they are constructed by rules that use move-if-change. Because of that, their timestamps are not updated. So Make attempts to build them more than once because they (nearly always) appear to be out of date.
I don't see how to solve that problem without giving up some (most?) uses of move-if-change. That would be OK, but will currently cause lots of things to be rebuilt if the Makefile changes because the generated files depend on the Makefile because that's where the information that is substituted into the generated files comes from. Maybe we can decouple some of this configuration info from the Makefile?
|
Wed 02 Mar 2016 04:51:57 AM UTC, comment #8:
Sorry, it's the BUILT_SOURCES that are also foo_SOURCES for some program listed in ALL_LOCAL_TARGETS.
For example
Each of those files are prerequisites of the octave-gui program, and also in BUILT_SOURCES which makes them prerequisites of the "all" target. The reason they show up twice is because "make all" forces its prereqs first and then calls make recursively.
|
Wed 02 Mar 2016 04:42:12 AM UTC, comment #7:
Which targets are listed in both BUILT_SOURCES and ALL_LOCAL_TARGETS? Maybe I'm missing something, but I think the only in both lists is run-octave, and I guess it could be removed from ALL_LOCAL_TARGETS if it is needed in BUILT_SOURCES.
|
Wed 02 Mar 2016 12:58:12 AM UTC, comment #6:
I think the reason these files are built twice in a single make is because they are both BUILT_SOURCES and ALL_LOCAL_TARGETS. The "all" target depends on BUILT_SOURCES as a prerequisite (so that builds them once) and then does a $(MAKE) all-recursive (which builds them a second time).
I'm able to reproduce this one easily by doing a clean build, then "touch Makefile" and "make all" again. All of the files in question have Makefile as a prerequisite, but their timestamps are not updated when they are built.
|
Tue 01 Mar 2016 10:46:26 PM UTC, comment #5:
Making libgnu non-recursive is definitely doable. I made some effort on that front a couple months ago with gnulib upstream, but haven't gotten to a solution yet. Gnulib has a non-recursive hack, but much of it assumes the library directory is called "lib".
I see what you're saying about the attempt to build these files twice in a single make run.
|
Tue 01 Mar 2016 10:36:44 PM UTC, comment #4:
I think there is something screwy with the DOCSTRING build as well. If you can narrow it down to something repeatable I'd file a new bug report about that.
The issue in this report is that make shouldn't need to call the move-if-change rule twice on the same file.
I think it is related to the use of BUILT_SOURCES. For example, when I run
The first file that gets built is defaults.h using move-if-change. A whole bunch of these other BUILT_SOURCES get constructed and then this:
It would appear that it finishes up the last of the BUILT_SOURCES and then executes 'make all-recursive'. This starts with the directories mentioned in SUBDIRS, only libgnu these days, and then changes to the top-level directory and starts building all of the BUILT_SOURCES again.
One way out of this, I imagine, would be to ditch SUBDIRS completely and have just a single Makefile. But this means somehow creating a module.mk file for the libgnu directory.
Just for grins, I commented out SUBDIRS. It isn't strictly necessary after my changeset today:
Octave builds just fine, but it still uses the move-if-change rule twice on lots of files.
|
Tue 01 Mar 2016 10:06:04 PM UTC, comment #3:
I've experimented with "touch"ing those files and running make again, lots of things are recompiled, but if I keep repeating runs of touch and make, eventually it gets to the point where nothing shows up as needing to be rebuilt.
Maybe related (I don't want to hijack this bug, though) is the doc/interpreter/.texi files are rebuilt twice almost every time I build using parallel make. I think this has something to do with the DOCSTRING_FILES rules and that they can be updated after the .texi are built, which then forces them to be rebuilt again.
|
Tue 01 Mar 2016 09:53:54 PM UTC, comment #2:
Are you sure the files are actually being rebuilt?
I see these "GEN" lines multiple times, but it's because the rules use the move-if-changed function, which tries to build the file again but doesn't update the timestamp on the file if its content doesn't need to be updated. This leaves the file with its old timestamp, so make perpetually thinks it needs to be rebuilt, but the build rule does not update it.
|
Tue 01 Mar 2016 07:30:57 PM UTC, comment #1:
This is still an issue as of ad1dbbd23f1d (3/1/16).
The list of files created twice for every build is:
|
Sun 27 Sep 2015 07:35:53 PM UTC, original submission:
Attached is the result of 'make |& tee mymake.log' after Octave has already been built. It should be straightforward, since there is nothing to actually build, but instead there are many files which are generated twice and two files which are generated three times. In all cases nothing has changed so move-if-changed rule does not replace the file. But this seems like a lot of unnecessary overhead and probably indicates that we don't have the build system fully configured correctly.
mymake.log : log file of build
duplicates.list: shows files built multiple times along with repeat count.
|