bugmake - Bugs: bug #58735, When rebuilding makefiles, make...

 
 

bug #58735: When rebuilding makefiles, make tries them in reverse order

Submitter:  Jörg Schilling <schily>
Submitted:  Wed 08 Jul 2020 03:10:27 PM UTC
   
 
Severity:  3 - Normal Item Group:  None
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  None Operating System:  Any
Fixed Release:  4.4 Triage Status:  Small Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 19 Jul 2020 11:03:56 PM UTC, comment #7: 

A fix for the "try to rebuild included makefiles in reverse order" issue has been pushed to Git and will be in the next release.

Thanks for reporting!

Paul D. Smith <psmith>
Group administrator
Fri 10 Jul 2020 08:04:31 PM UTC, comment #6: 

I just want to be clear: changing make to allow it to be able to rebuild makefiles incrementally is not just a breaking change, it is also a sizeable development effort.

Today GNU make works in three discrete stages: first it reads all makefiles, then it organizes and completes the DAG, then it builds all the goal targets.

GNU make proceeds through each stage in order, every time.  The current codebase does not allow going back and forth between these stages.  So it is not possible to see an include operation, complete the DAG, rebuild the included file, then go back and continue reading more makefiles.

Paul D. Smith <psmith>
Group administrator
Fri 10 Jul 2020 07:36:43 PM UTC, comment #5: 


> If I provide more dependency information, gmake runs more commands in parallel when in parallel mode and this will cause more problems since only one of these commands needs to be run.


As discussed in the other bug, the right answer to this problem is to explain to make what your target's recipes really do; hiding more details from make only make it less likely to perform the build correctly.

> The general problem is that gmake implements the include feature in a way that is in conflict with the behavior of other, older implementations.


As I mentioned before, I'm not prepared to discuss decisions made by other people 32+ years ago.  If you want to debate the appropriateness of that decision you can reach out to Roland and Richard and have that conversation.  My suspicion is that they had their own ideas of how "include" should work, and didn't think that the fact some other existing (proprietary, at the time) make's "include" worked differently should keep them from implementing a better method.  But, you'll have to ask them.

At this point the disparity has existed for the aforementioned 32+ years and I have no interest in breaking GNU make makefiles in an effort to resolve it.

> You did not verify that the specific behavior of gmake is needed by any makefile in the public.


Unless I'm misunderstanding your description of the behavior, using it would break every GNU make makefile where the rule to create the include file appeared after the include line itself.  So every makefile that currently contains:


include inc1.mk
inc1.mk: ; echo FOO = bar > $@


or anything similar, will break.

> I just ask you to introduce orthogonal behavior in gmake and to evaluate existing rules for include files the same way as it is done for Makefiles: just before the file is opened the first time.


I think there's some confusion.  GNU make only has one algorithm to rebuild makefiles.  There's no difference in behavior for the initial makefile, makefiles specified on the command line, or included makefiles: they all are handled identically.

If there is no initial makefile, then obviously there are no other include files to build and make must rely completely on built-in rules.  However, it still follows exactly the same steps as when an initial makefile already exists:

1) Try to load the makefile, don't fail if it doesn't exist
2) Set the goal target(s) to all makefiles (just Makefile in this case)
3) Try to rebuild those files
4) If any file needs to be rebuilt but cannot be, then:
   * If it was not included with -include or sinclude, fail
5) If any makefile was updated, start over from scratch.
6) Else, run the build

Paul D. Smith <psmith>
Group administrator
Fri 10 Jul 2020 10:03:06 AM UTC, comment #4: 

If I provide more dependency information, gmake runs more commands in parallel when in parallel mode and this will cause more problems since only one of these commands needs to be run.

So this is also a problem that is related to an unhappy combination of contradictory behavior in gmake.

The general problem is that gmake implements the include feature in a way that is in conflict with the behavior of other, older implementations.

I cannot speak for UNIXv8 and this is not really useful to look at, since extremely few people had access to the make experiments from UNIXv8.

I however can confirm that SunOS did introduce the so called "SunPro Make" in January 1986 to a large community.

Gmake first appeared in late 1988. If gmake did have "include" from it's beginning, this would have been 3 years later than SunPro Make.

Given that gmake implements a lot of the features that have been made available to a larger community by SunPro Make in January 1986 (e.g. pattern macro expansions and pattern matching rules for implicit targets) and given that gmake has been originally developed on SunOS, it is obvious that gmake in 1988 intended to copy the features of SunPro Make.

You did not verify that the specific behavior of gmake is needed by any makefile in the public. Other make implementations use a different method and the other make implementations do not have resulting problems.

The way gmake works slows down work, as it re-execs gmake. I am however not asking you to remove this implementation detail....

I just ask you to introduce orthogonal behavior in gmake and to evaluate existing rules for include files the same way as it is done for Makefiles: just before the file is opened the first time.

BTW: SunPro Make is not something from the past that no longer exists and it is (as a result of a request from me) OpenSource since December 2006. I made it highly portable and maintain it since 9 years and it is part of the "schilytools" bundle; together with smake, the make implementation I wrote in 1984 and maintain since then.

So everybody can check the behavior of SunPro Make.

Jörg Schilling <schily>
Thu 09 Jul 2020 07:24:50 PM UTC, comment #3: 


> This wrong order causes gmake or the compiler to try to access files that have not yet been made


Well, this will only happen if you do not declare the dependencies of your included makefiles.

If you need to have inc2.mk built before inc1.mk can be built, then you should specify that in the makefile:


include inc1.mk
inc1.mk : inc2.mk ; build inc1.mk using inc2.mk


If you provide dependency information in your makefile, then the order in which makefiles are attempted to be rebuilt is irrelevant.

I have no problem inverting the order, but any makefile which relies on ordering of goal targets, rather than explicit dependency relationships, is brittle at best.

> Make always executes the commands for all Makefiles and other files it reads via the include directive just before it actually tries to open the related file for the first time. This happens, whenever a related rule exists at that time that has already been read from a Makefile, from an included file or from the so called built-in rules.


I'm not sure where this quoted text comes from.  It doesn't come from anything I ever said, or anything in the GNU make manual.  GNU make has never worked like that, and it will never work like that.

> Just to remind you, this is a bug that I reported (in a partly different way) in 1998 already and at that time, you admitted this is a bug


Unfortunately, I do not have archives going back past 2000 available to me.  If you can point me to such archives that would be helpful.  However, there must have been a miscommunication.  At no time have I ever considered make's meta-behavior here (reading all makefiles first, then rebuilding the out of date ones) to be a bug.

> Given that the include directive is a make feature that exists longer than gmake, I expect gmake to implement the orthognal behavior people know from other, older implemenations.


Well, I can't speak to what Roland or Richard knew or did not know, or should have known, about "other implementations" at the time they implemented GNU make's current behavior back in the 1980's.  However, GNU make's current algorithm for rebuilding out of date makefiles is well-defined and has important advantages over the model you quote above, and those advantages are actively being used by (many) thousands of makefiles.  Changing GNU make's "include" to work in the way described above is completely out of the question.

I'm sorry if an email I sent caused you to think that there was a chance that this would happen someday: I certainly didn't intend to imply that.

Paul D. Smith <psmith>
Group administrator
Thu 09 Jul 2020 02:02:38 PM UTC, comment #2: 

Thank you for confirming that gmake executes the commands for out of date targets that result from the include directive in the inverse (which is the wrong) order. This wrong order causes gmake or the compiler to try to access files that have not yet been made - or causes gmake to complain they are missing even though they already exist at the time, the error message is displayed (see my other bug report).

*
Make always executes the commands for all Makefiles and other files it reads via the include directive just before it actually tries to open the related file for the first time. This happens, whenever a related rule exists at that time that has already been read from a Makefile, from an included file or from the so called built-in rules.
*

Unfortunately this is not how gmake works under all circumstances. For Makefiles, this also applies to gmake, a Makefile that is not present, but may e.g. be retrieved via SCCS is correctly retrieved by gmake before it opens that Makefile for reading. But for files related to the include directive, you have been describing the behavior currently implemented by gmake, which is incorrect.

Just to remind you, this is a bug that I reported (in a partly different way) in 1998 already and at that time, you admitted this is a bug that would take some time for fixing. 22 years is "some time", so when will this bug be fixed?

Given that the include directive is a make feature that exists longer than gmake, I expect gmake to implement the orthognal behavior people know from other, older implemenations.

My general problem here is that gmake has more than a single bug that cause problems on platforms that come with gmake installed as as "make". The problems I see, partially result from attempts to work around the gmake problems. Some of the problems appear in serial mode, others in parallel mode. I currently have a workaround concept for gmake that works in serial mode, but not in parallel mode and people are asking for parallel compilations.

Jörg Schilling <schily>
Wed 08 Jul 2020 04:34:17 PM UTC, comment #1: 

I'm not sure I understand the issue being reported: an example would help.

Make works by first reading the initial makefile(s) and all (existing) included files.  During this step it doesn't complain about missing makefiles.  Then after that is all completed, make tries to rebuild all the makefiles (existing or not).  If any fail to rebuild (and were not included with "-include" / "sinclude"), then make fails.  If all were rebuilt, then make starts over from scratch and re-reads everything.

It is true that currently (due to an implementation detail) they are built in reverse order, as if you'd typed "make inc3.mk inc2.mk inc1.mk Makefile".  If you have correct dependencies for your included makefiles this shouldn't matter, but that can easily be changed so they are built in the same order they were encountered.

Of course if you enable parallelism you really have to have correct dependencies anyway or you can't be sure what order things will actually run in.

Is the reverse order the issue you're concerned with?  I'm not sure what you meant by "wrong include concept" reported in 1998.

If you mean, you think that if make encounters a missing included file, it should stop and try to build that makefile immediately, then continue to read, well, that's one way it could work but that's not how it works.  There are some advantages to that, but there are also advantages to the way make works today.  I haven't seen anything showing it would be worth the huge backward-compatibility break to change the way make works in this respect.

Paul D. Smith <psmith>
Group administrator
Wed 08 Jul 2020 03:10:27 PM UTC, original submission:  

gmake, when in parallel mode, executes commands for out of date files needed by the include directive in a random order instead of executing them in the order needed by the include directives.

It even tries to read the "include file" before "file" has been created by the related rule command.

This causes files to be frequently created after the "include" directive that likes to read these files is run.

This is related to a bug report about a wrong include concept in gmake that I made in 1998 already.

BTW: rules for out of date makefiles that are on the command line are handled correctly.

Jörg Schilling <schily>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by schily (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-07-19 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Operating SystemNone Any
        Fixed ReleaseNone 4.4
        Triage StatusNone Small Effort
    2020-07-09 psmith Summaryif depending on include, gmake does not execute commands for out of date targets in the right order When rebuilding makefiles, make tries them in reverse order

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code