bugmake - Bugs: bug #60595, make doesn't always restart when a...

 
 

bug #60595: make doesn't always restart when a makefile is rebuilt

Submitter:  André Chalella <andrechalella>
Submitted:  Thu 13 May 2021 07:32:32 AM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  4.3 Operating System:  Any
Fixed Release:  4.4 Triage Status:  Medium Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 06 Sep 2021 02:46:40 AM UTC, comment #8: 

This fix has been pushed to Git.  Thanks!

Paul D. Smith <psmith>
Group administrator
Thu 03 Jun 2021 06:06:49 AM UTC, comment #7: 

I agree that the following scenario should throw an error, if that's what you meant:

- a makefile should have been read in AND
- such makefile could not be read in (e.g. does not exist) AND
- make has finished rebuilding all makefiles AND
- make has determined that it does not need to re-exec.

I don't understand why make doesn't do that today, as shown by your simple repro case (and by my real makefile when "include" is in the wrong order). Maybe it's a separate bug?

Just clear up on something: in my opinion fixing this bug alone wouldn't increase occurrences of this error, because it will only trigger more re-execs, never fewer. I say this because in your line below I got the impression that you meant that fixing this bug alone will automatically start triggering the error:

> As a note when we do fix this problem the above makefile will fail on the "include" line after the restart because the rule that says it built m2.d actually didn't.

André Chalella <andrechalella>
Tue 01 Jun 2021 12:18:54 PM UTC, comment #6: 

I understand that today, make won't throw an error.

I'm saying that after we fix this issue so that make recognizes that a file needed to be rebuilt, then make could throw an error.

Put another way, in my mind the fact that an included file can't be read is separate from the process make may use to rebuild the included file.  Or to be specific if, after make has tried to rebuild all the makefiles and determines that it doesn't need to re-exec itself, it's still the case that some makefile which should have been read in could not be read in, then that seems like an error to me regardless of whether make invoked some rule to build the makefile or not.

Paul D. Smith <psmith>
Group administrator
Tue 01 Jun 2021 04:01:04 AM UTC, comment #5: 

Thank you for all the information and for considering the issue in such careful manner!

> I assume in your real makefile you've accounted for this and something does build the target makefile (or maybe you include it with "-include").


No, when building the included makefile is skipped, touching the file is skipped entirely (i.e this part of the recipe is basically "echo Skipping; exit"). I haven't encountered any problems, that is, with regular "include" (not "-include" or "sinclude"), it works fine without a single error or even a warning: building the dependency makefile is skipped with just a custom message, then make restarts, then it correctly builds said makefile, then make restarts again, then it builds my project correctly.

I know the manual describes a failure mode like you said, but I haven't seen it happen even once. Let me explain in more detail, quoting the manual:

> If an included makefile cannot be found in any of these directories, a warning message is generated, but it is not an immediately fatal error;


I don't get any warning messages when the include files don't exist yet. Make just carries on and tries to build them before restarting. It might be worth mentioning that, in the "include" line, I specify the relative location of each included file, like: "include dep/mod/helper.d dep/program.d".

> Once it has finished reading makefiles, make will try to remake any that are out of date or don’t exist.


Yes, that happens just fine.

> Only after it has tried to find a way to remake a makefile and failed, will make diagnose the missing makefile as a fatal error.


When making the makefile is skipped, there is no fatal error, like I said. Maybe that's because the recipe terminates successfully ("exit") instead of abnormally or with an error value?

With "make -d", when a makefile is skipped as described above, make outputs "Reaping winning child" and "Successfully remade target file 'dep/program.d'", even though the file still doesn't exist and will be created only in the next restart (which occurs flawlessly). I thought this was simply because the recipe exits without error, like I said above.

For all that, it never struck me as necessary preparing for the scenario you described. Does it look like I'm doing something wrong?

André Chalella <andrechalella>
Mon 31 May 2021 01:06:08 PM UTC, comment #4: 

Thanks for the description.  Here's a much simpler repro case:


all: ; @echo RESTARTS=$(MAKE_RESTARTS)

m1.d: ; touch $@

m2.d: m1.d ; test -f $< || touch $@

include m2.d m1.d


If you run this makefile with:


rm -f m?.d && make


it will correctly restart make and print RESTARTS=1.

If you change the last line to reverse the order of include:


include m1.d m2.d


then it will not restart make and print RESTARTS=.

As a note when we do fix this problem the above makefile will fail on the "include" line after the restart because the rule that says it built m2.d actually didn't.  I assume in your real makefile you've accounted for this and something does build the target makefile (or maybe you include it with "-include").

Paul D. Smith <psmith>
Group administrator
Mon 31 May 2021 12:49:49 PM UTC, comment #3: 

I agree that this is a bug: the behavior should match the manual.

Just to note that in the next version of GNU make the order of rebuilding included makefiles will be reversed so that it happens in the "expected" way (first file is checked first).  There was an issue with portability that necessitated it.  The NEWS file for the next release says:


* WARNING: Backward-incompatibility!
  Previously the order in which makefiles were remade was not explicitly
  stated, but it was (roughly) the inverse of the order in which they were
  processed by make.  In this release, the order in which makefiles are
  rebuilt is the same order in which make processed them, and this is defined
  to be true in the GNU make manual.


Seems like we should address this bug in that release as well to avoid problems.

Paul D. Smith <psmith>
Group administrator
Sun 23 May 2021 06:24:12 AM UTC, comment #2: 

I have attached an example Makefile (with comments) that illustrates my use case. It models a project where 'program1' uses 'module1' which in turn uses 'module2'.

Dependency resolution is managed through .d files, which in the real system are autogenerated from the real source files.

The module*.d recipe creates a 'guard' file as side-effect, whose purpose is to signal to the program*.d recipe that it must wait the next restart to do real work. That is because the program*.d recipe needs the rules in module*.d correctly loaded in make. When make restarts, it removes the guard file.

In the example, 'make' makes the fake program 'program1' correctly, while 'make wrong' triggers the buggy behavior. The only difference is the include order.

I suggest following the sequence below:

    $ make
    $ touch program1.d && make
    $ touch module1.d && make
    $ touch module2.d && make

You shall see make correctly updating and rebuilding only the needed files. However, run:

    $ touch module2.d && make wrong

You will see that the build occurs before 'program1.d' is properly rebuilt, i.e make doesn't restart after updating 'module1.d', contrary to the manual. Run 'make wrong' again and you will see that only now 'program1.d' gets updated, but 'program1' has already been built, so it is too late.

(file #51463)

André Chalella <andrechalella>
Thu 20 May 2021 11:12:31 PM UTC, comment #1: 

Can you please attach a makefile which reproduces the issue?

Dmitry Goncharov <dgoncharov>
Thu 13 May 2021 07:32:32 AM UTC, original submission:  

GNU make manual, in section 3.5 How Makefiles Are Remade, says:

> After all makefiles have been checked, if any have actually been changed, make starts with a clean slate and reads all the makefiles over again.


I found the hard way this is not quite true. If, during makefile rebuilding ("Updating makefiles...."), the following conditions are met, then make won't restart, carrying on as if nothing was updated.

  1. One makefile is rebuilt as a prerequisite of another makefile;
  2. The goal makefile of #1 is not rebuilt (e.g. recipe executes but exits without modifying the file, so its mtime doesn't change).


I know this may seem an obscure corner case and maybe it is, and I apologize if that's the case. I don't have much experience with "real world" makefiles; I write mine based on the manual (which is excellent) and the rebuild-restart functionality has made possible for me to write a simple yet powerful dependency resolver for Fortran.

The thing is, it stopped working when I incidentally changed "include $(files2) $(files1)" to "include $(files1) $(files2)" somewhere in it, and it took me a couple days, many rounds of "make -d" and reading the source to understand why.

(in the example line above, $(files1) are distant prerequisites to $(files2))

I can and will (at least for now) keep the original, working order, but it is kind of the "wrong" order of files logic-wise, plus it only works because make rebuilds makefiles in the reversed order in which they were included. I know why make rebuilds in reverse and it's a valid reason, but as Paul said in another answer it is an implementation detail and I wouldn't like to rely on it.

I can provide a minimal working example if the issue is not clear enough.

I see two possible solutions:

  1. In main.c:2272 (case us_none) check current mtimes against makefile_mtimes[] and jump to re_exec if any is newer.
  2. If it must not be fixed in code, a small update on the docs to specify that makefiles rebuilt as prerequisites do not count for restarting make.


Thanks,
André

André Chalella <andrechalella>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #51463:  Makefile added by andrechalella (2KiB - application/octet-stream - Working example Makefile illustrating real use case)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by dgoncharov (Posted a comment)
  • -email is unavailable- added by andrechalella (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-09-06 psmith Triage StatusNone Medium Effort
    2021-09-06 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.4
    2021-05-23 andrechalella Attached File- Added Makefile, #51463

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code