bugmake - Bugs: bug #56892, toxic combination of -include and...

 
 

bug #56892: toxic combination of -include and match-anything rules

Submitted by:  David Boyce <boyski>
Submitted on:  Sat 14 Sep 2019 08:28:32 PM UTC  
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.2.1 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 14 Sep 2019 09:29:54 PM UTC, comment #3: 

I can think of one way to address these types of issues that might not be too impactful:

Make knows both how deep inside a recursion it is (via MAKELEVEL) and also how many times its re-exec'd itself (via MAKE_RESTARTS).  It could check to see if either of those values seems outrageous and stop.  For example, if our MAKELEVEL is > 100 or our MAKE_RESTARTS > 1000, or something, we'd fail with an error.

I don't know if we'd need to provide a way for the user to reset these limits in case they had a legitimate need for "outrageous" values.

Paul D. Smith <psmith>
Project Administrator
Sat 14 Sep 2019 09:28:24 PM UTC, comment #2: 

Ok. WRT your second point, you are of course right and I didn't think it out properly. WRT the delayed warning being new as of 4.2.1, I didn't know that and agree that it's a serious compatibility issue, so I'm happy to drop this.

David Boyce <boyski>
Sat 14 Sep 2019 08:59:19 PM UTC, comment #1: 

It is definitely not the case that we can ignore -include if the makefile doesn't exist.  In fact until recently, virtually every makefile that needed to be rebuilt would be included with "-include", because otherwise you'd get a warning that the file didn't exist before make would try to build it.

Only as of GNU make 4.2 will the warning be delayed until after make decides whether it can be made so you don't need to use -include but lots and lots of makefiles still do it.  See bug #102.

I'm not sure I see how your option #2 helps.  If I understand your setup correctly it's not that make is ever re-execing itself even once.  What happens is that make tries to rebuild nosuchfile.mk and finds a match-anything rule that runs a recursive make, the recursive make tries to rebuild nosuchfile.mk and finds a match-anything rule that runs a recursive make, etc.  None of these sub-makes ever exit so we never re-exec ourselves.

In other words, it's not that the same make is re-execing itself forever but that each instance of make is forking a new child make that forks another new child make, etc. until presumably the system freaks out.  In other words this seems like a fork bomb.

Paul D. Smith <psmith>
Project Administrator
Sat 14 Sep 2019 08:28:32 PM UTC, original submission:  

I'm not saying this is a bug necessarily but want to submit it for some sort of resolution. I an into an infinite loop situation recently. On my advice, in a very old and complex recursive makefile suite (not of my design or under my ownership) a co-worker added a construct like the following:

-include foobar.mk
foobar ?= XYZ

The idea is that foobar.mk would be a one-line include file which assigns the "foobar" variable if it exists and if not we fall back to the default value "XYZ". This is simple and solved his problem in unit testing but when plugged into the large old suite it resulted in an infinite loop.

The reason is twofold: (a) if an included makefile doesn't exist make will try to run a recipe to create it and (b) somewhere deep in this makefile suite was a match-anything rule. From here it's obvious: make went looking for a recipe that claimed to be able to make foobar.mk, landed on the match-anything rule which does a recursive make invocation (and does not create foobar.mk), and we're off to the races with an infinite make loop.

The situation is illustrated by the following test file:

$ cat Makefile
.PHONY: all
all:
@:$(info making $@)

%:
# @test -f nosuchfile.mk
$(MAKE) $@

#-include nosuchfile.mk

As it stands this works fine:

$ make
making all

If the -include line is enabled the infinite loop happens, but if the test -f line is also uncommented things work correctly again.

My first thought is that it makes no sense to invoke a recipe to create a nonexistent file included via -include because the whole point of -include is to say "I know this file may not exist and that's ok". However, the documentation says of -include that it:

"... acts like include in every way except that there is no error [...] if any of the filenames [...] do not exist or cannot be remade."

I guess the "or cannot be remade" is a strong implication that -include files are subject to remaking so maybe it would hard to "fix" this for compatibility reasons? And of course there's a difference between a file being out of date and not existing and it makes perfect sense to want to update a -included file if it exists.

However, I still think that in the special case where foobar.mk does not exist and is included with -include the remake attempt should be elided and the inclusion silently skipped. Whether that's too much of a compatibility break is a valid question though.

But even stepping back from those details, with either "include" or "-include", why should make continue if it ran the recipe and the recipe did not in fact create the file as it claimed it would? In other words, why not do a check for existence after running the remake recipe (as illustrated in the test case)? It seems to me that dying with "no such file or directory" would be preferable to allowing an infinite loop, and furthermore I think this could be added without breaking (documented) compatibility.  AFAICT the manual discusses only situations where the included file "cannot be remade".  Currently "remaking" seems to be a synonym for "running the recipe which claims to make it" but tightening that definition to to require that the recipe actually creates the file seems both more robust and entirely compatible, not to mention intuitive.

Thus I suggest one or both of these minor tweaks:

1. Fail (with include) or continue (with -include) when, after running the remake recipe, the file does not exist.

2. Ignore (do not attempt to make) a file included by -include if it does not exist at all. But if if does exist, go ahead and try to update it.

David Boyce <boyski>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by boyski (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-09-14 psmith Item GroupBug => Enhancement

    Back to the top


    Powered by Savane 3.5