bugmake - Bugs: bug #64746, Exponential Runtime in make 4.4.1...

 
 

bug #64746: Exponential Runtime in make 4.4.1 when export is used

Submitter:  Till Backhaus <tback>
Submitted:  Wed 04 Oct 2023 09:31:42 PM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.4.1 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 06 Oct 2023 06:29:14 PM UTC, comment #3: 

Many have asked for a new "optional simple assignment" variant.  I'm not opposed to it.

Another option that will work with most currently available versions of GNU make, although it looks somewhat scary, is described here:

https://make.mad-scientist.net/deferred-simple-variable-expansion/

I haven't actually tested it to make sure it fixes this specific problem but based on my understanding of the issue it should.

Paul D. Smith <psmith>
Group administrator
Fri 06 Oct 2023 06:13:02 PM UTC, comment #2: 

That explanation was really helpful. I discovered the difference between recursively expanded variables and simple expanded variables only after posting this issue, but I was still confused why recursive expanded variables would cause this behavior.
 
comment #1:

> Prefer simply expanded variables with $(shell).
>
> Given that sush questions keep arising, maybe we should update NEWS with a warning about recursively expanded variables in combination with $(shell).


Yes, I think that's a great idea.

I'd also like to add a little more context about my use case: I use this to read an .env file and then define variables that are still undefined.

-include .env
export

BUILD_OS ?= $(shell uname -s)
...

Up until 4.4 this wasn't a problem because $(shell) didn't see exported
variables. What I'd like to have in 4.4 would be "conditional simple
expansion" from a new ?:= operator that works like this:

ifeq ($(origin VAR_1), undefined)
  VAR_1 := $(shell echo 1)
endif

Until then I can use the verbose form from above, or I could introduce
another variable to do it in two lines:

VAR_1_ := $(shell echo 1)
VAR_1 ?= ${VAR_1_}


Till Backhaus <tback>
Fri 06 Oct 2023 03:42:18 PM UTC, comment #1: 

VAR_1 is a recursively expanded variable. When make defines this
variable, the value is set to string '$(shell echo 1)'. No subshell is
spawned at this time. Each time make expands this variable, make
spawns a shell and has the shell execute 'echo 1' and the output of
the subshell is the result of the expansion.

This is in contrast with
VAR_X := $(shell echo 1)

VAR_X is a simply expanded variable. When make defines this variable,
make spawns a shell and has the shell execute 'echo 1' and sets the
value of the variable to the output of the subshell. Then each time
this variable is expanded, make won't spawn the shell again.

The difference is critical in this scenario. With variables exported
to subshell, each time make spawns a subshell, it has to expand all
these variables. E.g. in order to expand VAR_7, make has to spawn a
subshell. In order to spawn this subshell, make has to export VAR_6,
VAR_5, etc. Each of those requires its own subshell, which again
requires make to export these variables, etc.

Prefer simply expanded variables with $(shell).


Given that sush questions keep arising, maybe we should update NEWS with a warning about recursively expanded variables in combination with $(shell).

Dmitry Goncharov <dgoncharov>
Wed 04 Oct 2023 09:31:42 PM UTC, original submission:  

This is my first report here, I might lack the proper etiquette.

I'm on fedora 38 which installs gnu-make 4.4.1. I'm reporting a regression related to the changed
behavior of export:

> * WARNING: Backward-incompatibility!
>  Previously makefile variables marked as export were not exported to commands
>  started by the $(shell ...) function.  Now, all exported variables are
>  exported to $(shell ...).  If this leads to recursion during expansion, then
>  for backward-compatibility the value from the original environment is used.
>  To detect this change search for 'shell-export' in the .FEATURES variable.



This rather short Makefile shows exponential runtime depending on the number of variables.


export

VAR_1 ?= $(shell echo 1)
VAR_2 ?= $(shell echo 2)
VAR_3 ?= $(shell echo 3)
VAR_4 ?= $(shell echo 4)
VAR_5 ?= $(shell echo 5)
VAR_6 ?= $(shell echo 6)
VAR_7 ?= $(shell echo 7)

foo:
        echo foo


Runtime with 7 variables is ~20s on reasonably current hardware (AMD Ryzen 7 4750U):


time make foo
echo foo
foo

real 0m21.663s
user 0m4.082s
sys 0m18.027s



make foo --debug=verbose
GNU Make 4.4.1
Built for x86_64-redhat-linux-gnu
Copyright (C) 1988-2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'foo'.
 File 'foo' does not exist.
Finished prerequisites of target file 'foo'.
Must remake target 'foo'.
echo foo
Makefile:7: not recursively expanding VAR_5 to export to shell function
...
Makefile:4: not recursively expanding VAR_2 to export to shell function
foo
Successfully remade target file 'foo'.


The number of expansions grows exponentially:
I modified the Makefile to contain different numbers of variables and got
an exponential number of suppressed expansions:


make foo --debug=verbose | grep -c recurs
82201

Variables to Expansions:
1 var 1
2 vars 6
3 vars 33
4 vars 196
5 vars 1305
6 vars 9786
7 vars 82201


Best regards,
Till

Till Backhaus <tback>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by dgoncharov (Posted a comment)
  • -email is unavailable- added by tback (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code