bugmake - Bugs: bug #65533, gmake-4.4.1 has a performance...

 
 

bug #65533: gmake-4.4.1 has a performance regression: at least the nwchem project now builds much slower

Submitter:  Yuri <yurivict>
Submitted:  Fri 29 Mar 2024 06:50:10 AM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.4.1 Operating System:  None
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Fri 29 Mar 2024 05:53:46 PM UTC, comment #4: 

I cloned the nwchem git repository and my suspicions were correct.

I see many, many instances of recursive variable assignment using $(shell ...) syntax, such as:

src/nwc_columbus/sifs/GNUmakefile
38:ALLF = $(filter-out ./sifs_stubs.F, $(shell find . -name '*.F'))

(this is just one example there are 300+ spread across the makefiles).

At the same time I see that various makefiles are forcing all variable assignments to be exported; for example:

src/peigs/src/c/Makefile.proto:.EXPORT_ALL_VARIABLES:


There are also instances of .EXPORT_ALL_VARIABLES in some Makefiles, but these are probably not a problem because the Makefiles appear to be for BSD make (GNUmakefile is used for GNU Make).  So, there appear to be about 5 makefiles that have this setting.

But, when the Makefile.proto above is included in another makefile, that sets that parameter for every make instance that includes it.

The simplest solution here is to go through and remove all those .EXPORT_ALL_VARIABLES: settings.  This is just a very bad idea.  If there are specific variables that need to be exported then those, and only those, should be exported.

Unfortunately there is not any simple solution right now, except either fix the makefiles or else limit your version of GNU Make to 4.3 or earlier before this change was made.

I have been thinking about how to mitigate this issue and I just don't have any good ideas.  There doesn't seem to be any good heuristic we can use to avoid the behavior without also introducing bizarre and unexplainable edge cases.

Paul D. Smith <psmith>
Group administrator
Fri 29 Mar 2024 04:58:32 PM UTC, comment #3: 

comment #2:

> the gmake process should consume close to 100% of CPU much of the time doing these variable expansions

I like your thinking but that's that's not what Paul's theory predicts.  Perhaps an example will help to explain why not.  Given:

martind@stormy:~/tmp/make-65533$ cat Makefile
export FOO = $(shell set -x; echo FOO; sleep 0.1)
export BAR = $(shell set -x; echo BAR; sleep 0.1)
export BAZ = $(shell set -x; echo BAZ; sleep 0.1)
export QUX = $(shell set -x; echo QUX; sleep 0.1)
$(info QUX is $(QUX))
all:;
martind@stormy:~/tmp/make-65533$

... then older versions of Gnu Make, here represented by 4.2.1, behave like this:

martind@stormy:~/tmp/make-65533$ time /usr/bin/make
+ echo QUX
+ sleep 0.1
QUX is QUX
make: 'all' is up to date.

real        0m0.105s
user        0m0.004s
sys        0m0.000s
martind@stormy:~/tmp/make-65533$

The new version of Gnu Make does this:

martind@stormy:~/tmp/make-65533$ time make
+ echo BAZ
+ sleep 0.1
+ echo BAR
+ sleep 0.1
+ echo BAR
+ sleep 0.1
+ echo BAZ
+ sleep 0.1
+ echo FOO
+ sleep 0.1
+ echo BAZ
+ sleep 0.1
+ echo FOO
+ sleep 0.1
+ echo FOO
+ sleep 0.1
+ echo BAZ
+ sleep 0.1
+ echo BAR
+ sleep 0.1
+ echo BAR
+ sleep 0.1
+ echo FOO
+ sleep 0.1
+ echo FOO
+ sleep 0.1
+ echo BAR
+ sleep 0.1
+ echo BAZ
+ sleep 0.1
+ echo QUX
+ sleep 0.1
QUX is QUX
make: 'all' is up to date.

real        0m1.647s
user        0m0.036s
sys        0m0.004s

1 QUX but 5 each of FOO, BAR and BAZ - it takes 16 times as long and essentially none of the time is going in Gnu Make's string manipulation, but all in the shell commands.  It's not the contention that the shell commands are themselves expensive, though they will involve more work than Gnu Make's variable expansion, but that the number of invocations of them has increased exponentially.  4 invocations were perhaps intended, though only 1 was previously experienced, but 16 have now happened.

Martin Dorey <mdorey>
Fri 29 Mar 2024 02:58:24 PM UTC, comment #2: 

Hi Paul,

comment #1:

> Since I don't have any idea what the "nwchem" project is or how its makefiles work and you haven't provided any short examples, I can't say for sure.


Thank you for your insight.

For the record:
NWChem is a popular computational chemistry software package: https://www.nwchem-sw.org/
Here is their top-level makefile: https://github.com/nwchemgit/nwchem/blob/master/src/GNUmakefile

Assuming that your explanation is correct, the gmake process should consume close to 100% of CPU much of the time doing these variable expansions. However, I don't observe this. gmake's CPU usage is mostly <1%, and all processes listed by top look like this most of the time:

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
36100 nobody        1  68    0    13M  2136K piperd   1   0:00   0.70% gmake
65966 nobody        1  68    0    13M  2272K wait     4   0:00   0.00% gmake
46544 nobody        1  68    0    13M  1948K wait     3   0:00   0.00% gmake
61005 nobody        1  20    0    13M  1316K wait     1   0:00   0.00% make
83976 nobody        1  55    0    13M  1684K wait     5   0:00   0.00% sh
77926 nobody        1  68    0    14M  2952K wait     4   0:00   0.00% bash
46543 nobody        1  20    0    13M  1712K wait     4   0:00   0.00% sh
77925 nobody        1  68    0    13M  1912K wait     2   0:00   0.00% sh
77927 nobody        1  68    0    12M  1544K piperd   6   0:00   0.00% cut

It looks like they invoke a lot of external processes (cut, sed, perl, awk), and this is somehow extremely slower than before when there is a lot of them.


Yuri

Yuri <yurivict>
Fri 29 Mar 2024 12:08:50 PM UTC, comment #1: 

Since I don't have any idea what the "nwchem" project is or how its makefiles work and you haven't provided any short examples, I can't say for sure.

However, it's almost 100% sure that the problem is that the project's makefiles use a lot of $(shell ...) functions, with recursive variable assignment; e.g. things like:

  FOO = $(shell some script here)

along with either forcing all variables to be exported:

  .EXPORT_ALL_VARIABLES:

or, just exporting directly a lot of variables that contain shell commands, like:

  export FOO = $(shell some script here)


This has always been very inefficient because this shell script is invoked every time the variable is expanded, instead of only once when the variable is assigned.

But, starting in GNU Make 4.4 this potentially became much more inefficient due to this change:
https://git.savannah.gnu.org/cgit/make.git/tree/NEWS?h=4.4#n73

To fix this, you should ensure that your makefiles adhere to at least one of these rules:

  • Always use simple assignment (FOO := $(shell ...)) when using the $(shell ...) function unless there's a reason that this can't work.
  • Avoid setting .EXPORT_ALL_VARIABLES: special target and instead explicitly export only the variables you need.


Paul D. Smith <psmith>
Group administrator
Fri 29 Mar 2024 06:50:10 AM UTC, original submission:  

When the FreeBSD port devel/gmake was updated from 4.3 to 4.4.1 on 2024-03-03, the nwchem project build slowed down. It used to build in 1-1.5 hours, and now it builds in 10+ hours.

The nwchem build runs a lot of shell commands.

Some GNU make changes caused this.

Did anyone else reported performance problems?
How can performance regression be fixed?

Yuri <yurivict>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mdorey (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by yurivict (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-caa5.
    Corresponding source code