bugmake - Bugs: bug #63111, Regression. make runs out of file...

 
 

bug #63111: Regression. make runs out of file descriptors.

Submitter:  Dmitry Goncharov <dgoncharov>
Submitted:  Mon 26 Sep 2022 09:35:21 PM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  4.4 Operating System:  Any
Fixed Release:  4.4 Triage Status:  Small Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 02 Oct 2022 06:25:13 PM UTC, comment #7: 

I pushed Andreas's fix for the file descriptor problems.

Thanks!

Paul D. Smith <psmith>
Group administrator
Sat 01 Oct 2022 11:04:43 PM UTC, comment #6: 

I took a shot at creating a warning.  It's not so simple.

I think that the thing we want to warn about is:

  • we used a pattern rule with multiple targets to build the wanted target
  • the wanted target was actually updated
  • some also-make target was not updated


This is because we don't want to raise a warning about a pattern rule that intentionally doesn't update any of its targets (maybe it decides that things are up to date even if the timestamps don't show this).  We only want to warn about pattern rules that don't update ALL the targets.

Unfortunately this is not so easy to determine the way things work today.  The main problem is that we don't do all the investigation of an also-make target that we do of the primary target, so we don't actually know whether it was updated or not.  We can detect whether it was not created at all, but not whether it was updated.  Maybe that's good enough?  It won't help the situation where the also-make file exists but is not up to date and is not updated by running the recipe: in this case there would be no warning but (with the changes for bug #12078 ) the target would still be rebuilt every time you run make.

The smaller problem is that whether the main target was updated or not can only be determined in specific spots and there are more than one: there's at least one for goal targets and one for non-goal targets, so we'd need to add the check in two places.

Paul D. Smith <psmith>
Group administrator
Wed 28 Sep 2022 07:52:24 PM UTC, comment #5: 


>> 2) Keep the change for explicit grouped targets, and revert the change for pattern rule grouped targets, say that these two will always work differently.
>
> i suspect, this difference in behavior can cause trouble than the original bug.

I guess you mean, can cause more trouble?  In the short term this will be less trouble, I'm sure, since we aren't changing existing behavior for (apparently) somewhat used "features".  I do agree that going forward it will be more problematic if they diverge.

>> 3) Keep the change for explicit grouped targets, revert the change for pattern rule grouped targets...
>
> The same differnce in behavior.

Yes but: the behavior is different than explicit grouped targets for this release but it's the same as the behavior for pattern rules since forever, so it's backwards compatible, and it gives a warning (but not error) so that people using build systems that are affected by this can fix their environments... the goal is that in the next release we'd remove the warning, install the change of behavior, and re-align the two.

I'm not sure how tricky this idea would be though.  Maybe I'll try to code it and see what it looks like.

> Given that dtc is fixed and the Andreas patches fix coreutils, let us see, maybe we are good?

We've had 2 projects with issues so far just in the initial testing, which concerns me given the number of projects out there.

And remember, people upgrade make without necessarily upgrading, or wanting to upgrade, all their other software.  Saying "it's fixed in the latest Git HEAD for the project" is not necessarily great.

Paul D. Smith <psmith>
Group administrator
Tue 27 Sep 2022 09:35:26 PM UTC, comment #4: 

comment #3:

> There are two parts to this: one is not freeing / cleaning temp files on re-exec.  Andreas posted a patch which I will check, thanks Andreas!


The coreutils failure is the leak of fd. There is no infinite recursion. i can report that coreutils succeed with the Andreas patches.


> 2) Keep the change for explicit grouped targets, and revert the change for pattern rule grouped targets, say that these two will always work differently.


i suspect, this difference in behavior can cause trouble than the original bug.


> 3) Keep the change for explicit grouped targets, revert the change for pattern rule grouped targets...


The same differnce in behavior.


> 4) Keep things the way they are in release 4.3.90, and just say that any build systems that have this behavior will need to be updated before they can use GNU make 4.4.


Given that dtc is fixed and the Andreas patches fix coreutils, let us see, maybe we are good?


Dmitry Goncharov <dgoncharov>
Tue 27 Sep 2022 04:23:38 PM UTC, comment #3: 

There are two parts to this: one is not freeing / cleaning temp files on re-exec.  Andreas posted a patch which I will check, thanks Andreas!

The second is the infinite re-exec issue.  This is another facet of the same issue reported by bug #63098

The commit Dmitry points to contains two changes: one for explicit grouped targets and one for pattern rule grouped targets.  I think we should do one of these things:

1) Decide that the change should not be made, and that both bug #12078 and bug #62809 should be reclassified as "Not a Bug" and that the issues mentioned in those bugs are just the way things work.

2) Keep the change for explicit grouped targets, and revert the change for pattern rule grouped targets, say that these two will always work differently, and reclassify bug #12078 as "Not a Bug".

3) Keep the change for explicit grouped targets, revert the change for pattern rule grouped targets, but have make generate a warning when a pattern rule doesn't create all the targets saying that a future version of make will consider the targets to be out of date still.  Then in the next release of GNU make we would re-instate the change for pattern rule grouped targets.

4) Keep things the way they are in release 4.3.90, and just say that any build systems that have this behavior will need to be updated before they can use GNU make 4.4.

I'm not excited about either of options #1 or #2, although it's probably worth re-examining the bugs and just making sure the change is right.

Personally I'm leaning more towards option #3, although ultimately I'm not sure how much it will really help.  Build systems which are "historical" are not going to fix themselves regardless of how many releases we warn, and build systems which are maintained can just apply a fix.

Paul D. Smith <psmith>
Group administrator
Tue 27 Sep 2022 07:18:58 AM UTC, comment #2: 

Note that dtc has already fixed the problematic pattern rule (see commit 98a0700).

Andreas Schwab <schwab>
Mon 26 Sep 2022 09:40:01 PM UTC, comment #1: 

A user reported an issue where make runs out of file descriptors.

The original bug report is here
https://lists.gnu.org/archive/html/bug-make/2022-09/msg00122.html


This makefile in question has the following problematic make code


%.tab.c %.tab.h %.output: %.y
    @$(VECHO) BISON $@
    $(BISON) -b $(basename $(basename $@)) -d $<
%.d: %.c
    @$(VECHO) DEP $<
    $(CC) $(CPPFLAGS) $(CFLAGS) -MM -MG -MT "$*.o $@" $< > $@
-include $(DTC_OBJS:%.o=%.d)


The following takes place.
1. make needs to include parser.tab.d.
2. make finds rule '%.d: %.c', which depends on parser.tab.c.
3. make needs to build parser.tab.c.
4. make finds rule '%.tab.c %.tab.h %.output: %.y'. Note, this rule also has
parser.output listed as a target.
5. make runs the recipe and builds parser.tab.c. Note, the recipe does not
build parser.output.
6. make runs the recipe of rule '%.d: %.c' and generates parser.tab.d.
7. make re-executes itself to read updated parser.tab.d.
8. Normally, there should be no more re-exec. However, due to missing parser.output, the cycle repeats.

Here is a minimal makefile which reproduces the issue.


$ ls
hello.x  makefile
$ cat makefile
all: hello.q
%.q %.bogus: %.x; touch $*.q
hello.d: hello.q; touch $@
-include hello.d
$ ASAN_OPTIONS=detect_leaks=0 ~/src/make/l64/make -R --debug=m
GNU Make 4.3.90
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
Updating makefiles....
  File 'hello.d' does not exist.
    File 'hello.q' does not exist.
   Must remake target 'hello.q'.
touch hello.q
   Successfully remade target file 'hello.q'.
 Must remake target 'hello.d'.
touch hello.d
 Successfully remade target file 'hello.d'.
Re-executing[1]: /home/dgonchar/src/make/l64/make -R --debug=m
GNU Make 4.3.90
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
Updating makefiles....
    Grouped target peer 'hello.bogus' of file 'hello.q' does not exist.
   Must remake target 'hello.q'.
touch hello.q
   Successfully remade target file 'hello.q'.
  Prerequisite 'hello.q' is newer than target 'hello.d'.
 Must remake target 'hello.d'.
touch hello.d
 Successfully remade target file 'hello.d'.
Re-executing[2]: /home/dgonchar/src/make/l64/make -R --debug=m
GNU Make 4.3.90
^C


This is clearly a problem in the makefile. However, make-4.3 allows this.
The change in behavior was introduced in
fabb03eac412b5ea19f1a97be31dc8c6fa7fc047.

i see atleast the following options
1. revert fabb03eac412b5ea19f1a97be31dc8c6fa7fc047.
2. keep the code as is and expect the users to fix their makefile. So far, Andreas reported dtc and coreutils affected.

Dmitry Goncharov <dgoncharov>
Mon 26 Sep 2022 09:35:21 PM UTC, original submission:  

.

Dmitry Goncharov <dgoncharov>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by dgoncharov (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-10-02 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.4
        Triage StatusNone Small Effort

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code