bugmake - Bugs: bug #54854, multi-target rules invoked too...

 
 

bug #54854: multi-target rules invoked too often with -j2

Submitter:  Bernhard M. <bmwiedemann>
Submitted:  Wed 17 Oct 2018 11:08:15 AM UTC
   
 
Severity:  3 - Normal Item Group:  None
Status:  Not A Bug Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Component Version:  None Operating System:  None
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 25 Mar 2024 07:09:06 PM UTC, comment #9: 

Hopefully it will solve it!  That would be nice.  Cheers!

Paul D. Smith <psmith>
Group administrator
Mon 25 Mar 2024 07:00:10 PM UTC, comment #8: 

Yep that's it.

I can't believe how I missed this part. So a simple &: would solve the issue for me?

!!! Wow !!!

Definitely not my brightest moment.

Thanks a lot. Much appreciated.

Vassilis Virvilis <vasvir>
Mon 25 Mar 2024 06:44:47 PM UTC, comment #7: 

If you can rely on having GNU Make 4.3 or better, you can use grouped explicit targets to get the same behavior make provides to pattern rules, with explicit rules:

https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html

If you must support versions of GNU Make prior to 4.3 (released in 2020) then you will have to use alternate methods.

Paul D. Smith <psmith>
Group administrator
Mon 25 Mar 2024 06:27:34 PM UTC, comment #6: 

Thanks for taking the time to explain with a simple and easy to follow example.

From the looks of it I have to agree that this is a legitimate case and cannot to lead to a broken build.

Unfortunately my case is a rule for a program that builds two real files (not .PHONY). The problem is the second -j thread overwrites the output the first. I agree that the solution is to restructure the rules in a way to avoid behavior and this is what I have done but it took a while to figure it out.

Hopefully this thread will be findable by other people who may wonder about the exact GNU Make behavior and possible workarounds.

Thanks again.


Vassilis Virvilis <vasvir>
Mon 25 Mar 2024 04:34:42 PM UTC, comment #5: 

Definitely there are a lot of valid uses.  Generally, any time where you want to define the same recipe for lots of different targets but you don't want to have to write them all out one at a time.

Your assertion that this construct is not common is just not true: this appears all over the place in all sorts of makefiles.  In fact I just checked a makefile generated by GNU's automake tool, and it uses the construct.  Here's one simple example of an obvious way this might be used:

SUBDIRS := $(wildcard */.)

all: $(SUBDIRS)

$(SUBDIRS):
        $(MAKE) -C $@

.PHONY: all $(SUBDIRS)


There's no reason this should give any sort of warning, even when used with -j.

Paul D. Smith <psmith>
Group administrator
Mon 25 Mar 2024 03:34:23 PM UTC, comment #4: 

comment #3:

> If by "a warning (or a note)" you mean something printed by GNU Make, instead of content in the documentation (this is already described there) then no, we can't do that, because this behavior has many uses, and is used in many situations and in many thousands of makefiles.  Generating messages would not be appropriate.


I see.

How about adding the warning/note if -d is specified? This surely won't be as disturbing and could still help if somebody debugs broken builds.

Is there any use case where this is the desired behavior? I can see builds breaking because of this (multiple invocations of the same rule) but not any valid ones. The condition to issue the warning shouldn't be very common (-j N && N > 1 && rule with multiple target is invoked with no patterns)

Finally I would like to note that there is a similar case where Make issues a warning. I believe when -j is used with a submake $(MAKE) without a plus sign (+). That warning was very helpful for me at least.

Thanks anyway.

Vassilis Virvilis <vasvir>
Sun 24 Mar 2024 11:19:47 PM UTC, comment #3: 

If by "a warning (or a note)" you mean something printed by GNU Make, instead of content in the documentation (this is already described there) then no, we can't do that, because this behavior has many uses, and is used in many situations and in many thousands of makefiles.  Generating messages would not be appropriate.

Paul D. Smith <psmith>
Group administrator
Fri 06 Oct 2023 08:37:10 AM UTC, comment #2: 

I was also bitten by this.

I understand the argument and why this is not considered a bug.

Would you consider a warning (or a note) if multiple targets (without patterns) and -j N with N > 1 are used simultaneously?

Thanks

Vassilis Virvilis <vasvir>
Wed 17 Oct 2018 01:19:36 PM UTC, comment #1: 

This is not a bug, this is how every version of make has always worked, since make was invented.

Your reproducer is not really minimal :).  Here's a minimal reproducer:


all: foo.1 bar.1 biz.1
foo.1 bar.1 biz.1: ; touch foo.1 bar.1 biz.1


When you write a rule with three targets that's exactly the same thing as writing three separate rules; see https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html :


all: foo.1 bar.1 biz.1
foo.1: ; touch foo.1 bar.1 biz.1
bar.1: ; touch foo.1 bar.1 biz.1
baz.1: ; touch foo.1 bar.1 biz.1


If you run in parallel make will run all of these rules at the same time.  It has no way of knowing that one of these recipes will build all three targets.  If you run serially than make still would consider running all recipes, except that it sees that the other targets have been updated after the first one is complete by looking at the modification times.

You can solve this problem by using pattern rules:


all: foo.1 bar.1 biz.1
foo.% bar.% baz.%: ; touch foo.1 bar.1 biz.1


because the way make treats pattern rules with multiple targets is different: see https://www.gnu.org/software/make/manual/html_node/Pattern-Intro.html

Or you can rewrite your rules to build one target per invocation rather than building lots of targets at once: this is how make was designed to work.

If you'd like to discuss further please open a thread at the -email is unavailable- or -email is unavailable- mailing lists.

Paul D. Smith <psmith>
Group administrator
Wed 17 Oct 2018 11:08:15 AM UTC, original submission:  

Seen in the wild at https://github.com/github/hub/pull/1892
I made a minimal reproducer (attached).

To reproduce:
tar xf makebug.tar.gz
cd makebug/
make clean ; time make -j2
make clean ; time make -j1

notice how the -j2 takes longer than -j1
because in make -j1, the multi-target rule to create all man-pages is just invoked once, but with -j2 it is invoked three times - once per input file, not only wasting compute resources, but also creating a chance for race conditions.

Maybe there is a better (parallelism-safe) way to specify such cases where one command turns many input files into many output files than the bison-example at https://www.gnu.org/software/make/manual/make.html#Pattern-Examples ?

Bernhard M. <bmwiedemann>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #45219:  makebug.tar.gz added by bmwiedemann (16KiB - application/gzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by vasvir (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by bmwiedemann (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-10-17 psmith StatusNone Not A Bug
        Open/ClosedOpen Closed
    2018-10-17 bmwiedemann Attached File- Added makebug.tar.gz, #45219

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code