bugmake - Bugs: bug #58056, Forced prerequisite order is not...

 
 

bug #58056: Forced prerequisite order is not honored with pattern rules

Submitter:  None
Submitted:  Fri 27 Mar 2020 11:14:42 AM UTC
   
 
Severity:  3 - Normal Item Group:  Documentation
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  4.3 Operating System:  Any
Fixed Release:  4.4 Triage Status:  Small Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 02 Oct 2022 01:34:19 PM UTC, comment #16: 

I reread your report and Dmitry's explanation with a clearer head and I agree with Dmitry that this is intended behavior.

I'm going to close this issue as a doc bug with the changes to the docs I've added to try to clarify the intent of OO prerequisites.

Paul D. Smith <psmith>
Group administrator
Sun 02 Oct 2022 01:27:10 PM UTC, comment #15: 

I updated the documentation to try to make this clearer:

Old docs:

> Occasionally, however, you have a situation where you want to impose a specific ordering on the rules to be invoked @emph{without} forcing the target to be updated if one of those rules is executed.


New docs:

> Occasionally you may want to ensure that a prerequisite is built before a target, but @emph{without} forcing the target to be updated if the prerequisite is updated.


Note that the next release of GNU make will implement the .WAIT special prerequisite found on other versions of make such as BSD make or Solaris make.  This will allow you to declare "wait points" between prerequisites to serialize them.  This is not a reliable way to manage ordering but it has the advantage of being easy to use, in simple situations.

Paul D. Smith <psmith>
Group administrator
Tue 31 Mar 2020 07:59:32 PM UTC, comment #14: 

In this bash session you can see when foo_A is missing make considers everything is up to date. Unlike when foo_B is missing. That's why make does not built foo_A during the initial travers. Because make considers foo_A an intermediate.

$ ls
makefile
$ cat makefile
all: foo.bar

%.bar: %_A %_B
        @echo $@

%_A:
        @echo $@

foo_B:
        @echo $@
$ make
foo_B
foo_A
foo.bar
$ touch foo_B foo.bar
$ ls foo_A
ls: cannot access foo_A: No such file or directory
$ make
gmake: Nothing to be done for `all'.
$ rm foo_B
$ make
foo_B
foo_A
foo.bar

Dmitry Goncharov <dgoncharov>
Mon 30 Mar 2020 08:20:48 PM UTC, comment #13: 

Note I haven't looked at the actual behavior you're reporting, of misordered recipes, and I haven't understood Dmitry's explanation.  Just looking at the behavior you see I do agree that it's wrong.  However there may be some underlying reason for it that I'm not remembering at the moment.

However, even if it were fixed to work as I think it should, that doesn't mean it works the way you seem to want.  In particular, if you invoked make with -j then make would definitely attempt to build all the prerequisites, both normal and order-only, at the same time.


The docs first describe the behavior of a normal prerequisite; in particular:

> first, it imposes an order in which recipes will be invoked: the
> recipes for all prerequisites of a target will be completed before
> the recipe for the target is run. Second, it imposes a dependency
> relationship: if any prerequisite is newer than the target, then
> the target is considered out-of-date and must be rebuilt.


This means that "foo: bar baz" ensures that both "bar" and "baz" will be complete before "foo" is invoked.  Note it says NOTHING about the relative order in which "bar" and "baz" will be invoked.  Secondarily, it says that if either "foo" or "bar" end up being more up to date than "foo", then "foo" will be rebuilt.

Next it discloses that an order-only prerequisite relaxes the second requirement:

> Occasionally, however, you have a situation where you want to impose
> a specific ordering on the rules to be invoked without forcing the
> target to be updated if one of those rules is executed.


This means that order-only prerequisites still adhere to the first behavior (both "bar" and "baz" are completed before "foo" is invoked), but they do not use the second behvior: just because "bar" or "baz" are newer than "foo", does NOT mean that "foo" will be rebuilt.

None of this entire section discusses any dependency between two prerequisites of the same rule.  All issues of ordering here are discussing ordering between the target and its prerequisites.

There is, in fact, no way to order the prerequisites of a target from within the list of prerequisites itself.  If you want to impose ordering you must do it in a separate rule, where the target will be built after its prerequisites.

Paul D. Smith <psmith>
Group administrator
Mon 30 Mar 2020 07:35:01 PM UTC, comment #12: 

If this behavior is allowed, I think the documentation should clarify what the order-only prerequisites actually means. I still haven't figured out their purpose or how they operate based on the documentation or what has been said here. To me, the documentation basically implies that everything after the '|' is serialized.

Sorry if this is getting of topic, but the issue exists in the first place because of this confusion. I usually never assume any serialization of the prerequisites, but I needed to enforce it in a particular scenario and though the '|' was for this.

Anonymous
Mon 30 Mar 2020 05:00:29 PM UTC, comment #11: 

Well, building intermediate prerequisites during this initial travers along with regular prerequisites is wasteful, if all regular prerequisites are up to date.
If order matters then one prerequisite should explicitly depend on another.

Dmitry Goncharov <dgoncharov>
Mon 30 Mar 2020 04:55:09 PM UTC, comment #10: 

It's true that POSIX can't require anything of pattern rules since they are not part of POSIX.  But that doesn't mean we shouldn't try to make them behave the same way, as a matter of QoI.

Paul D. Smith <psmith>
Group administrator
Mon 30 Mar 2020 04:47:03 PM UTC, comment #9: 

i believe make complies with this requirement.
This out of order behavior is caused by mixing intermediate prerequisites and prerequisites in the same rule.
Posix does not specify intermediate prerequisites such as %_A.
Once your makefile has such prerequisites you are outside the posix realm.


Dmitry Goncharov <dgoncharov>
Mon 30 Mar 2020 04:36:11 PM UTC, comment #8: 

There may be some confusion.  Order-only prerequisites have no implied ordering amongst themselves, which is what Dmitry meant.

That is, if you have:


foo: bar biz | baz boz


then all the prerequisites bar, biz, baz, boz will be completed before foo is run.  However, this does not say that baz will be completed before boz is run (or any of the other prerequisites).

However I'm not convinced this has anything to do with the issue you've reported.  I"m with Sam this is a red herring.

There is indeed an order of processing which is guaranteed by make when run serially, mandated by the POSIX spec as well, which says:

> The make utility shall treat all prerequisites as targets
> themselves and recursively ensure that they are up-to-date,
> processing them in the order in which they appear in the rule.

Paul D. Smith <psmith>
Group administrator
Mon 30 Mar 2020 04:34:17 PM UTC, comment #7: 

comment #6:

> Wait, so order-only prerequisites does NOT mean serialized make for these prerequisites? "a situation where you want to impose a specific ordering on the rules to be invoked" ← That's not what this means?


Right. That is not what it means. The ordering it's talking about is that all the prerequisite recipes will run before the target recipe, if indeed the target recipe runs at all. Nothing about serializing the prerequisite recipes. Nothing about the relative order of the prequisite recipes.

But again, we are somewhat off-topic.

Sam Kendall <sckendall>
Mon 30 Mar 2020 04:17:24 PM UTC, comment #6: 

comment #4:

> So I claim that parallel make and order-only prerequisites are distractions here. We should start with the documentation: what does it guarantee about the build order of the prereqs in the example makefile variants, in a serial make?


Wait, so order-only prerequisites does NOT mean serialized make for these prerequisites? "a situation where you want to impose a specific ordering on the rules to be invoked" ← That's not what this means?

Anonymous
Mon 30 Mar 2020 04:12:07 PM UTC, comment #5: 


comment #3:

> make builds order only prerequisites in parallel.


How can the ordering on the rules be respected then?

Anonymous
Mon 30 Mar 2020 04:11:39 PM UTC, comment #4: 

I think we can assume the original report was about serial execution of make. Also, if I remove the vertical bar from the example makefile (or from the variation with "foo_A" instead of "%_A"), I get the identical behavior. I tried versions 3.82 and 4.3.

So I claim that parallel make and order-only prerequisites are distractions here. We should start with the documentation: what does it guarantee about the build order of the prereqs in the example makefile variants, in a serial make?

I couldn't find any such guarantees. I've seen countless (serial) makefiles that count on prereqs building serially, e.g., "do_something: step1 step2 step3". Apparently we can't count on that order if one of the prereqs matches a pattern rule; but I didn't find anything about that, either.

The GNU make doc is generally very clear and complete. I suspect I'm missing something.


Sam Kendall <sckendall>
Mon 30 Mar 2020 03:22:57 PM UTC, comment #3: 

make builds order only prerequisites in parallel.

Dmitry Goncharov <dgoncharov>
Mon 30 Mar 2020 02:11:29 PM UTC, comment #2: 

comment #1:

> This behavior is intended.
> [...]
> 2. Another reason is to allow parallel execution. If all prerequisites had to be built in the specified order how would you built them in parallel?


Did you notice the '|'? foo_A and foo_B are order-only-prerequisites, which should disable any parallelism.

https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html

Anonymous
Sun 29 Mar 2020 02:27:08 AM UTC, comment #1: 

This behavior is intended.

1. To figure out if a target has to be rebuilt make traverses the list of prerequisites and finds out if any of the prerequisites, but not intermediate prerequisites, need to be rebuilt.
During this traverse make rebuilds all prerequisites that are out of date. If, during this traverse, make finds out that some prerequisite was out of date, then make proceeds to rebuild intermediate prerequisites.

Intermediate prerequisites are not rebuild during this traverse along with prerequisites because, if all prerequisites are up to date then the target is up to date and there is no need to built intermediate prerequisites.

In your example
foo_A is an intermediary prerequisite.
foo_B is a prerequisite.

2. Another reason is to allow parallel execution. If all prerequisites had to be built in the specified order how would you built them in parallel?

Dmitry Goncharov <dgoncharov>
Fri 27 Mar 2020 11:14:42 AM UTC, original submission:  

Given the following Makefile:


all: foo.bar

%.bar: | %_A %_B
        @

%_A:
        @echo $@
foo_B:
        @echo $@


We get:


% make
foo_B
foo_A


Instead of:


% make
foo_A
foo_B


The forced order "%_A %_B" in the prerequisites of "%.bar" is not respected.

Note that replacing "foo_B:" with "%_B" does not trigger the bug.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by sckendall (Posted a comment)
  • -email is unavailable- added by dgoncharov (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-10-02 psmith Item GroupBug Documentation
        StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Operating SystemNone Any
        Fixed ReleaseNone 4.4
        Triage StatusNone Small Effort

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code