bugmake - Bugs: bug #56484, [RFE] compile modified files first

 
 

bug #56484: [RFE] compile modified files first

Submitter:  None
Submitted:  Wed 12 Jun 2019 09:07:39 PM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.2.1 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 25 Aug 2019 05:14:11 AM UTC, comment #3: 

The POSIX standard requires that prerequisites are processed in order; it says:

> The make utility shall treat all prerequisites as targets themselves and recursively ensure that they are up-to-date, processing them in the order in which they appear in the rule.


There are lots of makefiles written to rely on this behavior, that do not (correctly) support parallelism.

GNU make will also process prerequisites in the same order when parallel jobs are enabled.  Of course, the operating system may allow jobs to finish out of order (jobs that were started first finish after jobs started second).  Also, if make cannot build target A because one of its prerequisites is still being built, it can start a later target that would normally not be built until A completed.

Of course, by adding the -j option you are excusing yourself from strict adherence to the above POSIX requirement, so it's possible that GNU make could do something different when -j is in effect.  Or some new option could be used to enable this behavior.

There's only one way I can think of to make this work, that doesn't involve massive changes.  However, even so it's a very large effort, and I have no plans to work on this myself.  Maybe someone else would want to take it on so I'll leave this open as an enhancement request.

The first problem is that make has no way to determine "was this file changed after the last build", because make has no idea when the last build happened.  Make doesn't track when it was run: it uses the filesystem's time last modified value on targets and their prerequisites to determine what's out of date.  Basically you're asking that some prerequisites have a special meaning, such that if they're newer than the target that gives them a higher priority.  The obvious and simple way to do this is treat the first prerequisite (the one used for $<) as the special one and say if that prerequisite is newer than the target then this target should be rebuilt before targets where $< is not newer.

The second problem is that GNU make does not work by first creating a list of all the things it needs to rebuild before it builds anything.  So there's no way to take the entire list and re-order it.  Instead, make starts with the first target, looks at its first prerequisite, sees if that needs to be built and builds it, then looks at the next and builds that, etc. until it finally discovers there's nothing left to build, then it's done.

The only way this could be done without completely rewriting make's build algorithm would be for make to walk the dependency graph twice.  The first time through it would rebuild only targets whose initial prerequisite ($<) is newer.  Then when it was done with that, it would go back and walk the whole graph again, this time building everything else that was out of date.

Paul D. Smith <psmith>
Group administrator
Thu 13 Jun 2019 12:58:12 AM UTC, comment #2: 


> I don't think this could be an opt-out because make has a very longstanding expectation that prerequisites are built in left-to-right order (with -j1).


Is it explicitly documented or just an implicit behavior? When I think of it: if the order isn't guaranteed with -jn, what would be the reason for -j1 make an exception other than an accident? And I doubt nowadays developers and distro packagers aren't using the -j option to speed up compilation. Hence the out-of-order compilation should be widely tested by now.

So, if there's no explicit documentation for the order, I'd rather vote for opt-out.

P.S.: sorry, I couldn't answer, because I reported the bug through savannah site, which turned out to be absolutely terrible (that is, the TL;DR). I had to create this throw-away account to be able at least to reply.

Konstantin Kharlamov <kkharlamov>
Wed 12 Jun 2019 09:46:14 PM UTC, comment #1: 

I don't think this could be an opt-out because make has a very longstanding expectation that prerequisites are built in left-to-right order (with -j1).

David Boyce <boyski>
Wed 12 Jun 2019 09:07:39 PM UTC, original submission:  

Problem: if a project has 300 C files and a single header that's included everywhere, and then someone changes 2 files: the header and any C file, most likely one wants to know whether the changes caused build errors. However, after executing `make`, in the worst case, the changed C file may get built after the 299 others, and fail the build.

Ultimately, when rebuilding a project, one is interested (well, besides of course having the binaries) in "did my changes cause any problem?" question.

Solution: before building, sort files to build by "was it changed after last build" field, and compile changed ones first.

Such behavior sounds benefical to most existing projects, and could be controlled by an env. variable, like MAKE_NO_SORT, to opt out.

# Steps to reproduce (in terms of terminal commands):

   $ cat Makefile
   OBJECTS = main.o empty1.o empty2.o
   HEADERS = header.h

   default: myapp

   %.o : %.c $(HEADERS)
   gcc -c $< -o $@

   myapp: $(OBJECTS)
   gcc $(OBJECTS) -o $@
   $ touch empty1.c empty2.c header.h && echo "int main() {}" > main.c
   $ make
   gcc -c main.c -o main.o
   gcc -c empty1.c -o empty1.o
   gcc -c empty2.c -o empty2.o
   gcc main.o empty1.o empty2.o -o myapp
   $ touch header.h empty2.c
   $ make

## Expected output:
   gcc -c empty2.c -o empty2.o
   gcc -c main.c -o main.o
   gcc -c empty1.c -o empty1.o
   gcc main.o empty1.o empty2.o -o myapp

## Actual output:
   gcc -c main.c -o main.o
   gcc -c empty1.c -o empty1.o
   gcc -c empty2.c -o empty2.o
   gcc main.o empty1.o empty2.o -o myapp

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by kkharlamov
  • -email is unavailable- added by kkharlamov (Posted a comment)
  • -email is unavailable- added by kkharlamov
  • -email is unavailable- added by boyski (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-06-13 kkharlamov Carbon-Copy- Added nongnusucks
    2019-06-13 kkharlamov Carbon-Copy- Added kkharlamov

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code