bugmake - Bugs: bug #19448, Re-exec after "include file...

 
 

bug #19448: Re-exec after "include file rebuild" is more dependent on filesystem timestamps than strictly necessary.

Submitter:  Christopher Lester <kesterlester>
Submitted:  Thu 29 Mar 2007 10:04:20 PM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  3.81 Operating System:  Any
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 07 Jun 2009 07:23:47 PM UTC, comment #7: 

Switching to an enhancement request.

Paul D. Smith <psmith>
Group administrator
Tue 01 May 2007 04:32:50 AM UTC, comment #6: 

In fact, your original idea of passing -W <foo> for each included file "foo" so that the re-invoked make would realize it should not be built again WAS implemented (by me) in an earlier version of GNU make.

However, it lasted only a few days out in the wild, because it immediately hit build environments where many hundreds or even thousands of files were included... and the re-exec of make failed because the environment was not sufficiently large to be able to pass all those options through exec!

Considering there is really no way to ensure that you have enough space for all possible uses, I took that code back out and reverted to the original (and still current) behavior.

However, recently a new suggestion has been made which would enhance make to understand "@-files", which are apparently becoming more common even on some UNIX platforms as a way to pass large numbers of arguments.  If that feature was added to GNU make then make could take advantage of it to propagate the -W flags in a way that wouldn't blow out the environment.  See patch #5809

Paul D. Smith <psmith>
Group administrator
Sat 28 Apr 2007 08:46:27 AM UTC, comment #5: 

What Christopher suggests might work, in principle, for Make jobs that are performed sequentially.  But what about parallel execution, where several targets are remade asynchronously?  Either we rely on the filesystem to record time of creation for each target, or we will need to complicate the communications between the sub-make's enormously to pass the extra data (and even then I'm not sure it will work reliably, because with -j it is theoretically possible that two targets were created at the same time instance).

Anyway, given the fact that ``crappy filesystems'' are all but non-existent, why bother inventing complicated mechanisms like that?  It is much easier to get Make to use the finest time resolution the filesystem can give us.  Right now, Make assumes that the st_mtime member of struct stat is the best we can have, but that is not necessarily so on some systems.  Maybe introducing another abstraction that is independent of `stat' is a better way to handle these problems.

Eli Zaretskii <eliz>
Group Member
Sat 28 Apr 2007 01:07:29 AM UTC, comment #4: 

I agree with Paul.  GNU make and posix have long decreed that equal timestamps of target and source imply up-to-dateness, and I would not seek to change that.  [Any change to that rule would lead successive invocations of make (separated by, e.g., minutes) on a project that builds in under 1 second to ALWAYS rebuild.  That would be a disaster!]

My original (perhaps badly explained) point was supposed to relate to a different issue.  Let's define two terms to help make clear what I wanted to say.

(1) Define "user-make-invocation" to mean the process of a user typing "make" into a shell (or make being invoked by a script etc).

(2) Define "internal-make-invocation" to mean whatever process "make" has to undergo internally when it spots that it has just updated one of the include files of the current Makefile - and thereby realises that it must re-read in the Makefile (and the updated include file) and start over "afresh" in accordance with the present manual (which I support).

In the present implementation "internal-make-invocation" and "user-make-invocation" are basically the same thing - though they don't have to be in order to comply with the spec.  So far as I understand, "internal-make-invocation" is implemented by something almost equivalent to a call to system("make [initialArgs]");

My point is supposed to be that end users (well, at least ones like me) would like to be able to view "user-make-invocation" as a sort of "atomic" predictable command that doesn't depend on timestamp resolution DURING THE TIME IT RUNS -- in the same way as a single ordinary invocation of "make" doesn't depend on timestamp resolution if there are no fancy autogenerated inclusions.  No matter how crappy your filesystem time resolution is, make is clever enough to plan its build order before it builds the first file.  Sure, your crappy filesystem time resolution might cause you problems BETWEEN "user-make-invocations" (fair enough) but it NEVER causes you problems DURING a "user-make-invocation" if there are no fancy rebuilt includes.  This is the good thing about make.

Only if there are re-generated includes (or if the user recursive make) is crappy time resolution exposed DURING a "user-make-invocation" - and this itself is only because of the present blurring of (1) and (2).

OK - there is no way make can avoid exposing filesystem vagueries in the case of recursive make - so we forget that issue.

But in-principle it would be possible (if very difficult!) to design a  clever make implementation that didn't expose timestamp resolution DURING "user-make-invocation"s when using re-generated include files.   This might involve keeping a virutal filesystem of sorts in memory - one only needing to store time-ording-stamps (not timestamps!) against each target generated by a rule, so that subsequent inner re-execs of make could use this virtual database to split ties on second or subsequent runs.

I don't think that would be anti POSIX spirit or letter - I think it would actually support it all the better - as "user-make-invocations" would become deterministic and not unpredictable.

With the present implementation, it is technically not possible to predict the outcome of "make" on some Makefiles, even if you know the latency of the filesystem and have full knowledge of timestamps before a single "user-make-invocation" when regenerated includes are used.  Sometimes the outcome is just random, as my test example showed.

But it would be hard to fix this problem in the manner I have suggested, and I would quite understand why nobody would want to code it!  I've no clear idea of how it could be done for "make -j 4" usage etc.

Christopher Lester <kesterlester>
Fri 27 Apr 2007 11:52:05 PM UTC, comment #3: 

Sorry, but you're incorrect about the way original UNIX make worked.  In fact, every version of make that I'm familiar with does and has always considered equal timestamps as "up to date".  The POSIX standard for make is quite clear about this as well:

A target is considered out-of-date if it is older than any of its prerequisites or if it does not exist.

And also:

The make utility examines time relationships and shall update those derived files (called targets) that have modified times earlier than the modified times of the files (called prerequisites) from which they are derived.

It's also incorrect to classify this behavior as a recent change in GNU make; GNU make has always worked this way.

Paul D. Smith <psmith>
Group administrator
Fri 27 Apr 2007 10:56:55 PM UTC, comment #2: 

I will disagree with the original poster.  Make's job is to be conservative: after typing "make", you should be able to count on it that everything is up to date.  If that invokes a bit of extra work, the lost CPU time is far better than risking something not being updated.

For that reason, the original Unix make considered equal timestamps to be "out of date" unless it had other information to the contrary.  So if foo depends on bar and I type "make foo; touch bar; make foo", foo should be made the second time REGARDLESS of issues of timestamp resolution and clock races.

The current behavior (which seems to be the result of a recent change) is incorrect.

Anonymous
Fri 27 Apr 2007 10:40:34 PM UTC, comment #1: 

I'm going to disagree with the original submitter.  Make should be conservative: if it's not certain that a file is up to date, it should be rebuilt.

In particular, in the absence of other information, equal timestamps should be considered NOT up to date.  It is better to waste a bit of CPU time than to risk having an out-of-date file.
I'm pretty sure that was the behavior of the original Unix make, and for good reason.

Anonymous
Thu 29 Mar 2007 10:04:20 PM UTC, original submission:  


I have attached a 22 line Makefile.
For the purposes of this document, always run it in the following way:

"make clean ; make"

What you SHOULD see in the output is two occurrences of the word MOO (corresponding to two re-generations of the included "foo.d" file) like this:

-----------------------------------------------
[lester@localhost tmpfs]$ make clean ; make
Makefile:20: foo.d: No such file or directory
MOO
MOO
make: Nothing to be done for `all'.
[lester@localhost tmpfs]$
------------------------------------------------

which indeed is what you see when you run it in a directory with a good timestamp resolution such as a "tmpfs" directory under linux made with eg 'mount tmpfs ~/tmpfs/ -t tmpfs'.  :)

Unfortunately, when you run the same "make clean ; make" on a typical linux ext3 filesystem without extended attributes where stat returns timestamps discretised at the 1-second level, you instead only see ONE occurrence of the word MOO like this:

[------------------------------------------------
lester@localhost v3]$ make clean ; make
Makefile:20: foo.d: No such file or directory
MOO
make: Nothing to be done for `all'.
[lester@localhost v3]$
------------------------------------------------

corresponging to only one re-generation of "foo.d". [Run the above a few times - due to a race condition you may get lucky and see two MOOs a small fraction of the time.]

It is possible to "fix" things (i.e. make "make" function the same irrespective of the timestamp granularity of the underlying filesystem) by slowing down the process of building "foo.d" by inserting an appropriately long "sleep" after writing "foo.d". In the supplied makefile there is a "usleep 1100000" commented out which, if un-commented, should provide enough of a delay to fix things on a filesystem with a granularity of one second.

---------------------------------------------

Where is all this going?

(1) This is NOT supposed to be a complaint about make basing its rebuild decisions on timestamps - that's in make's blood. 

(2) Make cannot be held responsible for things outside its control, such as the poor time-resolution limitations in some filesystems.  Make has rules for dealing with "equal" time stamps which it must follow carefully (and probably does).

The point of the message is to highlight the freedom open in the interpretation of the words

"starts with a clean slate"

in the "remaking makefiles" part of the documentation:

   " ... After all makefiles have been checked, if any have actually been changed, make starts with a clean slate and reads all the makefiles over again. (It will also attempt to update each of them over again, but normally this will not change them again, since they are already up to date.)"

I am lead to believe that the mechanism by which the current make-implementation "starts with a clean slate" after rebuild of an include file is literally by internally calling "exec" with itself and with its original command-line-args.  [I've not checked the last statement in the sourcecode, so it could be wrong, but the rest of this message doesn't rely on it being true.]  Clearly there is a lot to be said for that approach from the standpoint of simplicity!  That's one line of code and you're going to be able to predict what it does.
If that is what make actually does (though the documentation doesn't REQUIRE that "starts with a clean state" be implemented in exactly that way) then the observed behaviour is probably 100% as expected.


Unfortunately this implementation choice means that the second (or n th) invocation of make forgets the order in which any just-made files were made, and is then at the mercy of the time-resolution of the filesystem of the currenty directory to carry that information over from make-invocation-(n) to make-invocation-(n+1), thereby introducing an un-necessary non-portability.

make-invocation-(n) may well know full-well that it generated file "a" before/after file "b", and even that it intended to do so, even if the filesystem timestamps are not capable of recording this and give them an identical timestamp.

In my opinion, make-invocation-(n+1) could make use of information that was known to make-invocation-(n) fully within the spirit and letter of "starts with a clean slate" ... to my mind those words are supposed to signify that make should re-process the whole makefile and any new includes and all variables from scratch, and should use the best information it has about file-update times etc, to build a new dependency graph and hit "go".  In my opinion those same words are not supposed to FORCE make to throw away information it already has which could be used as timestamp tie-breakers, and put it at the mercy of a worse approximation sitting on the filesystem.

[ For example, a single invocation of a non-recursive make can build 1000 files lighting fast, and they might all get the same timestamp, but make rightly doesn't care because it decided on their build order even before it made the first of them.  Rightly so.  I'd like make-auto-re-exec to be "as much like this as possible" within the constraints of the rules. ]

Naturally I admit that just because make "could" be implemented in some other way than it is at present, while maintining better-than-present portability across filesystems, does not mean that it would be easy to do so.

It would be way beyond me for a start!

But perhaps an expert could find some way of fixing things with a clever trick such as passing make-invocation-(n+1) an extra argument like "-W file.d" for every ".d" file that make thinks it just made which is forcing a re-exec.   That idea is probably broken, but something related might work.  I'm out of my depth.

Yours,

Christopher

Christopher Lester <kesterlester>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #12337:  Makefile added by kesterlester (712B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by eliz (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by kesterlester (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-06-07 psmith Item GroupNone Enhancement
    2007-03-29 kesterlester Attached File- Added Makefile, #12337

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code