bugmake - Bugs: bug #46242, Race condition when input file is...

 
 

bug #46242: Race condition when input file is updated while compiling

Submitted by:  Egmont Koblinger <egmont>
Submitted on:  Sun 18 Oct 2015 05:28:22 PM UTC  
 
Severity: 3 - NormalItem Group: Enhancement
Status: NonePrivacy: Public
Assigned to: NoneOpen/Closed: Open
Component Version: 4.0Operating System: Any
Fixed Release: NoneTriage Status: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sun 18 Oct 2015 05:34:32 PM UTC, comment #1:

Clarification:

7. Go back to the terminal. make might have completed by now. If not, you might either -- if it has finished processing that file and is already working on another -- interrupt it, or wait for it to complete.

Egmont Koblinger <egmont>
Sun 18 Oct 2015 05:28:22 PM UTC, original submission:

There's an inherent race condition in the way "make" handles timestamps. If you save a newer version of a file while make is running and is just compiling that input file, the change won't be picked up by subsequent runs of "make", and you'll be left with an out-of-date binary.

Steps to reproduce:

1. Have some project.
2. Open one of the large source files that takes noticeable time to compile.
3. Edit this file in some valid way (changing the functionality) but do not save it yet.
4. In a terminal, run "make" to compile your project.
5. While make is running, go back to your editor, but keep your eyes on make's output.
6. When make launches that particular gcc (or whatever) that compiles this large file, save the new version.
7. Go back to the terminal. make might have completed by now. If not, you might either interrupt it, or wait for it to complete.
8. Execute make again to catch up with the edits you performed. Wait for it to complete.
9. Launch your app, notice that it does not include your recent changes.

The behavior is obvious from the way make checks the timestamps. The source of the file has an earlier timestamp than the compiled object file, since you saved the file before gcc wrote out its output. But these are not corresponding versions.

-----

I was thinking about it for a couple of days now whether it's a bug in gcc or make. gcc could be modified to touch back the object file to the timestamp when it started up, and that would protect against it. (It'd also need to write a temporary file and rename in the end, just in case it's interrupted - I'm not sure if it does this or not). However, this would then needed to be implemented in all compilers, and even in every one-off command that people use from Makefiles to create files (generated source code, translations updated with msgfmt, and a whole lot more). So I think this approach is not feasibe, and wouldn't be the right design. gcc's job is to compile, make's job is to do the orchestration, so in my opinion this should somehow be solved in make.

Make does obviously check the timestamp of all input files when it starts executing a rule. Upon the rule's completion, I think it should check the input files' timestamps again, and if any of them were updated in the mean time, it should re-execute the given rule.

This change, on its own, would prevent the above scenario to occur as long as you don't interrupt any of the make/gcc/whatever processes. However, a badly timed interrupt could still leave an obsolete compiled file with a new timestamp on the filesystem, causing subsequent makes not picking up the change. And by this "badly timed" I mean a giant window: the entire duration of re-running gcc or whichever compiler.

When make detects that one of the input files was altered, it could immediately remove the relevant output file (or touch back to an older timestamp) before re-executing the rule. This would not completely eliminate this race window, but would cut it down to magnitudes shorter. After all, I assume gcc exits immediately after creating its output (what else would it do?) and make's very next steps would be to check the timestamps.

As mentioned in #46193 comment 1, ideally rules should write the output to a temporary file and atomically rename. If this is not the private internal business of a rule, but rather something make is in control of, the ideal workflow would be: First run the external command (e.g. gcc) telling it explicitly to store its output in a temporary file, then check if any of the source timestamps changed (and rerun gcc if it did), and finally rename. This probably couldn't be done automatically for any rule, since it'd break when the rule spells out the dest file name rather than referring to it as $@. But maybe it could automatically be enabled for rules that do contain $@ and this wouldn't break too many Makefiles. Or maybe we don't need to overcomplicate and we're okay with this tiny race window explained at the end of the previous paragraph.

Or perhaps you have an even better idea :)

Egmont Koblinger <egmont>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by psmith (Updated the item)
  • -unavailable- added by egmont (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Tue 31 May 2016 12:07:11 PM UTCpsmithItem GroupBug=>Enhancement

    Back to the top


    Powered by Savane 3.1-cleanup1