bugmake - Bugs: bug #46242, Race condition when input file is...

 
 

bug #46242: Race condition when input file is updated while compiling

Submitter:  Egmont Koblinger <egmont>
Submitted:  Sun 18 Oct 2015 05:28:22 PM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.0 Operating System:  Any
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Sun 18 Oct 2015 05:34:32 PM UTC, comment #1: 

Clarification:

7. Go back to the terminal. make might have completed by now. If not, you might either -- if it has finished processing that file and is already working on another -- interrupt it, or wait for it to complete.

Egmont Koblinger <egmont>
Sun 18 Oct 2015 05:28:22 PM UTC, original submission:  

There's an inherent race condition in the way "make" handles timestamps. If you save a newer version of a file while make is running and is just compiling that input file, the change won't be picked up by subsequent runs of "make", and you'll be left with an out-of-date binary.

Steps to reproduce:

1. Have some project.
2. Open one of the large source files that takes noticeable time to compile.
3. Edit this file in some valid way (changing the functionality) but do not save it yet.
4. In a terminal, run "make" to compile your project.
5. While make is running, go back to your editor, but keep your eyes on make's output.
6. When make launches that particular gcc (or whatever) that compiles this large file, save the new version.
7. Go back to the terminal. make might have completed by now. If not, you might either interrupt it, or wait for it to complete.
8. Execute make again to catch up with the edits you performed. Wait for it to complete.
9. Launch your app, notice that it does not include your recent changes.

The behavior is obvious from the way make checks the timestamps. The source of the file has an earlier timestamp than the compiled object file, since you saved the file before gcc wrote out its output. But these are not corresponding versions.

-----

I was thinking about it for a couple of days now whether it's a bug in gcc or make. gcc could be modified to touch back the object file to the timestamp when it started up, and that would protect against it. (It'd also need to write a temporary file and rename in the end, just in case it's interrupted - I'm not sure if it does this or not). However, this would then needed to be implemented in all compilers, and even in every one-off command that people use from Makefiles to create files (generated source code, translations updated with msgfmt, and a whole lot more). So I think this approach is not feasibe, and wouldn't be the right design. gcc's job is to compile, make's job is to do the orchestration, so in my opinion this should somehow be solved in make.

Make does obviously check the timestamp of all input files when it starts executing a rule. Upon the rule's completion, I think it should check the input files' timestamps again, and if any of them were updated in the mean time, it should re-execute the given rule.

This change, on its own, would prevent the above scenario to occur as long as you don't interrupt any of the make/gcc/whatever processes. However, a badly timed interrupt could still leave an obsolete compiled file with a new timestamp on the filesystem, causing subsequent makes not picking up the change. And by this "badly timed" I mean a giant window: the entire duration of re-running gcc or whichever compiler.

When make detects that one of the input files was altered, it could immediately remove the relevant output file (or touch back to an older timestamp) before re-executing the rule. This would not completely eliminate this race window, but would cut it down to magnitudes shorter. After all, I assume gcc exits immediately after creating its output (what else would it do?) and make's very next steps would be to check the timestamps.

As mentioned in #46193 comment 1, ideally rules should write the output to a temporary file and atomically rename. If this is not the private internal business of a rule, but rather something make is in control of, the ideal workflow would be: First run the external command (e.g. gcc) telling it explicitly to store its output in a temporary file, then check if any of the source timestamps changed (and rerun gcc if it did), and finally rename. This probably couldn't be done automatically for any rule, since it'd break when the rule spells out the dest file name rather than referring to it as $@. But maybe it could automatically be enabled for rules that do contain $@ and this wouldn't break too many Makefiles. Or maybe we don't need to overcomplicate and we're okay with this tiny race window explained at the end of the previous paragraph.

Or perhaps you have an even better idea :)

Egmont Koblinger <egmont>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Updated the item)
  • -email is unavailable- added by egmont (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-05-31 psmith Item GroupBug Enhancement

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code