bugmake - Bugs: bug #40056, make should automatically detect...

 
 

bug #40056: make should automatically detect targets with low resolution timestamps

Submitted by:  James Ralston <ralston>
Submitted on:  Wed 18 Sep 2013 05:12:18 AM UTC  
 
Severity: 3 - NormalItem Group: Bug
Status: NonePrivacy: Public
Assigned to: NoneOpen/Closed: Open
Component Version: 4.0Operating System: POSIX-Based
Fixed Release: NoneTriage Status: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 04 Mar 2014 12:38:56 AM UTC, comment #6:

I will definitely fix this before the next release.

Paul D. Smith <psmith>
Project Administrator
Tue 04 Mar 2014 12:10:10 AM UTC, comment #5:

Ping?

make's behavior is still wrong (as I detailed in comment #3), and make is still broken.

While I still think adding a "precision" element to the "file" struct is the best solution, my patch to (essentially) apply .LOW_RESOLUTION_TIMESTAMP automaticallys work around the problem.

Thoughts?

James Ralston <ralston>
Tue 24 Sep 2013 03:12:50 AM UTC, comment #4:

Philip is correct: I can find no evidence that any version of glibc
(released or development) implements pathconf(path,
_PC_TIMESTAMP_RESOLUTION).

So, unfortunately, currently the only way under Linux to determine if a file has nanosecond timestamp resolution is to simply test whether the file's nanosecond timestamp is nonzero.

I can assert that recent versions of "cp" and "install" do preserve nanosecond timestamps. So if glibc implements pathconf(path,
_PC_TIMESTAMP_RESOLUTION) someday, it should be possible to use it to determine file timestamp granularity without looking at the nanosecond timestamp file, because any system new enough to have pathconf(path,
_PC_TIMESTAMP_RESOLUTION) would also have non-broken versions of "cp" and "install".

James Ralston <ralston>
Tue 24 Sep 2013 03:05:32 AM UTC, comment #3:

In response to Paul's 2013-09-22T07:15:04+0000 comment:

To be clear: my patch simply applies the .LOW_RESOLUTION_TIMESTAMP special target's logic (add 0.999999999 seconds to a timestamp that lacks nanosecond resolution) implicitly to all low-resolution timestamp files. That's it. My patch doesn't "always round up timestamps to the next second"; it just uses the same technique that .LOW_RESOLUTION_TIMESTAMP does.

But I agree with you, in that I think .LOW_RESOLUTION_TIMESTAMP is both wrong and a hack. It's an attempt to work around the root problem with make's handling of timestamps: make incorrectly attributes additional precision on timestamps where no such precision exists.

To illustrate why make's behavior is wrong, let's say that we are attempting to calculate the area of a rectangle. We have two measurements: 6.4 meters for length, and 3.78 meters for width.

If we multiply these two measurements with a calculator, the calculator will produce:

6.4 × 3.78 = 24.192

The answer of 24.192 m² is clear, simple... and wrong. It is wrong because it implies precision that does not exist in the original measurements.

Specifically, since the first factor has 2 significant digits and the second factor has 3, the final answer cannot contain more than 2 significant digits. So the correct answer to "what is the area of this rectangle?" is 24 m², not 24.192 m².

This might seem non-intuitive, but it's easier to understand if you consider the accuracy range of the original measurements: the instrument used to measure the length was accurate only to a decimeter, while the instrument used to measure the width was accurate to a centimeter. So, accounting for measurement error, these are the minimum/maximum potential areas:

6.3 × 3.77 = 23.751
6.5 × 3.79 = 24.635

This is why we cannot be any more accurate that "24 m²" in our answer.

The same principle is true for timestamp comparisons. Say that we have two different files, foo and bar, with the following timestamps:

foo: 2013-09-23 15:55:01.292619574
bar: 2013-09-23 15:55:01

Internally, make converts these timestamps into nanoseconds since the epoch:

foo: 1379966105292619574
bar: 1379966105000000000

If foo is a dependency of bar, and make needs to decide whether bar needs to be rebuilt, it will compare the timestamps, determine that 1379966105292619574 is greater (newer) than 1379966105000000000, and conclude that bar needs to be rebuilt.

This logic is clear, simple... and wrong. It is wrong because in its comparison, make relied on the erroneous precision that it conferred onto bar, when it converted bar's timestamp to nanoseconds. But bar has no such precision.

In order to perform timestamp comparisons correctly, make must constrain the granularity of its comparison to the precision of the least precise element being compared. In this example, since bar is precise to the second, and foo is precise to the nanosecond, that means that make must compare foo and bar only to the seconds to decide whether foo is more recent than bar.

Thus, the arguably best way to correct make's behavior is to add a "precision" element to the "file" struct, and then take that precision into account when performing all timestamp comparisons.

However, that would be a non-trivial change, and since this is the first time I've looked at make's source code, I'm not confident I can implement the change correctly. (Plus, I'm not going to commit to such a change unless I have some assurance that make's maintainers would be receptive to it.)

That's why I just generalized the .LOW_RESOLUTION_TIMESTAMP special target's logic implicitly to all low-resolution timestamp files. I fully admit that doing so is a hack, but 1) it's no less of a hack than the existence of .LOW_RESOLUTION_TIMESTAMP to begin with, and 2) it produces more correct behavior than make exhibits currently.

And to be clear, make's current behavior with respect to timestamp comparisons means that in our environment (dependencies on ext4; targets on ext3), make is broken to the point of being unusable. I'm not criticizing make's timestamp comparison logic because we are seeking "perfect" instead of "good enough"; I'm criticizing it because it produces incorrect, broken behavior.

James Ralston <ralston>
Sun 22 Sep 2013 07:15:04 PM UTC, comment #2:

My concern about this patch is that there are plenty of examples of files which have an NS field of 0, even on ext4 filesystems which support sub-second timestamps. This could be true just by chance, but it's actually true much more often because many tools do not support sub-second timestamps. For example, the tar format has no facility for it. For a long time although you could retrieve sub-second timestamp information with stat(2), there was no way to set it, so programs like touch, cp -p, etc. could not set anything but full second timestamps. Etc.

I wrote a little program to print files with a 0 NS field on my ext4 partition and there were a LOT of them.

What your patch does is always round up timestamps to the next second, if NS == 0. As far as I can see this would not be what you want. If you had a file with a highres timestamp of x.1 and another file with a timestamp of x.0, then the change would assume that the x.0 should be x+1, or newer than the first one.

I just don't think this is right.

One option would be to allow the .LOW_RESOLUTION_TIME target to be defined with no prerequisites, which would mean that all targets should be assumed to have lowres timestamps. In this mode of course we wouldn't print any warnings about it.

Paul D. Smith <psmith>
Project Administrator
Sun 22 Sep 2013 06:28:49 PM UTC, comment #1:

Philip Guether writes on the bug-make mailing list:

Paul D. Smith <psmith>
Project Administrator
Wed 18 Sep 2013 05:12:18 AM UTC, original submission:

We use make on our Red Hat Enterprise Linux systems, using the ext3 and (now) ext4 filesystems.

While the stat(2) call on Linux has supported nanosecond resolution timestamps since kernel 2.5.48, and the autoconf/configure script for make properly detects high resolution timestamp support, the ext3 filesystem has no support for actually storing nanosecond timestamps, while the ext4 filesystem does.

So, when the source of a make rule is on ext4, and the target is on ext3, make will always think that the target is older than the source, because the target's timestamp will have the nanosecond resolution truncated. This has been biting us since we introduced ext4 into our environment.

We know about the .LOW_RESOLUTION_TIME special target. But using .LOW_RESOLUTION_TIME is both 1) intractable and 2) wrong.

Our environment has hundreds of extremely complex Makefiles with many rules and many targets. Since we cannot reliably know whether any arbitrary target will ultimately reside on an ext3 or ext4 filesystem, we would have to set the .LOW_RESOLUTION_TIME target for every target, and always remember to do so for every future target. This is impractical to the point of being impossible.

The fundamental issue here is that kernel and filesystem support is necessary to support high resolution timestamps. On some systems, these are one and the same, but on GNU/Linux systems, they are completely independent qualities. The former can be detected at compile time, but the latter can only be detected at run time. The current code correctly does the former, but does not do the latter.

The statfs(2) system call will return the f_type field in the struct statfs, but unfortunately, the field is the same for all variants of ext (ext2, ext3, ext4). So as far as I know (although I would be happy to be corrected), there is no library or system call that will reveal whether the filesystem that an arbitrary file is on supports high resolution timestamps.

However, all is not lost: one can infer that a file is stored on a filesystem that lacks high resolution timestamps by the fact that the nanosecond component of the timestamp is zero.

Given that, I would propose the following change to make's timestamp comparison behavior: if the OS supports high resolution timestamps, but the value of a target file's nanosecond timestamp is 0, then make implicitly applies the .LOW_RESOLUTION_TIME special target to the target file.

In the vast majority of cases, this would obviate the need to explicitly use the .LOW_RESOLUTION_TIME special target, as make will automatically detect when the target has a low resolution timestamp.

I searched both the bug lists and the mailing list archives, and I didn't see any previous discussion specific to this issue, so I've attached a patch to implement this behavior. Is there any reason why it cannot be applied?

James Ralston <ralston>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #29170:  make.patch added by ralston (2kB - application/octet-stream - patch to automatically apply .LOW_RESOLUTION_TIME as necessary)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by psmith (Posted a comment)
  • -unavailable- added by ralston (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 18 Sep 2013 05:12:18 AM UTCralstonAttached File-=>Added make.patch, #29170

    Back to the top


    Powered by Savane 3.1-cleanup