bugmake - Bugs: bug #27609, Stupid inference rule for yacc...

 
 

bug #27609: Stupid inference rule for yacc files can clobber C sources!

Submitted by:  Kaz Kylheku <kkylheku>
Submitted on:  Mon 05 Oct 2009 07:36:09 PM UTC  
 
Severity: 3 - NormalItem Group: Bug
Status: NonePrivacy: Public
Assigned to: NoneOpen/Closed: Open
Component Version: 3.80Operating System: Any
Fixed Release: NoneTriage Status: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Tue 13 Oct 2009 06:50:11 PM UTC, comment #7:

Also, another consideration is that the prerequisite .y is involved in an explicit rule. If an implicit rule matches some file as a prerequisite, but that file is already used as a prerequisite in some rule, then those rules are competing. Maybe make should assume that the explicit rule takes precedence.

I'm not so confident about this because there are situations where something is a prerequisite to more than one rule. (Like the obvious case of a header file being a dependency for many objects).

Nevertheless, there are situations where prerequisites are typically understood to be processed by just one rule. You probably would not want to pass a .y file through yacc twice in two different rules. If such a situation exists, and one of the rules is generated from an implicit template, but the other one is explicit, somehow the explicit one should win.

Or some syntax could denote ``weak'' rules.

You know how there are order-only prerequisites; there could be a similar extension to denote weak-exclusive prerequisites, and another extension to denote clobbers.

For the sake of concrete discussion, let's use the characters @ and & for clobbers and exclusive prerequisites. The rule could then look like:

%.c: & %.y @ y.tab.c
$(YACC) ...
mv ...

So now, if the rule matches some ``foo'' stem, we have a target foo.c, a exclusive prerequisite foo.y, and a clobber y.tab.c.

The exclusive prerequisite tells make that this rule must be exclusive for that prerequisite. If two explicit rules are exlusive for the same prerequisite, then the situation is erroneous: a diagnostic is emitted and the make fails. If the exclusivity conflict is between an implicit and explicit rule (like the above case), then the implicit rule is treated as weak: it is resolved in favor of the explicit rule.

The clobber element y.tab.c informs make that the rule body destroys the file y.tab.c, which is neither a target nor prerequisite. Make will suppress this rule if y.tab.c appears as a target or prerequisite in any other rule (possibly with some warning diagnostic?). Moreover, if two or more rules both specify y.tab.c as a clobber, the execution of those rules will be serialized (in either order, just not parallel).

Note that this allows phony clobber files to be used as mutexes for serializing rules, which could be handy! Suppose that two rules exist which, say, do something destructive in the same CVS sandbox (like an update) which can modify the CVS/Entries and whatnot. They could have a clobber called ``cvs-mutex'', and make would thus not run those rules in parallel. This may be easier than inserting fake dependencies to enforce an order. I know there is .NOTPARALLEL, but that's a big hammer.

Kaz Kylheku <kkylheku>
Tue 13 Oct 2009 06:11:20 PM UTC, comment #6:

I've thought about this a little bit more. Basically, here is what may be the real issue: the rule interferes with objects which are already entangle din other rules.

Be it as it may that it's required by POSIX and let's take it for granted that we want that rule because people find it useful, existing projects depend on it, et cetera.

The problem is that if I have a makefile in which the file ``y.tab.c'' is already involved in an explicit rule.

If the makefile has this rule:

y.tab.c: foo.y
$(YACC) ...

make should not be interfering with this makefile by launching a ``mv y.tab.c. xxx'' command!

This problem would mostly go away if make was smart enough to know that target X appears on the left hand side of a rule, then it must suppress any hidden rules which use X as a temporary file.

Surely, it's okay not to try to conform to the POSIX rule, when the makefile explicitly /wants/ to make a target called y.tab.c.

Remmeber, the rule which makes y.tab.c could be making that y.tab.c from any one of several Yacc files, not necessarily the same Yacc file which the implicit rule wants to use.

In my case, it was my custom rule which made the y.tab.c, but then the implicit rule moved it to the target (my C source which got clobbered). Neither the source file y.tab.c, nor the target, were appropriate to the implicit rule.

And what about parallelized builds? What if the implicit rule applies more than one way; will it work correctly under parallel make? Suppose that a foo.c is to be implicitly made from a foo.y, and a bar.c is to be impliclitly made from a bar.y. Both actions will use y.tab.c as an intermediate file. Are there some hidden dependencies established which serialize the execution of the actions?

See, this is why Stallman came up with ``posix me harder''.

Kaz Kylheku <kkylheku>
Mon 05 Oct 2009 10:20:11 PM UTC, comment #5:

First let me point out that the behavior you're concerned about is actually required by the POSIX standard for make; the standard requires this as a built in rule to all conforming implementations:

So we can continue to discuss this to see if there's some way to alleviate the potential bad effects, but ultimately I'm not going to remove this rule or modify it in a way that breaks the standard (and also many makefiles out there which are written to expect this behavior).

The heuristics you suggest for detecting yacc-generated files are not acceptable. Any builtin rule MUST be completely portable to all platforms that make runs on, which includes not only all POSIX platforms with just /bin/sh, not bash, as shells, but also VMS, Windows, DOS, OS/2, and Amiga platforms (currently) where other tools like "diff" might not even exist. Furthermore, different implementations of yacc may well generate different output in the first 3 lines. This is simply not robust OR portable enough to be included as the builtin rule for GNU make.

The complaint about make updating foo.c from foo.y even though foo.c already exists is a primary feature of GNU make (see the manual discussion of implicit rule chaining). Even if I wanted to, which I don't, there's no way to change this without breaking the vast majority of GNU makefiles, which rely on this very useful behavior in some form.

A builtin rule that builds a .o directly from a .y doesn't help your situation unless the current rule to build a .c from a .y were also removed completely, and that's not going to happen.

You can, of course, disable these rules in your own makefiles in various ways.

Paul D. Smith <psmith>
Project Administrator
Mon 05 Oct 2009 08:59:27 PM UTC, comment #4:

By the way, this is quite bad. The behavior happens even if I have an explicit rule like this:

foo.o: foo.c

If there is a foo.y, foo.c is clobbered, even though foo.c exists.

This is so retarded.

The search for the prerequisite of foo.o must stop when foo.c is found.

I can understand that the search, upon not finding foo.c, might go on to try other suffixes like foo.y.

That is to say, the rule should be about making a foo.o from a foo.y, not making a foo.c from a foo.y!!!

Yes, sure. Okay. If I have a Makefile like this:

program: foo.o

and the only file in the directory is foo.y (and no foo.c), then go ahead and run yacc to make foo.c from foo.y, and compile foo.c to foo.o.

Actually we don't need foo.c in that case. We just need some temporary name. The rule has to handle the complete job of translating .y to .o:

$(YACC) $(YACCFLAGS) foo.y
mv y.tab.c <temporary_name>
$(CC) $(CFLAGS) -c <temporary_name> -o foo.o

Now this actually makes sense. If there is no .c file, why not infer that .y is the prerequisite for a .o?

Just don't involve a foo.c file in the inference chain.

<temporary_name> can be based on foo.y somehow (just not in a naive way like foo.c!) Even if it was foo.y.tab.c, or foo.y.out, or whatever.

There could even be an implicit rule which, given a foo.o, finds the prerequisite foo.y.tab.c.

It's less likely that someone would write a file by hand called foo.y.tab.c, than foo.c.

Kaz Kylheku <kkylheku>
Mon 05 Oct 2009 08:43:19 PM UTC, comment #3:

Re: what can be done about it.

Enough silly venting. How about the question of what can be done about it? I have an idea. How about a check to see whether the .c file to be clobbered is actually a yacc-generated file?

If the file already exists, only do the move if, say, the first three lines of the the file are different from the first three lines of y.tab.c.

Otherwise emit a diagnostic.

If you have GNU bash, or some other shell which supports process substitution, the logic can be expressed as:

# Do the rename if $target does not exist, or if it
# matches source in the first three lines.

if ![ -e "$target" ] || !diff <(head -3 "$source") <(head -3 "$target"); then
mv -- "$source" "$target"
else
echo "refusing update $target by replacing it with $source"
echo "target seems to have contents unrelated to $source"
fi

Of course, it has to be done according to whatever portability guidelines apply to the ruleset.

The downside would be that the rule refuses to do the move when it should: there is a false difference in the first three lines on every incremental rebuild. This might happen if the yacc implementation does something different in the first three lines (like adding a date stamp). Or (probably very unlikely) if edits in the actual yacc spec are reflected in an actual change in the first three lines.

The output of Bison does have a date stamp in the first three lines, but it's not generated; it's part of the boiler plate. For example:

#ifndef lint
static char const
yyrcsid[] = "$FreeBSD: src/usr.bin/yacc/skeleton.c,v 1.28 2000/01/17 02:04:06 bde Exp $";

This stamp won't change unless someone checks in the generated parser under version control which not only does RCS keyword expansion, but recognizes and expands the custom FreeBSD keyword.

Suppose your CVS does that. So you'd be doing a ``cvs up'', and then a ``make'', and ... oops! The parser.c got refreshed, so did parser.y, and now GNU make refuses to rename y.tab.c to parser.c, because CVS modified the $FreeBSD: ... $ text.

You would have to "rm parser.c" and run make again.

Kaz Kylheku <kkylheku>
Mon 05 Oct 2009 08:12:24 PM UTC, comment #2:

If I have two different yacc files that generate different parsers in the same directory, then I'm going to be aware of that situation and resolve it. If I don't resolve it, then they will both clobber y.tab.c, which is my fault. I'm bitten by my own folly, and not by some hidden rule that destroys source code.

That the yacc program clobbers y.tab.c is not the result of some secret rule which only emerges when you've created an important file (not by means of yacc) and named it y.tab.c! Everyone knows that yacc makes a y.tab.c file. Not everyone knows that make will clobber your foo.c, because a foo.y exists.

There are going to be other issues anyway. If I have two parsers that go into the same link, there will be clashes on all the YY stuff. It's not enough to just rename the files.

If I don't like the name y.tab.c, I can easily write a rule like this:

myparser.c: myparser.y
$(YACC) $(YFLAGS) -o $@ $<

If I want that to be an implicit rule for general .y -> .c conversion, I can do it like this:

%.c: %.y
$(YACC) $(YFLAGS) -o $@ $<

Now I have a rule which clobbers .c files. I wrote it; it's spelled out in my Makefile. If that overwrites a non-generated .c file, it serves me right!

Kaz Kylheku <kkylheku>
Mon 05 Oct 2009 07:56:33 PM UTC, comment #1:

This is how the default rule for building YACC files in GNU make has always been, for 20-odd years. From the GNU make manual section "Catalog of Rules":

Creating a statically-named "y.tab.c" from "foo.y" was an inane idea in the first place: what happens if you have two different YACC files that generate different parsers in the same directory? Ditto, of course, for lex.

I'm not sure what we can do about this.

Paul D. Smith <psmith>
Project Administrator
Mon 05 Oct 2009 07:36:09 PM UTC, original submission:

I have a project in which there is a lex grammar and yacc grammar. Let's call them foo.l and foo.y. It's somewhat of a tradition to use the same name for both. There is no interference because one is translated to a lex.yy.c, and the other to y.tab.c.

So, silly me, I decided to add a third file, a main driver program, which I called foo.c, thinking that all three files foo.l, foo.y and foo.c would be independent.

In my Makefile, I added the dependency, by naming foo.o as a prerequisite of the main program. But I did not add a rule for how foo.o is obtained; all my other .o files have implicitly deduced .c prerequisites.

Lo and behold, when I ran make, this happened:

mv -f y.tab.c foo.c

There goes my foo.c file! Apparently, foo.c is connected to foo.y by prerequisite inference.

What kind of moron writes implicit rules, to be shipped in the default rule set of a make program, which guess up .c file names and proceed to clobber them?

And since when has foo foo.c ever been related to foo.y as a derived object?

If I want my y.tab.c to be called something else, I will write the rule for it.

Luckily the contents of foo.c came from another file which is under version control. I lost only two very minor edits.

Kaz Kylheku <kkylheku>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by psmith (Posted a comment)
  • -unavailable- added by kkylheku (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    No Changes Have Been Made to This Item

    Back to the top


    Powered by Savane 3.1-cleanup