bugmake - Bugs: bug #27609, Stupid inference rule for yacc...

 
 

bug #27609: Stupid inference rule for yacc files can clobber C sources!

Submitter:  Kaz Kylheku <kkylheku>
Submitted:  Mon 05 Oct 2009 07:36:09 PM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  3.80 Operating System:  Any
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 06 Oct 2013 02:34:50 AM UTC, comment #12: 

It's hard to diagnose problems based on general descriptions.  Please provide specific examples of what you did and what the result was.

However, it's definitely true that if you're relying on some of the built-in rules then turning them all off with .SUFFIXES: will break that.

Please see the GNU make manual section on suffix rules.  First, when defining suffixes you add them one suffix at a time, not as a group (use ".o" and ".c", not ".o.c", for example).  Second, suffixes as .SUFFIXES: prerequisites only adds to the existing suffix list, it doesn't replace the existing suffix list.

If you're trying to keep only some of the rules then first you have to turn them all off, then add back just the ones you want:


.SUFFIXES:
.SUFFIXES: .c .o



Paul D. Smith <psmith>
Group administrator
Sun 06 Oct 2013 02:01:37 AM UTC, comment #11: 

I tried .SUFFIXES: but now make tried to build the program even though lex.yy.o is a prerequisite, and is missing, without any complaints. This is, of course, because I have a rule which gives a dependency from lex.yy.o to lex.yy.c, but which has no recipe.

I then added .o.c to suffixes, in hopes that  this would turn on rules involving .o and .c files only.

The lex.yy.o: lex.yy.c rule now started to work again.

But alas, the bad rule now kicked in!

$ make
lex -i -8 hc.l
cc -g -Wall -W -ansi -D_XOPEN_SOURCE=500    -c -o lex.yy.o lex.yy.c
lex.yy.c:4548:17: warning: ‘yyunput’ defined but not used [-Wunused-function]
lex.yy.c:4589:16: warning: ‘input’ defined but not used [-Wunused-function]
lex  -t hc.l > hc.c   <--- HUH?

Is that a bug? This is make 3.81 on Ubuntu 11.04.

If not, what is the real way to put the through the heart of this beast (and anything like it, known or unknown), without losing the useful implicit rules?

How can I say, "I want only the implicit rules which involve only these file types and do not involve any other file types."

Kaz Kylheku <kkylheku>
Sat 05 Oct 2013 10:18:52 PM UTC, comment #10: 

Yes, both .y.c and .l.c suffix rules are required by the POSIX standard.  However, they've been present in GNU make for over 20 years, since before POSIX standardized make, so while POSIX requirements are one reason to keep them, their existence cannot be blamed on POSIX.

I recommend getting into the habit of adding a ".SUFFIXES:" target to your makefiles to disable the built-in suffix rules, so that the worst that can happen if your rules fail to apply for some reason is that make complains it doesn't know how to build the target.

Paul D. Smith <psmith>
Group administrator
Sat 05 Oct 2013 09:43:46 PM UTC, comment #9: 

What if these dangerous implicit rules made a rotating backup of their victim? What is the downside?

    mv foo.c.keep.1 foo.c.keep.2
    mv foo.c foo.c.keep.1
    mv y.tab.c foo.c

Kaz Kylheku <kkylheku>
Sat 05 Oct 2013 09:30:56 PM UTC, comment #8: 

I was just burned by this again! I have a small project with "hc.l" lexer, and a "hc.c" source file.  It's been working fine.

My Makefile was generating lex.yy.c from hc.l, compiling hc.c to hc.o, and lex.yy.c to lex.yy.o, and linking everything.

All of a sudden though, for some reason, an internal rule in make kicked in and did this:

   lex -t hc.l > hc.c

oops, my source file is gone; all I have is the most recent git repo copy which doesn't have all my recent work.

Is this is another POSIX-required misfeature?

This is the sort of thing for which Stallman invented --posix-me-harder.

Kaz Kylheku <kkylheku>
Tue 13 Oct 2009 06:50:11 PM UTC, comment #7: 

Also, another consideration is that the prerequisite .y is involved in an explicit rule.   If an implicit rule matches some file as a prerequisite, but that file is already used as a prerequisite in some rule, then those rules are competing. Maybe make should assume that the explicit rule takes precedence.

I'm not so confident about this because there are situations where something is a prerequisite to more than one rule. (Like the obvious case of a header file being a dependency for many objects).

Nevertheless, there are situations where prerequisites are typically understood to be processed by just one rule. You probably would not want to pass a .y file through yacc twice in two different rules. If such a situation exists, and one of the rules is generated from an implicit template, but the other one is explicit, somehow the explicit one should win.

Or some syntax could denote ``weak'' rules.

You know how there are order-only prerequisites; there could be a similar extension to denote weak-exclusive prerequisites, and another extension to denote clobbers.

For the sake of concrete discussion, let's use the characters @ and & for clobbers and exclusive prerequisites. The rule could then look like:

  %.c: & %.y @ y.tab.c
     $(YACC) ...
     mv ...


So now, if the rule matches some ``foo'' stem, we have a target foo.c, a exclusive prerequisite foo.y, and a clobber y.tab.c.

The exclusive prerequisite tells make that this rule must be exclusive for that prerequisite. If two explicit rules are exlusive for the same prerequisite, then the situation is erroneous: a diagnostic is emitted and the make fails. If the exclusivity conflict is between an implicit and explicit rule (like the above case), then the implicit rule is treated as weak: it is resolved in favor of the explicit rule.

The clobber element y.tab.c informs make that the rule body destroys the file y.tab.c, which is neither a target nor prerequisite.  Make will suppress this rule if y.tab.c appears as a target or prerequisite in any other rule (possibly with some warning diagnostic?). Moreover, if two or more rules both specify y.tab.c as a clobber, the execution of those rules will be serialized (in either order, just not parallel).

Note that this allows phony clobber files to be used as mutexes for serializing rules, which could be handy! Suppose that two rules exist which, say, do something destructive in the same CVS sandbox (like an update) which can modify the CVS/Entries and whatnot. They could have a clobber called ``cvs-mutex'', and make would thus not run those rules in parallel. This may be easier than inserting fake dependencies to enforce an order. I know there is .NOTPARALLEL, but that's a big hammer.



Kaz Kylheku <kkylheku>
Tue 13 Oct 2009 06:11:20 PM UTC, comment #6: 

I've thought about this a little bit more. Basically, here is what may be the real issue: the rule interferes with objects which are already entangle din other rules.

Be it as it may that it's required by POSIX and let's take it for granted that we want that rule because people find it useful, existing projects depend on it, et cetera.

The problem is that if I have a makefile in which the file ``y.tab.c'' is already involved in an explicit rule.

If the makefile has this rule:

   y.tab.c: foo.y
        $(YACC) ...

make should not be interfering with this makefile by launching a ``mv y.tab.c. xxx'' command!

This problem would mostly go away if make was smart enough to know that target X appears on the left hand side of a rule, then it must suppress any hidden rules which use X as a temporary file.

Surely, it's okay not to try to conform to the POSIX rule, when the makefile explicitly /wants/ to make a target called y.tab.c.

Remmeber, the rule which makes y.tab.c could be making that y.tab.c from any one of several Yacc files, not necessarily the same Yacc file which the implicit rule wants to use.

In my case, it was my custom rule which made the y.tab.c, but then the implicit rule moved it to the target (my C source which got clobbered). Neither the source file y.tab.c, nor the target, were appropriate to the implicit rule.

And what about parallelized builds? What if the implicit rule applies more than one way; will it work correctly under parallel make?  Suppose that a foo.c is to be implicitly made from a foo.y, and a bar.c is to be impliclitly made from a bar.y. Both actions will use y.tab.c as an intermediate file.  Are there some hidden dependencies established which serialize the execution of the actions?

See, this is why Stallman came up with ``posix me harder''.

Kaz Kylheku <kkylheku>
Mon 05 Oct 2009 10:20:11 PM UTC, comment #5: 

First let me point out that the behavior you're concerned about is actually required by the POSIX standard for make; the standard requires this as a built in rule to all conforming implementations:


.y.c:
    $(YACC) $(YFLAGS) $<
    mv y.tab.c $@


So we can continue to discuss this to see if there's some way to alleviate the potential bad effects, but ultimately I'm not going to remove this rule or modify it in a way that breaks the standard (and also many makefiles out there which are written to expect this behavior).

The heuristics you suggest for detecting yacc-generated files are not acceptable.  Any builtin rule MUST be completely portable to all platforms that make runs on, which includes not only all POSIX platforms with just /bin/sh, not bash, as shells, but also VMS, Windows, DOS, OS/2, and Amiga platforms (currently) where other tools like "diff" might not even exist.  Furthermore, different implementations of yacc may well generate different output in the first 3 lines.  This is simply not robust OR portable enough to be included as the builtin rule for GNU make.

The complaint about make updating foo.c from foo.y even though foo.c already exists is a primary feature of GNU make (see the manual discussion of implicit rule chaining).  Even if I wanted to, which I don't, there's no way to change this without breaking the vast majority of GNU makefiles, which rely on this very useful behavior in some form.

A builtin rule that builds a .o directly from a .y doesn't help your situation unless the current rule to build a .c from a .y were also removed completely, and that's not going to happen.

You can, of course, disable these rules in your own makefiles in various ways.

Paul D. Smith <psmith>
Group administrator
Mon 05 Oct 2009 08:59:27 PM UTC, comment #4: 

By the way, this is quite bad. The behavior happens even if I have an explicit rule like this:

  foo.o: foo.c

If there is a foo.y, foo.c is clobbered, even though foo.c exists.

This is so retarded.

The search for the prerequisite of foo.o must stop when foo.c is found.

I can understand that the search, upon not finding foo.c, might go on to try other suffixes like foo.y.

That is to say, the rule should be about making a foo.o from a foo.y, not making a foo.c from a foo.y!!!

Yes, sure. Okay. If I have a Makefile like this:

  program: foo.o

and the only file in the directory is foo.y (and no foo.c), then go ahead and run yacc to make foo.c from foo.y, and compile foo.c to foo.o.

Actually we don't need foo.c in that case. We just need some temporary name. The rule has to handle the complete job of translating .y to .o:

   $(YACC) $(YACCFLAGS) foo.y
   mv y.tab.c <temporary_name>
   $(CC) $(CFLAGS) -c <temporary_name> -o foo.o

Now this actually makes sense. If there is no .c file, why not infer that .y is the prerequisite for a .o?

Just don't involve a foo.c file in the inference chain.

<temporary_name> can be based on foo.y somehow (just not in a naive way like foo.c!)  Even if it was foo.y.tab.c, or foo.y.out, or whatever.

There could even be an implicit rule which, given a foo.o, finds the prerequisite foo.y.tab.c.

It's less likely that someone would write a file by hand called foo.y.tab.c, than foo.c.









Kaz Kylheku <kkylheku>
Mon 05 Oct 2009 08:43:19 PM UTC, comment #3: 

Re: what can be done about it.

Enough silly venting. How about the question of what can be done about it? I have an idea. How about a check to see whether the .c file to be clobbered is actually a yacc-generated file?

If the file already exists, only do the move if, say, the first three lines of the the file are different from the first three lines of y.tab.c.

Otherwise emit a diagnostic.

If you have GNU bash, or some other shell which supports process substitution, the logic can be expressed as:

  # Do the rename if $target does not exist, or if it
  # matches source in the first three lines.
 
  if ![ -e "$target" ] || !diff <(head -3 "$source") <(head -3 "$target"); then
    mv -- "$source" "$target"
  else
    echo "refusing update $target by replacing it with $source"
    echo "target seems to have contents unrelated to $source"
  fi

Of course, it has to be done according to whatever portability guidelines apply to the ruleset.

The downside would be that the rule refuses to do the move when it should: there is a false difference in the first three lines on every incremental rebuild. This might happen if the yacc implementation does something different in the first three lines (like adding a date stamp). Or (probably very unlikely) if edits in the actual yacc spec are reflected in an actual change in the first three lines.

The output of Bison does have a date stamp in the first three lines, but it's not generated; it's part of the boiler plate. For example:

#ifndef lint
static char const
yyrcsid[] = "$FreeBSD: src/usr.bin/yacc/skeleton.c,v 1.28 2000/01/17 02:04:06 bde Exp $";

This stamp won't change unless someone checks in the generated parser under version control which not only does RCS keyword expansion, but recognizes and expands the custom FreeBSD keyword.

Suppose your CVS does that. So you'd be doing a ``cvs up'', and then a ``make'', and ... oops! The parser.c got refreshed, so did parser.y, and now GNU make refuses to rename y.tab.c to parser.c, because CVS modified the $FreeBSD: ... $ text.

You would have to "rm parser.c" and run make again.

Kaz Kylheku <kkylheku>
Mon 05 Oct 2009 08:12:24 PM UTC, comment #2: 

If I have two different yacc files that generate different parsers in the same directory, then I'm going to be aware of that situation and resolve it. If I don't resolve it, then they will both clobber y.tab.c, which is my fault. I'm bitten by my own folly, and not by some hidden rule that destroys source code.

That the yacc program clobbers y.tab.c is not the result of some secret rule which only emerges when you've created an important file (not by means of yacc) and named it y.tab.c! Everyone knows that yacc makes a y.tab.c file.  Not everyone knows that make will clobber your foo.c, because a foo.y exists.

There are going to be other issues anyway. If I have two parsers that go into the same link, there will be clashes on all the YY stuff. It's not enough to just rename the files.

If I don't like the name y.tab.c, I can easily write a rule like this:

 myparser.c: myparser.y
    $(YACC) $(YFLAGS) -o $@ $<

If I want that to be an implicit rule for general .y -> .c conversion, I can do it like this:

 %.c: %.y
    $(YACC) $(YFLAGS) -o $@ $<

Now I have a rule which clobbers .c files. I wrote it; it's spelled out in my Makefile. If that overwrites a non-generated .c file, it serves me right!

Kaz Kylheku <kkylheku>
Mon 05 Oct 2009 07:56:33 PM UTC, comment #1: 

This is how the default rule for building YACC files in GNU make has always been, for 20-odd years.  From the GNU make manual section "Catalog of Rules":


Yacc for C programs
    n.c is made automatically from n.y by running Yacc with the command `$(YACC) $(YFLAGS)'.
Lex for C programs
    n.c is made automatically from n.l by running Lex. The actual command is `$(LEX) $(LFLAGS)'.


Creating a statically-named "y.tab.c" from "foo.y" was an inane idea in the first place: what happens if you have two different YACC files that generate different parsers in the same directory?  Ditto, of course, for lex.

I'm not sure what we can do about this.

Paul D. Smith <psmith>
Group administrator
Mon 05 Oct 2009 07:36:09 PM UTC, original submission:  

I have a project in which there is a lex grammar and yacc grammar. Let's call them foo.l and foo.y.   It's somewhat of a tradition to use the same name for both. There is no interference because one is translated to a lex.yy.c, and the other to y.tab.c.

So, silly me, I decided to add a third file, a main driver program, which I called foo.c, thinking that all three files foo.l, foo.y and foo.c would be independent.

In my Makefile, I added the dependency, by naming foo.o as a prerequisite of the main program. But I did not add a rule for how foo.o is obtained; all my other .o files have implicitly deduced .c prerequisites.

Lo and behold, when I ran make, this happened:

  mv -f y.tab.c foo.c

There goes my foo.c file! Apparently, foo.c is connected to foo.y by prerequisite inference.

What kind of moron writes implicit rules, to be shipped in the default rule set of a make program, which guess up .c file names and proceed to clobber them?

And since when has foo foo.c ever been related to foo.y as a derived object?

If I want my y.tab.c to be called something else, I will write the rule for it.

Luckily the contents of foo.c came from another file which is under version control. I lost only two very minor edits.

Kaz Kylheku <kkylheku>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by kkylheku (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2013-09-22 psmith Item GroupBug Enhancement

    Back to the top

    Powered by Savane 3.13-3230.
    Corresponding source code