bugmake - Bugs: bug #33034, "Makefile:23: *** mixed...

 
 

bug #33034: "Makefile:23: *** mixed implicit and normal rules. Stop." for Linux kernel out of source builds

Submitted by:  None
Submitted on:  Sat 09 Apr 2011 10:42:24 AM UTC  
 
Severity: 3 - NormalItem Group: Bug
Status: FixedPrivacy: Public
Assigned to: Paul D. Smith <psmith>Open/Closed: Closed
Component Version: 3.82Operating System: POSIX-Based
Fixed Release: SCMTriage Status: None

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sun 20 Oct 2013 05:09:53 PM UTC, comment #21:

I have pushed a change to the Git repository.

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Sun 13 Oct 2013 02:34:15 AM UTC, comment #20:

It's great to hear that such a simple fix addresses this regression.

As soon as this patch goes in, I'll follow up to the Debian bug about getting the patch backported, which should allow closing that bug and moving make from experimental to sid (and subsequently uploading 4.00).

Josh Triplett <joshtriplett>
Sun 13 Oct 2013 01:59:50 AM UTC, comment #19:

The attached version of boyski's patch with, as Paul suggests, just the change from fatal to error and the addition of "warning: ", fixes the kernel eg 2.6.32-5 driver build problem I otherwise have with eg make-3.82, make-4.0. (I've been using a copy of Debian's older version, with its memory corruption bugs, for my kernel driver builds.)

(file #29365)

Martin Dorey <mdorey>
Sat 12 Oct 2013 06:29:32 PM UTC, comment #18:

> That's disappointing.


I've already explained why. Makefile syntax, unlike the C ABI where you can always just define a new function, is not inherently extensible except in very limited ways, and I'm not interested in creating ever more obscure syntactical aberrations and/or adding tons of special backward-compatibility flags with their attendant headaches. In my opinion makefiles already have more than enough syntactical foibles. Take a look through the actual backward-compatibility issues raised in the NEWS file.

Of course, this is open source so you're free to provide a version of GNU make that does provide these types of options for backward compatibility, if you feel it's that important.

My opinion on backward-compat is that we make an effort to avoid backward-incompatibilities and try to ensure they have the smallest possible impact. This change would likely have never been noticed if it hadn't affected Linux, and it would not be a big deal (only 3 files needed to be changed) except so many people apparently want to build old versions of the kernel. How many people go back and build OpenOffice 1.0 on their brand new Ubuntu 14.04 systems, and so would care if the OpenOffice 1.0 makefiles no longer worked with GNU make 4.0?

If you cannot change your makefiles to avoid the backward-compat issues, you can always keep multiple versions of GNU make. Unlike other tools like Java, GCC, etc. make is a single, simple executable that can be placed anywhere you want and named anything you want, and it will continue to work just fine. Take two, they're small!

> I'm curious what you infer that from.


Two things from the manual: first the description of pattern rules explicitly describe what happens if you include a non-pattern prerequisite, but says nothing about allowing a non-pattern target. Second GNU make lists the different types of lines in a makefile and none of those allows a mash-up of implicit and explicit rules like this.

And two things from experience: if you write "foo bar%:" it works, but if you write "bar% foo:" it fails with an error. Also if you add a pattern prerequisite "foo bar% : baz%" that also fails to work (the target 'baz%', literally, is considered a prerequisite of the target 'foo'). Certainly any syntactical rules THAT obscure WOULD be mentioned explicitly in the manual, if this were intended to be a supported feature.

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Sat 12 Oct 2013 05:43:01 PM UTC, comment #17:

> If it turns out that simply changing the fatal() to an error() is sufficient to resolve this, I'm willing to make that change (I won't add a flag for it; the message will simply say the syntax is unsupported and should be updated).


Thank you.

> GNU make is not the kernel, and I reject attempts to impose the development ethos of the kernel on GNU make.


That's disappointing.

Personally, when holding up software as the model of backward compatibility, I usually point to debhelper, which still supports packages written for the very first version.

> it's easily, IMO, inferrable that the syntax is not intended to be valid


I'm curious what you infer that from. I haven't found anything to that effect outside of the 3.82 release notes.

> That bug should (a) not be release-critical, and (b) should be filed against the Linux kernel headers package, not GNU make.


The kernel has already been fixed. The RC bug on Make is for introducing a regression in Makefile parsing.

Josh Triplett <joshtriplett>
Sat 12 Oct 2013 04:25:09 PM UTC, comment #16:

I admit I didn't understand the frustration level with this change. The Linux kernel was the only package that ever reported any real issues, and there were (by my count) a total of three patches (all minor) over the span of a couple of months after the 3.82 release in 2010 to fix those problems.

The release-critical bug in Debian is, IMO, completely unjustified (and I didn't know about it anyway). That bug should (a) not be release-critical, and (b) should be filed against the Linux kernel headers package, not GNU make.

If it turns out that simply changing the fatal() to an error() is sufficient to resolve this, I'm willing to make that change (I won't add a flag for it; the message will simply say the syntax is unsupported and should be updated).

HOWEVER, I'm not willing to guarantee that this syntax will continue to be supported going forward. GNU make is not the kernel, and I reject attempts to impose the development ethos of the kernel on GNU make. The syntax of makefiles is so free-form that virtually any enhancement that is made will be a backward-compatibility issue and I'm simply not willing to commit to that kind of restriction. The kernel can always just introduce a different function name with new semantics--not so for makefiles. If people need that level of backward-compatibility I recommend they simply keep around multiple versions of GNU make to support older code; make's installation is trivial, just the one program with no extra support files, and even after renaming it works flawlessly.

In this particular case I expect that even if the syntax still can be made to work now, when we support the ability to define explicit rules with multiple targets generated from a single recipe invocation, this syntax might not survive. The fact is that this usage IS illegal according to the documentation (it may not be explicitly proscribed but writing down all the things that are NOT legal is an impossibility--it's easily, IMO, inferrable that the syntax is not intended to be valid).

I am not willing, at this point, to embrace the idea of "fixing" it so this syntax works fully and making it legal. In my opinion it will be simply too confusing and difficult to manage all the edge cases around trying to combine two (or more, in the future) different rule models into a single statement. I'm willing to look at any proposals someone has, but I warn you I'll be extremely skeptical. I just don't think it's that big of a deal to write the rules separately, so any proposal will have to be VERY compelling to justify the complexity given the trivial alternative.

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Sat 12 Oct 2013 08:35:07 AM UTC, comment #15:

David Boyce wrote:

> I agree with some of your points but there was a bit of cherrypicking involved in the quoting:
>
> > ... really, what's the chance a trivial one-line change to the top-level makefile will actually break your driver?
>
> I'm not sure you realize the scale here. It's not a change to just the top-level makefile, the pattern is repeated in a number of makefiles.


I was just going with was was said in the bug report. The "workaround" in comment #2 is indeed a trivial one-line change. If that's not all, this wasn't mentioned before, sorry.

To me this seems a great example of bad bug reporting:

- The original report and workaround make it look like a trivial issue.

- The explanation for the change (quoted in comment #3) was ignored (see comment #5).

- The whole issue was ignored for 2 years, just until the next major release which is just about the worst time to reinstate old behaviour for bugward compatibility.

> A quick find shows 1360 makefiles in a typical Linux (2.6.36) kernel


Fun fact: A quick find shows over 500000 files on my hard disk!

(Which is just as relevant to the issue; according to the git history search Josh recommended, it seems to be 3, in words three, affected Makefiles, two of them in arch, so possibly not even relevant to you. Over-inflating numbers doesn't help.)

> At least in our case, because these are so large and (supposed to be) unchanging we haven't stored them in version control; they're in flat file space aka NFS. With this many files to fix it would be necessary to write a program which could reliably find all mixed rule lines and fix them right the first time. The fact that they're not version controlled makes the stakes higher.


Though I see that it can be a problem for you, I think "higher stakes" is another exaggeration. It can't be too difficult to back up the changed makefiles, even without source control.

> > I don't suppose Paul would accept a patch that just reinstates the old behaviour without fixing it.)
>
> Of course not, that's been made quite clear and it makes sense. But what you left out from my comment was the question of whether it could be optionally unfixed at runtime. We all get that the old behavior was broken, but this happens to be a case where preserving bug-for-bug compatibility is important to many users.


That's not so clear to me. It ought to be possible to really fix the problem, i.e. accept mixed rules without the bugs in the original (unintended) feature. Of course, this is more work than just suppressing a (well justified) error message. If you or someone does this properly, I suppose Paul wouldn't object.

> > I've long accepted that Debian isn't useful to me out of the box...
>
> I don't think this point was specific to Debian; rather, it was intended as an illustration of the fact that 3+ years after 3.82 was released it still hasn't gained widespread acceptance. Many environments are sticking with 3.81 and I suspect this is one of the factors.


I thought other distributions had long moved past the affected versions of Linux and all the other unnamed "numerous" affected software. (Mind you, I sometimes like Debian's conservative release schedule, but it's pretty unique.)

Frank Heckenbach <frank>
Fri 11 Oct 2013 07:23:34 PM UTC, comment #14:

Yes, I realize it's a bit of a wing and a prayer but from the git log it looks like this fix got into the same commit as a rewrite of major sections of read.c. Without reading and thinking through all of it, I figured there was a chance that adding the fatal error had been the only thing done for this particular fix.

David Boyce <boyski>
Fri 11 Oct 2013 06:34:24 PM UTC, comment #13:

You should check if "make foo.o" works, and if dependencies on either "all" or "foo.o" or both work as expected.

I'll give the patch a try with an older Linux kernel and see if it fixes the build issue. I'm somewhat skeptical that removing the fatal error alone will solve the problem without changing something in the code that actually parses such rules, but it'd be awesome if that's all it took. Worth a shot.

Josh Triplett <joshtriplett>
Fri 11 Oct 2013 03:17:36 PM UTC, comment #12:

I'm uploading a patch that adds an --allow-mixed-rules option which converts the fatal error to a warning. This is the only testing it's been subjected to:

% cat Makefile
all %.o:
@echo Making "$@"

% make-4.0
Makefile:1: *** mixed implicit and normal rules. Stop.

% make-4.0 --allow-mixed-rules
Makefile:1: warning: mixed implicit and normal rules
Making all

I don't claim to understand all the logic in read.c but from a quick examination of the git log this seemed to have a good chance of being all that's required. I'm not in a position to test it thoroughly at this time.

(file #29357)

David Boyce <boyski>
Fri 11 Oct 2013 03:08:42 PM UTC, comment #11:

Frank,

I agree with some of your points but there was a bit of cherrypicking involved in the quoting:

> ... really, what's the chance a trivial one-line change to the top-level makefile will actually break your driver?


I'm not sure you realize the scale here. It's not a change to just the top-level makefile, the pattern is repeated in a number of makefiles. A quick find shows 1360 makefiles in a typical Linux (2.6.36) kernel and as I mentioned we have "dozens or hundreds" of copies of such kernels. There are an unknown number of places where this pattern occurs and the placement and formatting may vary, possibly even with line wrapping.

At least in our case, because these are so large and (supposed to be) unchanging we haven't stored them in version control; they're in flat file space aka NFS. With this many files to fix it would be necessary to write a program which could reliably find all mixed rule lines and fix them right the first time. The fact that they're not version controlled makes the stakes higher.

Yes, all these problems are solvable but it's a substantial project and very likely way more work than patching make.

> I don't suppose Paul would accept a patch that just reinstates the old behaviour without fixing it.)


Of course not, that's been made quite clear and it makes sense. But what you left out from my comment was the question of whether it could be optionally unfixed at runtime. We all get that the old behavior was broken, but this happens to be a case where preserving bug-for-bug compatibility is important to many users.

> I've long accepted that Debian isn't useful to me out of the box...


I don't think this point was specific to Debian; rather, it was intended as an illustration of the fact that 3+ years after 3.82 was released it still hasn't gained widespread acceptance. Many environments are sticking with 3.81 and I suspect this is one of the factors.

David

David Boyce <boyski>
Fri 11 Oct 2013 03:51:39 AM UTC, comment #10:

> Yes, formally it's not, but really, what's the chance a trivial one-line change to the top-level makefile will actually break your driver? :)


It's actually several non-trivial multi-line changes to various makefiles in the Linux source tree. Look through the kernel's git history for "3.82" and you'll find several commits worth of makefile changes working around the "mixed implicit and normal rules" regression.

And in any case, having to apply any patches while digging back in the source tree is painful.

Josh Triplett <joshtriplett>
Fri 11 Oct 2013 03:37:34 AM UTC, comment #9:

> The rare times it has, notably the time that led to the introduction of the -fno-delete-null-pointer-checks option now used in the Linux kernel and other projects, have been huge debacles that generated incredible pain in numerous projects.


"Numerous projects", are you sure? OTOH, there's certainly numerous projects (including all of mine) which do benefit from improved optimization.

Honestly, when I read about the bug that led to -fno-delete-null-pointer-checks (as described in http://blog.regehr.org/archives/970), I couldn't help but BHAT. How can anyone think this code is OK in any way? Even if you didn't anticipate the security problem it became, the code just looks wrong. (In fairness, I read some speculation that the code was originally correct with lines 2 and 3 swapped, and was anti-fixed because C89 doesn't support mixed declaration order. If so, the code is still wrong, and the one who did it just made a dumb mistake.)

So I don't quite agree about the "huge debacles" part. Of course, a large code base like Linux contains a number of bugs, and some of them will be found one way or the other. This one was found by improved optimization in gcc, and unfortunately turned out a severe bug. OK, shit happens, but does this mean we should hold back any improvements to our tools?

In isolated and severe cases it may make pragmatic sense to implement specific workarounds such as -fno-delete-null-pointer-checks. But doing this for every change that potentially affects any broken code out there would be maintenance hell as Paul noted. I don't think this problem is severe enough to justify such effort since the worst thing that happens AFAIK is a compile-time error that can easily be googled by now.

> The release notes for make 3.82 don't give any indication about why make no longer accepts pattern rules and explicit targets in the same rule, just that it doesn't.


As quoted in comment #3: "However, this was not documented as acceptable and if you listed any explicit targets AFTER the pattern targets, the entire rule would be mis-parsed." I read this as, this "feature" was undocumented and never worked correctly, and instead of keeping a half-broken feature, it was removed. Seems clear enough to me.

> It means that, for years to come, users will need a copy of make-3.81 sitting around, and have to know to use that for building older software when they see that particular error message. And this change is the reason why make 3.82 is still sitting in Debian experimental and won't go into unstable anytime soon. (And all the amazing features of make 4.0 seem likely to sit in experimental for a long time for the same reason.)


That would be really sad. Why can't they keep two different versions in stable, as they do with gcc, say "make" (new version) for normal users, and "make-3.81" for bugward compatibility?

Then again, I've long accepted that Debian isn't useful to me out of the box (a few packages, I've reported bugs with reproducible tests and patches, and they've just been ignored since several major Debian releases, which is quite a while), so I'll have to manually fix them anyway. So if it turns out as you say, I'll just add GNU make to my list of "broken-in-Debian" packages ...

> Re comment #7: Please do, that would be highly welcome.


You do realize that comment is almost 2 years old? Not sure if Jonathan is still going to do it if he hasn't yet. (It may be more complex than he thought, since it was broken before, see above, and I don't suppose Paul would accept a patch that just reinstates the old behaviour without fixing it.)

David Boyce wrote in #6:

> Of course we also have the option of going back and "fixing" the makefiles for each old kernel, but once you've changed 2.6.18-128.7.1 then it's no longer 2.6.18-128.7.1.


Yes, formally it's not, but really, what's the chance a trivial one-line change to the top-level makefile will actually break your driver? :)

Frank Heckenbach <frank>
Wed 09 Oct 2013 01:42:56 PM UTC, comment #8:

Re comment #5:

GCC is exceptionally careful about introducing backward-incompatible changes that break existing codebases, to the point where it provides options like -fpermissive to help older code continue to build. As long as you don't blindly use options like -Werror, GCC generally doesn't break older codebases, for exactly this reason. The rare times it has, notably the time that led to the introduction of the -fno-delete-null-pointer-checks option now used in the Linux kernel and other projects, have been huge debacles that generated incredible pain in numerous projects.

On top of that, GCC's rare instances of making parsing more strict have generally either been for standards compliance or to introduce some new feature or optimization that depended on the stricter behavior. The release notes for make 3.82 don't give any indication about why make no longer accepts pattern rules and explicit targets in the same rule, just that it doesn't. What motivated this change, and what amazing enhancement to GNU make does it allow that would not be possible while retaining that behavior?

This backward-incompatibile change makes it impossible to build older versions of the Linux kernel with current make, which breaks features like "git bisect" (for tracking down a bug in a range of kernel versions down to a specific commit). It means that, for years to come, users will need a copy of make-3.81 sitting around, and have to know to use that for building older software when they see that particular error message. And this change is the reason why make 3.82 is still sitting in Debian experimental and won't go into unstable anytime soon. (And all the amazing features of make 4.0 seem likely to sit in experimental for a long time for the same reason.)

Given all that, is there any fundamental reason why make couldn't support this behavior for backward compatibility, either by default, or at least with an option to turn on that backward compatibility? In particular, would you consider merging a patch to that effect if someone else wrote it?

Re comment #7: Please do, that would be highly welcome.

Josh Triplett <joshtriplett>
Fri 04 Nov 2011 03:54:17 AM UTC, comment #7:

Just for the record, I have been looking for a spare moment to come up with a patch to fix this compatibility problem (which affects many projects other than the Linux kernel, too).

If that's a bad idea for some reason, please feel free to let me know why. :) And if anyone else starts before I do, I won't mind --- on the contrary, I'll be very happy.

Jonathan Nieder <jrn>
Fri 20 May 2011 07:46:25 PM UTC, comment #6:

In fairness, this is a special situation. I have the same problem as "tz", and I expect quite a few other people do too.

My company makes drivers. The corollary is that we keep dozens if not hundreds of entire Linux kernel source build trees going back 10 years or so in order to build drivers for them. The way Linux kernel drivers are built makes it necessary to use the kernel build system. Thus it's not sufficient for us to fix our own makefiles, or even that Linux kernels which came out in the last year are fixed themselves. As things stand, we'll be unable to upgrade from GNU make 3.81 until all pre-2010 kernels have drained out of our support matrix, which could easily be another 10 years. Of course we also have the option of going back and "fixing" the makefiles for each old kernel, but once you've changed 2.6.18-128.7.1 then it's no longer 2.6.18-128.7.1.

So I at least think this issue is different from other incompatibilities and worthy of special consideration. If the original fix was a complete rewrite of parsing then maybe that's the end of the story, but if it's just a little patch which could be switched at runtime, would it be possible to add a special target to control that switch? I might sign up for the work if you agree it's not an outrageous thing to support.

David Boyce <boyski>
Fri 20 May 2011 06:10:11 PM UTC, comment #5:

I'm not trying to be flip, but what do you do when you upgrade to a new version of the compiler and older code no longer builds correctly due to more stringent requirements? I see this in code all the time (not GNU make) with new versions of GCC for example.

I really am not interested in changing all the error messages to give details about which version of GNU make that error was introduced, and I'm also not interested in adding flags to every version of GNU make which allows for backward-compatibility with older versions. That seems completely unwieldy from a development and maintenance point of view.

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Fri 20 May 2011 04:23:09 PM UTC, comment #4:

It would help if for the next version there was a backward compatibility switch - there are a lot of archived kernel source trees (and probably a lot of other projects) that this breaks.

It would also help if the error message noted it was a change (mixed ... no longer allowed as of 3.82), because it is very confusing to do a bunch of updates and find the old builds no longer work.

tz <tz1>
Sat 09 Apr 2011 02:06:05 PM UTC, comment #3:

This is an intentional change. See the GNU make NEWS file:

You must split these rules into two rules: one for the pattern and one for the explicit targets. The Linux kernel source has already been modified in this way (in newer kernels).

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Sat 09 Apr 2011 11:51:25 AM UTC, comment #2:

As workaround for coLinux can change the template scripts/mkmakefile.

      • old:
      • new:

See also the attached patch.

(file #23162)

Henry Nestler <henryn>
Sat 09 Apr 2011 10:48:32 AM UTC, comment #1:

Sorry. forgotten to login while creating this bug.

The bug was original reported by a coLinux user:
http://article.gmane.org/gmane.linux.colinux.general/5230

Make 3.81 works.

Henry Nestler <henryn>
Sat 09 Apr 2011 10:42:24 AM UTC, original submission:

A build of Linux kernel out of source got this error, if target name was give on command line. It gots this error:

Makefile:23: *** mixed implicit and normal rules. Stop.

The relevant lines in Makefile:
16: all := $(filter-out all Makefile,$(MAKECMDGOALS))
23: $(all) %/: all

Full example:
cd /tmp
wget http://www.kernel.org/pub/linux/kernel/v2.6/longterm/v2.6.33/linux-2.6.33.9.tar.bz2
tar xjf linux-2.6.33.9.tar.bz2
mkdir linux-2.6.33.9-build
cd linux-2.6.33.9
make O=../linux-2.6.33.9-build defconfig
cd ../linux-2.6.33.9-build
make vmlinux

If replacing the command "make vmlinux" with "make", then it works. But it is no workaround, because I need to call "make modules_install" later.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #29365:  smaller-allow.patch added by mdorey (467B - text/x-patch - boyski's patch without the option)
file #29357:  allow.patch added by boyski (2kB - application/octet-stream)
file #23162:  make-3.82-mkmakefile.patch added by henryn (281B - text/x-patch - Change kernel Makefile template)
file #23155:  Makefile added by None (523B - application/octet-stream - Linux kernel Makefile from make defconfig in build directory)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by mdorey (Updated the item)
  • -unavailable- added by frank (Posted a comment)
  • -unavailable- added by joshtriplett (Posted a comment)
  • -unavailable- added by jrn (Posted a comment)
  • -unavailable- added by boyski (Posted a comment)
  • -unavailable- added by tz1 (Posted a comment)
  • -unavailable- added by psmith (Posted a comment)
  • -unavailable- added by henryn (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 9 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sun 20 Oct 2013 05:09:53 PM UTCpsmithStatusNot A Bug=>Fixed
      Assigned toNone=>psmith
      Fixed ReleaseNone=>SCM
    Sun 13 Oct 2013 01:59:50 AM UTCmdoreyAttached File-=>Added smaller-allow.patch, #29365
    Fri 11 Oct 2013 03:17:36 PM UTCboyskiAttached File-=>Added allow.patch, #29357
    Sat 09 Apr 2011 02:06:05 PM UTCpsmithStatusNone=>Not A Bug
      Open/ClosedOpen=>Closed
    Sat 09 Apr 2011 11:51:25 AM UTChenrynAttached File-=>Added make-3.82-mkmakefile.patch, #23162
    Sat 09 Apr 2011 10:42:25 AM UTCNoneAttached File-=>Added Makefile, #23155

    Back to the top


    Powered by Savane 3.1-cleanup