I've been running the unit tests for months and never had a failure in targets/SECONDARY.  I still don't, using the latest downloaded code (there is an odd issue with variable/SHELL that I don't see at home but do see here as a result of my checkins over the last week; I think the test is wrong bug I don't understand why I didn't see this at home).

I tried to reproduce the INTERMEDIATE file failure by editing the INTERMEDIATE test file and adding "-j2" and "-j5" to the "run_make_with_options" invocations for test 3, 4, and 5 and I got correct behavior there as well; no errors reported.

I wonder if this is somehow behaving differently on Linux?

Knut St Osmundsen,
Your fix is similar to the one I mentioned in comment 18.

Yes, the idea is that tests pass cleanly. I just assumed that Paul tested his fix against the test cases. I can also reproduce the problems with the INTERMEDIATE tests if run with -j.

Paul, How do we get you to reopen this bug as I still am not a member of this project after 2.5 years of asking?

This bug affects a patch for GCC that would provide automated makefile dependencies.  It causes a lockup for "make -j2" and we had to revert the patch.  Getting this patch in a stable release would be a very good thing...

OK, I did a bunch more testing and created test cases to cover all the aspects of this bug reported here and in 3330.  Fix committed.  Thanks to both of you for your work on this issue.

OK, I went through both this bug and bug 3330 last night, and I do see the problem; thanks for all your work and the patch you provided Icarus.

However, I'm not entirely sure that the way you solved this problem is the best one.  Setting the state of the intermediate file to cs_not_started seems incorrect to me; it does seem to solve the problem, at least for the cases described in these bug reports, but it means that after we're done the intermediate file is marked as "not started" which I'm not sure is correct.

Also, I do see J. David's issue with a pattern rule invoked twice on my vanilla CVS head tree with the original patch applied.

I have an alternative change that feels more correct: after all the prereqs of the intermediate file are done, if we don't need to make the intermediate file then we call notice_finished_file() on it to complete it.  With this patch, J. David's example works correctly as do the other examples in both this bug and bug 3330.

I attach the code change only here: obviously the real fix will involve change logs, test cases, etc.  I think your idea of having a timeout is a good one, Icarus, but I think I will make it general so that EVERY test we run has a timeout; these can be modified on a per test basis of course.

Please let me know whether this patch solves all the issues you have in real life and any other comments you have.

(file #13283)

Sorry J David Bryan, as you say there is no obvious way to edit comments once they have been submitted. I am just very frustrated with the current lack of progress with GNU make, and you caught me on a bad day.

Looking over my current sources I see I have updated my sources since my patch. If you email me directly (savannah at icarus freeuk com)I will send you an updated patch which is actually the fix to your problem, it is not the fix to 16051.

Paul, it would be nice to believe you. However you released 3.81 without testing bug 3330. If you had done this testing you would have found this bug before releasing 3.81.

Personally I consider this bug to be a "release critical" bug, by which I mean it has to be fixed before a release is made, and just fixing this bug is a reason for creating a new release. It takes a perfectly valid input, and causes make to enter an infinite loop. If "cat" had a bug so that if it was given a filename with "dog" in it it went into an infinite loop then people would expect a fix faster than a year, let alone 4 years.

Do you agree Paul?

I hear your frustration but I cannot apply patches to the source code that I do not fully grok, even if it is a one-line patch.  By this I don't mean just "what does the one line do", which is easy, but rather why is that line necessary, why is it only necessary when running -jN but not serially, what is special about intermediate files in this context, etc.  It's always possible that the fix you've provided ISN'T the best way to resolve the bug.

You were explaining the issue back in November after I asked for this; however, at the end of your last comment you wrote "More later..." so I filed this bug waiting for "more", and then it got lost in the shuffle.  Certainly before any release candidates are created (if not before) I go through all the open bugs; this one would have been re-examined then if not sooner.  If you've said everything you have to say about this I'll look at it again this weekend.

(comment by J. David Bryan, original submitter)

The patch from Icarus Sperry does fix the original problem.  However, it appears to introduce a new one.

I have taken the original make-3.81 sources and added the patch to produce "make-381p".  Then, with this makefile:

%.c : %.b
cp $< $@

%.b : %.a
cp $< $@

all : 1.c 2.c

...executing this command in an otherwise empty directory:

$ touch 1.a 2.a && make-381p -j 2

...produces this output:

cp 1.a 1.b
cp 2.a 2.b
cp 1.b 1.c
cp 2.a 2.b  [ <-- NOTE ]
cp 2.b 2.c
rm 1.b 2.b

...whereas the stock make-3.81 with the same command produces:

cp 1.a 1.b
cp 2.a 2.b
cp 1.b 1.c
cp 2.b 2.c
rm 1.b 2.b

Note that the patched version copies 2.a to 2.b twice.  Running the patched version with "-j 1" produces the correct behavior.

I've confirmed this on sparc-sun-solaris2.9 as well as the x86 Win32 version.

Thanks Icarus; that's great.  Especially new testcases: that always helps.  One thing: in general I like to have the ChangeLog entry describe what the change does and how it fixes things, rather than just describing the symptoms of the bug.  Do you have any comments on how this change fixes the problem, or why it manifests during -j2 in particular?  Thanks!

Hi Icarus.  The easiest way to get a patch into GNU make is to attach it to the bug report and/or create a separate patch item (the first is preferred).  We'll review it and apply it as-is or else discuss it with you if necessary.

Also, if the patch involves significant amounts of new code you will need to assign copyright to the FSF, and your employer will also need to do so if you're employed for programming at all.  If you're concerned about this contact me to discuss it.

Thanks for making the effort to fix this!

Line numbers are relative to remake.c, version 1.130.
The problem appears to be in check_dep. Stepping through the second and later times it traverses the graph, in the case where "file" has the record for "intermed".

It increases depth, and marks the structure as updating.
The if on line 941 is false, this is an intermediate file.
So take the else on 953. The if on 958 is false, we have cmds for this file. The if on line 966 is false, as we have cmds still!

At line 976 mtime is 1, NONEXISTENT_MTIME, so the if on 978 is false, so we take the else part, lines 983 to 1032. This is a loop over the dependencies, in this case there are 2, "src" and "phony".

Stepping through for "src"
The if on 993 is false, "src" is not being updated.
line 1013, maybe_make = 0
The call to check_dep (line 1014) returns 0, and leaves maybe_make unchanged.
The if on 1016 is true, so *must_make_ptr = 0 (unchanged)
The if on 1019 is false (dep_status==0)
The if on lines 102[23] is false, the command status is cs_finished.

Stepping through for "phony"
The if on 993 is false, "phony" is not being updated.
line 1013, maybe_make = 0
The call to check_dep (line 1014 returns 0, but changes maybe_make to 1.
The if on 1016 is false, this is an order only dependency,
so *must_make_ptr is unchanged.
The if on 1019 is false (dep_status == 0)
The if on lines 102[23] is false, the command_status is cs_finished.

It then unmarks the "intermed" file as being updated, and returns.

However this leaves a tree

                    + src
target -- intermed -|
                    + phony

where src & phony are in state "cs_finished", target and intermed are in state cs_deps_running, and nothing is going to change this.

The "maybe_make" local variable in the update_file_1("target",1) (to abuse notation) is left as zero, but its dependency ("intermed") is still in cs_deps_running state, so it is not updated.

I have another makefile which shows the same effect.
I am trimming it at the moment, but things it has in common with the second makefile below is an "order only" dependency and a ".INTERMEDIATE". make -j1 works fine. make -j2 works fine the first time. make -j2 burns infinite amounts of CPU the second time, looping, thinking that things are already being made.

This is the currect CVS version of Make.

I tried to minimize the makefile but couldn't get any rules out without removing buggy behavior. I could get .SECONDARY out of the way, however. The first line in the makefile:

.SECONDARY: test.1 test.2 test.3

Can be replaced with

.INTERMEDIATE: test.3
.PRECIOUS: test.3

If I comment .INTERMEDIATE out, the behavior goes away.

Not offhand.  I haven't looked at it carefully enough.  I did spend about 10 mins: just enough to verify that I can reproduce it.  I also enabled debugging and I can see that it's looping forever, not just hanging.  Make seems to think that it's already started the rules to build the prerequisites, but they never complete so it continues to loop trying to build them.  It was a few days ago so I don't remember the details.  I'm super-swamped with "real work" right now but my plan was to set a breakpoint at the start of the first entry to build "test", then follow along and see how it gets into this state.  Since you've discovered that removing .SECONDARY will fix the problem, debugging the two alongside one another (one debug session with a makefile with .SECONDARY and one without) will also be helpful, I expect.  If I had to guess I'd say it's a bizarre interaction between .SECONDARY and order-only prerequisites... I guess that much is obvious given what you've discovered.  The bug does still happen if you use "-rR" options to make, which will significantly cut down on the amount of irrelevant work make will have to do (and the number of loops you'll have to walk through).

Running make-3.81rc1 with the attached makefile will hang if "-j 2" is specified, but will run properly if "-j 1" is selected instead.  To duplicate, put the attached makefile into a blank directory.  Running with "touch test.0 && make -j 2" runs properly the first time and produces:

touch test.1
touch test.2
touch test.3
touch test.int
touch test.4
rm test.int

Running "touch test.0 && make -j 2" again hangs repeatably part-way through:

touch test.1
touch test.2

Running instead with "touch test.0 && make -j 1" repeatably produces the same output as the first run.  Note that if "test.int" is touched externally, so that's it's pre-existing, then "-j 2" works properly.  Deleting "test.int" and retrying will hang.

I have reproduced this under both Windows and Unix, and on single and dual-processor systems.