bugmake - Bugs: bug #15919, Make-3.81 rc1 hangs with -j 2 but...

 
 

bug #15919: Make-3.81 rc1 hangs with -j 2 but not with -j 1

Submitter:  None
Submitted:  Mon 27 Feb 2006 06:38:39 PM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  4.0 Operating System:  Any
Fixed Release:  3.82 Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 06 Jun 2009 11:24:50 PM UTC, comment #30: 

I debugged this and applied the change Knut suggested; after this glibc with make -j10 builds without error.

Paul D. Smith <psmith>
Group administrator
Fri 05 Jun 2009 02:24:06 PM UTC, comment #29: 

I took a copy of glibc 2.9 and tried building it with -j10 on my uniprocessor and on my dual core (actually single hyperthreading CPU) and it worked fine.

But when I took it to a 4-way hyperthreading (8 core) system and ran with -j10 I did see some bad behavior: not hangs though.  Instead it seemed the same recipe was being invoked >1 time which eventually caused the build system to fail, because at the end of the recipe there's a job that removes a file.  If the multiple recipes run in the "right" way then one of the instances removes that file before the next one gets a chance to use it.

I'll look at this.

Paul D. Smith <psmith>
Group administrator
Thu 04 Jun 2009 09:09:29 PM UTC, comment #28: 

I've been running the unit tests for months and never had a failure in targets/SECONDARY.  I still don't, using the latest downloaded code (there is an odd issue with variable/SHELL that I don't see at home but do see here as a result of my checkins over the last week; I think the test is wrong bug I don't understand why I didn't see this at home).

I tried to reproduce the INTERMEDIATE file failure by editing the INTERMEDIATE test file and adding "-j2" and "-j5" to the "run_make_with_options" invocations for test 3, 4, and 5 and I got correct behavior there as well; no errors reported.

I wonder if this is somehow behaving differently on Linux?

Paul D. Smith <psmith>
Group administrator
Thu 04 Jun 2009 05:31:58 PM UTC, comment #27: 

Any word on a new release incorporating this fix?
It has been over a year since this bug prevented GCC
from implementing automatic dependency tracking:
http://gcc.gnu.org/ml/gcc-patches/2008-03/msg01680.html

Tom Tromey <tromey>
Fri 19 Dec 2008 07:33:06 AM UTC, comment #26: 

Knut St Osmundsen,
Your fix is similar to the one I mentioned in comment 18.

Yes, the idea is that tests pass cleanly. I just assumed that Paul tested his fix against the test cases. I can also reproduce the problems with the INTERMEDIATE tests if run with -j.

Paul, How do we get you to reopen this bug as I still am not a member of this project after 2.5 years of asking?



Icarus Sparry <icarus>
Wed 29 Oct 2008 03:15:47 AM UTC, comment #25: 

Is Test #9  of targets/SECONDARY supposed to run cleanly?   I'm seeing a duplicate command here (cp 2.a 2.b). I've tried current CVS head and CVS dated 2007-08-20, neither succeed, so I'm suspecting the 15th of August remake.c change.

I see a very similar problem with targets/INTERMEDIATE Test #3 and #5 when running them with -j N (where N > 1); both tests gets the 'cp bar.f bar.e' command executed twice. (Same code bases ofc.)  These two tests ran cleanly on a patched 2007-05-26 CVS snapshot that I've been using for more than a year now.

The tests succeeds again if I add if (file->command_state != cs_running) in front of the set_command_state (file, cs_not_started) call on line 990 in remake.c. I cannot say if this is the right thing or not - it's been a year or two since I last look at the update process and this is a feature I don't rarely use.  (Btw. observed on Mac OS X.)

Let me know if this is a genuine problem and if the fix needs more work.

Knut St. Osmundsen <bird_tori>
Thu 27 Mar 2008 06:31:25 PM UTC, comment #24: 

This bug affects a patch for GCC that would provide automated makefile dependencies.  It causes a lockup for "make -j2" and we had to revert the patch.  Getting this patch in a stable release would be a very good thing...

Anonymous
Wed 15 Aug 2007 05:40:38 PM UTC, comment #23: 

Glad its fixed. Now all I need do is work on you to get a quick release out the door.

I no longer have access to the codebase that provoked the original bug report, the company has gone "dot bust".

Icarus Sparry <icarus>
Wed 15 Aug 2007 01:56:18 PM UTC, comment #22: 

OK, I did a bunch more testing and created test cases to cover all the aspects of this bug reported here and in 3330.  Fix committed.  Thanks to both of you for your work on this issue.

Paul D. Smith <psmith>
Group administrator
Tue 31 Jul 2007 09:40:50 PM UTC, comment #21: 

(comment by J. David Bryan, original submitter)

I have built separate versions of make with Paul's patch from Comment #20 and Icarus' patch from Comment #18.  Both fix the problem in the original bug submission.  However, Paul's patch appears to fail with this sequence of commands (using the original makefile):

$ touch test.0 && make-p
touch test.1
touch test.2
touch test.3
touch test.int
touch test.4
rm test.int

[the above is correct]

$ touch test.0 && make-p
touch test.1
touch test.2
touch test.int
touch test.4
rm test.int

[the above is correct -- test.3 is not remade]

$ touch test.0 test.3 && make-p
touch test.int
touch test.4
rm test.int

[the above is incorrect -- test.1 and test.2 are not remade]

If I repeat the last command using Icarus' patched version, it works as expected:

$ touch test.0 test.3 && make-i
touch test.1
touch test.2
touch test.int
touch test.4
rm test.int

[the above is correct]

If I run the last command with Paul's patched version, I note the following lines in the debug output log produced with the "-d" option:

    Considering target file `test.2'.
    File `test.2' was considered already.

...except that test.2 wasn't previously considered, according to the log.

Anonymous
Tue 10 Jul 2007 02:16:44 PM UTC, comment #20: 

OK, I went through both this bug and bug 3330 last night, and I do see the problem; thanks for all your work and the patch you provided Icarus.

However, I'm not entirely sure that the way you solved this problem is the best one.  Setting the state of the intermediate file to cs_not_started seems incorrect to me; it does seem to solve the problem, at least for the cases described in these bug reports, but it means that after we're done the intermediate file is marked as "not started" which I'm not sure is correct.

Also, I do see J. David's issue with a pattern rule invoked twice on my vanilla CVS head tree with the original patch applied.

I have an alternative change that feels more correct: after all the prereqs of the intermediate file are done, if we don't need to make the intermediate file then we call notice_finished_file() on it to complete it.  With this patch, J. David's example works correctly as do the other examples in both this bug and bug 3330.

I attach the code change only here: obviously the real fix will involve change logs, test cases, etc.  I think your idea of having a timeout is a good one, Icarus, but I think I will make it general so that EVERY test we run has a timeout; these can be modified on a per test basis of course.

Please let me know whether this patch solves all the issues you have in real life and any other comments you have.

(file #13283)

Paul D. Smith <psmith>
Group administrator
Tue 10 Jul 2007 07:20:18 AM UTC, comment #19: 

I said all bugs would be reexamined before a release: I didn't say they would all be fixed before a release.  Bug 3330 was looked at (by me) before 3.81 was released, but I thought it was bound up with the other issues surrounding .SECONDARY and I elected not to spend time trying to fix it then.  It appears that I didn't look at it closely enough; if I'd been more diligent I would have seen that it was something else.  That was an error in judgment on my part.

I'm sorry but I don't consider this bug critical enough to make a new release for it alone.  This issue has been in the code since at least GNU make 3.75 (that's the farthest back that I tested it) which is over 10 years old, and in all that time as far as I can tell from searching the mailing lists, etc. only yourself and J. David Bryan have run into it.  That's not to say it's not a bad bug: it is, and it will be fixed in 3.82.  But as for an immediate bug fix release, I don't think it's justified in this case.  There are plenty of instances where programs have bugs that crash or return incorrect results on valid input, and an immediate release is not created for each of those.  The question is, how many people will the problem affect and what is the impact of the bug (security flaw, erases a disk, etc.)?  This bug simply doesn't rise to the level needed to justify an emergency release to fix it, in my opinion.

Paul D. Smith <psmith>
Group administrator
Mon 09 Jul 2007 04:17:49 AM UTC, comment #18: 

Sorry J David Bryan, as you say there is no obvious way to edit comments once they have been submitted. I am just very frustrated with the current lack of progress with GNU make, and you caught me on a bad day.

Looking over my current sources I see I have updated my sources since my patch. If you email me directly (savannah at icarus freeuk com)I will send you an updated patch which is actually the fix to your problem, it is not the fix to 16051.

Paul, it would be nice to believe you. However you released 3.81 without testing bug 3330. If you had done this testing you would have found this bug before releasing 3.81.

Personally I consider this bug to be a "release critical" bug, by which I mean it has to be fixed before a release is made, and just fixing this bug is a reason for creating a new release. It takes a perfectly valid input, and causes make to enter an infinite loop. If "cat" had a bug so that if it was given a filename with "dog" in it it went into an infinite loop then people would expect a fix faster than a year, let alone 4 years.

Do you agree Paul?

Icarus Sparry <icarus>
Fri 06 Jul 2007 03:29:56 PM UTC, comment #17: 

(comment by J. David Bryan, original submitter)

> ...they can not even spell my name correctly!


My apologies.  Of course, I noted the misspelling as soon as I had posted comment 14, but there seemed to be no way to edit a bug post after submission.  I thank you for the original patch and the followup, and I'm sorry that the typo caused you such distress.

Anonymous
Fri 06 Jul 2007 02:32:52 PM UTC, comment #16: 

I hear your frustration but I cannot apply patches to the source code that I do not fully grok, even if it is a one-line patch.  By this I don't mean just "what does the one line do", which is easy, but rather why is that line necessary, why is it only necessary when running -jN but not serially, what is special about intermediate files in this context, etc.  It's always possible that the fix you've provided ISN'T the best way to resolve the bug.

You were explaining the issue back in November after I asked for this; however, at the end of your last comment you wrote "More later..." so I filed this bug waiting for "more", and then it got lost in the shuffle.  Certainly before any release candidates are created (if not before) I go through all the open bugs; this one would have been re-examined then if not sooner.  If you've said everything you have to say about this I'll look at it again this weekend.

Paul D. Smith <psmith>
Group administrator
Fri 06 Jul 2007 12:44:40 AM UTC, comment #15: 

If I take the current (July 4th 2007) sources of make from CVS (Paul has just updated the license to GPL 3), and apply my patch I get the following output

$ make -j2
cp 1.a 1.b
cp 2.a 2.b
cp 1.b 1.c
cp 2.b 2.c
rm 1.b 2.b

Note that there have been 7 changes to remake.c since 3.81 was released. In particular there has been a fix for bug 16051, which may affect the result.

It is hard to get enthusiastic about fixing bugs, or even reporting them when the current maintainer sits for over a year on a request to be added to the project. Not accepting, not rejecting, just ignoring the request.

Then one attempts to help other people, and they can not even spell my name correctly!

Icarus Sparry <icarus>
Wed 04 Jul 2007 07:53:09 PM UTC, comment #14: 

(comment by J. David Bryan, original submitter)

The patch from Icarus Sperry does fix the original problem.  However, it appears to introduce a new one.

I have taken the original make-3.81 sources and added the patch to produce "make-381p".  Then, with this makefile:

%.c : %.b
cp $< $@

%.b : %.a
cp $< $@

all : 1.c 2.c

...executing this command in an otherwise empty directory:

$ touch 1.a 2.a && make-381p -j 2

...produces this output:

cp 1.a 1.b
cp 2.a 2.b
cp 1.b 1.c
cp 2.a 2.b  [ <-- NOTE ]
cp 2.b 2.c
rm 1.b 2.b

...whereas the stock make-3.81 with the same command produces:

cp 1.a 1.b
cp 2.a 2.b
cp 1.b 1.c
cp 2.b 2.c
rm 1.b 2.b

Note that the patched version copies 2.a to 2.b twice.  Running the patched version with "-j 1" produces the correct behavior.

I've confirmed this on sparc-sun-solaris2.9 as well as the x86 Win32 version.

Anonymous
Tue 14 Nov 2006 04:53:41 PM UTC, comment #13: 

How much explanation do you want for a 1 line patch, which just initialises a variable?

The way that make builds when it is in parallel make mode is different from the way that it builds when it is building sequentially. The bug manifests itself for any number of jobs greater than 1, there is nothing special about 2, and my test case sets it to infinite.

The guts of the GNU make engine is two mutually recursive routines, check_dep and update_file. (update_file_1 is just called from one place, and so could be usefully declared "inline", or make could be compiled with -finline-functions-called-once)

The underdocumented requirement for check_dep is to build an non-intermediate files that this file depends on, and set *must_make_pointer to be 1 if this needs to be made. There is an undocumented requirement when jobs are run in parrallel for the state not to be left in cs_deps_running if all of the dependencies have completed. The is what my fix does.

More later.....

Icarus Sparry <icarus>
Sun 12 Nov 2006 10:07:12 PM UTC, comment #12: 

Thanks Icarus; that's great.  Especially new testcases: that always helps.  One thing: in general I like to have the ChangeLog entry describe what the change does and how it fixes things, rather than just describing the symptoms of the bug.  Do you have any comments on how this change fixes the problem, or why it manifests during -j2 in particular?  Thanks!

Paul D. Smith <psmith>
Group administrator
Sun 12 Nov 2006 07:30:33 AM UTC, comment #11: 

This bug appears to be the latest incarnation of bug #3330. When that was submitted make didn't go into an infinite loop, but the makefile that was listed then caues make 3.81 to go into a loop.

Attached is a patch which fixes the bug, and adds a new test case to ensure there is no regression.

Icarus Sparry <icarus>
Tue 31 Oct 2006 02:07:55 AM UTC, comment #10: 

Hi Icarus.  The easiest way to get a patch into GNU make is to attach it to the bug report and/or create a separate patch item (the first is preferred).  We'll review it and apply it as-is or else discuss it with you if necessary.

Also, if the patch involves significant amounts of new code you will need to assign copyright to the FSF, and your employer will also need to do so if you're employed for programming at all.  If you're concerned about this contact me to discuss it.

Thanks for making the effort to fix this!

Paul D. Smith <psmith>
Group administrator
Mon 30 Oct 2006 05:45:39 PM UTC, comment #9: 

I have a fix, which works with the two Makefiles that are attached to this bug, plus my stripped down makefile, plus the real life makefile that it was stripped from.

It also passes the "make check" stage.

It also fixes another old bug.

Paul, how do I get commit access to CVS?


Icarus Sparry <icarus>
Fri 27 Oct 2006 01:14:45 AM UTC, comment #8: 

Line numbers are relative to remake.c, version 1.130.
The problem appears to be in check_dep. Stepping through the second and later times it traverses the graph, in the case where "file" has the record for "intermed".

It increases depth, and marks the structure as updating.
The if on line 941 is false, this is an intermediate file.
So take the else on 953. The if on 958 is false, we have cmds for this file. The if on line 966 is false, as we have cmds still!

At line 976 mtime is 1, NONEXISTENT_MTIME, so the if on 978 is false, so we take the else part, lines 983 to 1032. This is a loop over the dependencies, in this case there are 2, "src" and "phony".

Stepping through for "src"
The if on 993 is false, "src" is not being updated.
line 1013, maybe_make = 0
The call to check_dep (line 1014) returns 0, and leaves maybe_make unchanged.
The if on 1016 is true, so *must_make_ptr = 0 (unchanged)
The if on 1019 is false (dep_status==0)
The if on lines 102[23] is false, the command status is cs_finished.

Stepping through for "phony"
The if on 993 is false, "phony" is not being updated.
line 1013, maybe_make = 0
The call to check_dep (line 1014 returns 0, but changes maybe_make to 1.
The if on 1016 is false, this is an order only dependency,
so *must_make_ptr is unchanged.
The if on 1019 is false (dep_status == 0)
The if on lines 102[23] is false, the command_status is cs_finished.

It then unmarks the "intermed" file as being updated, and returns.

However this leaves a tree

                    + src
target -- intermed -|
                    + phony

where src & phony are in state "cs_finished", target and intermed are in state cs_deps_running, and nothing is going to change this.

The "maybe_make" local variable in the update_file_1("target",1) (to abuse notation) is left as zero, but its dependency ("intermed") is still in cs_deps_running state, so it is not updated.

Icarus Sparry <icarus>
Thu 26 Oct 2006 04:17:03 AM UTC, comment #7: 

Here is the Makefile, stripped as much as I think I can. In particular if I change the intermed dependency on phony to be a normal one, rather than an order only one then everything works as I would expect. Run "make clean" (or touch src) to get things started.

The version that Make reports is 3.81.90, and was pulled from CVS yesterday (24th Oct 2006). The tabs have been replaced by spaces, so hopefully it will look reasonable,

# Run with make -Rj

target: intermed
        touch $@

.INTERMEDIATE: intermed
intermed: src | phony
        touch $@

.PHONY: phony
phony:
        : Running Order only phony target

clean:
        rm -fr target
        test -e src || touch src

Icarus Sparry <icarus>
Thu 26 Oct 2006 12:33:14 AM UTC, comment #6: 

I have another makefile which shows the same effect.
I am trimming it at the moment, but things it has in common with the second makefile below is an "order only" dependency and a ".INTERMEDIATE". make -j1 works fine. make -j2 works fine the first time. make -j2 burns infinite amounts of CPU the second time, looping, thinking that things are already being made.

This is the currect CVS version of Make.

Icarus Sparry <icarus>
Wed 01 Mar 2006 08:26:06 PM UTC, comment #5: 

(comment by J. David Bryan, original submitter)

One additional characteristic: aborting make after it's hung, e.g., with CTRL+C, and then re-running "make -j 2" will complete the expected set of actions.  That is:

$ touch test.0 && make -j 2
touch test.1
touch test.2
[ make hangs here ]
[ now kill it with CTRL+C ]

$ make -j 2
touch test.int
touch test.4
rm test.int

$

So whatever conditions pertain that cause the hang do not recur when make is restarted.

Anonymous
Tue 28 Feb 2006 08:13:55 AM UTC, comment #4: 

I tried to minimize the makefile but couldn't get any rules out without removing buggy behavior. I could get .SECONDARY out of the way, however. The first line in the makefile:

.SECONDARY: test.1 test.2 test.3

Can be replaced with

.INTERMEDIATE: test.3
.PRECIOUS: test.3

If I comment .INTERMEDIATE out, the behavior goes away.

Boris Kolpackov <bosk>
Group Member
Tue 28 Feb 2006 06:55:07 AM UTC, comment #3: 

After some more testing, it appears that removing test.3 from .SECONDARY prevents make from hanging, not test.2.

Boris Kolpackov <bosk>
Group Member
Mon 27 Feb 2006 07:38:49 PM UTC, comment #2: 

Not offhand.  I haven't looked at it carefully enough.  I did spend about 10 mins: just enough to verify that I can reproduce it.  I also enabled debugging and I can see that it's looping forever, not just hanging.  Make seems to think that it's already started the rules to build the prerequisites, but they never complete so it continues to loop trying to build them.  It was a few days ago so I don't remember the details.  I'm super-swamped with "real work" right now but my plan was to set a breakpoint at the start of the first entry to build "test", then follow along and see how it gets into this state.  Since you've discovered that removing .SECONDARY will fix the problem, debugging the two alongside one another (one debug session with a makefile with .SECONDARY and one without) will also be helpful, I expect.  If I had to guess I'd say it's a bizarre interaction between .SECONDARY and order-only prerequisites... I guess that much is obvious given what you've discovered.  The bug does still happen if you use "-rR" options to make, which will significantly cut down on the amount of irrelevant work make will have to do (and the number of loops you'll have to walk through).

Paul D. Smith <psmith>
Group administrator
Mon 27 Feb 2006 07:27:08 PM UTC, comment #1: 


> This one looks like it's going to be a big PITA to fix.


Do you have any thoughts on why it hangs when we use the implicit rule and does not when we use a normal one. After instantiation of the implicit rule, the two cases should be equivalent.

Boris Kolpackov <bosk>
Group Member
Mon 27 Feb 2006 06:38:39 PM UTC, original submission:  

Running make-3.81rc1 with the attached makefile will hang if "-j 2" is specified, but will run properly if "-j 1" is selected instead.  To duplicate, put the attached makefile into a blank directory.  Running with "touch test.0 && make -j 2" runs properly the first time and produces:

touch test.1
touch test.2
touch test.3
touch test.int
touch test.4
rm test.int

Running "touch test.0 && make -j 2" again hangs repeatably part-way through:

touch test.1
touch test.2

Running instead with "touch test.0 && make -j 1" repeatably produces the same output as the first run.  Note that if "test.int" is touched externally, so that's it's pre-existing, then "-j 2" works properly.  Deleting "test.int" and retrying will hang.

I have reproduced this under both Windows and Unix, and on single and dual-processor systems.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #13283:  intermed.diff added by psmith (1KiB - text/x-patch - Proposed patch, using notice_finished_file().)
file #13245:  makefile added by None (63B - application/octet-stream - example makefile for comment #14)
file #11234:  patch added by icarus (3KiB - application/octet-stream - Fix for bugs 15919 and 3330, plus new test case)
file #1750:  makefile added by bosk (178B - application/octet-stream)
file #1745:  makefile added by None (176B - application/octet-stream - Test case makefile)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by tromey (Posted a comment)
  • -email is unavailable- added by bird_tori (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by icarus (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 15 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-07-28 psmith Fixed Release4.0 3.82
    2009-06-06 psmith StatusNone Fixed
        Open/ClosedOpen Closed
    2009-06-04 psmith StatusFixed None
        Open/ClosedClosed Open
    2007-08-15 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.0
    2007-07-10 psmith Attached File- Added intermed.diff, #13283
    2007-07-04 None Attached File- Added makefile, #13245
    2006-11-12 icarus Attached File- Added patch, #11234
        Carbon-Copy- Added icarus
    2006-02-28 bosk Attached File- Added makefile, #3449
    2006-02-27 None Attached File- Added makefile, #3444

    Back to the top

    Powered by Savane 3.14-f13d.
    Corresponding source code