bugmake - Bugs: bug #41781, Provide a fast fail path when a...

 
 

bug #41781: Provide a fast fail path when a target is compromized during a parallel build

Submitted by:  None
Submitted on:  Wed 05 Mar 2014 04:51:33 PM UTC  
 
Severity: 3 - NormalItem Group: Enhancement
Status: NonePrivacy: Public
Assigned to: Paul D. Smith <psmith>Open/Closed: Open
Component Version: 4.0Operating System: Any
Fixed Release: NoneTriage Status: Medium Effort

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 28 Apr 2014 02:46:58 PM UTC, comment #2:

Hi Paul. Thanks for the clarifications. I do not use recursive make, I have one makefile and one DAG (I read the recursive make considered harmful paper), but I understand that for completeness it should be covered.

Finding the source of the failure that caused all the other targets to abort is not obvious. My users were struggling with this when I was usign that patch. Perhaps a different error message for the jobs that were aborted vs. the job that caused the abortion could help sort things out in post-processing. Could it be a different exit code? I think if someone actively choses to use a fast fail option, they may be willing to accept some amount of unpleasant aspects.

"This means that make will still not kill any existing jobs that are running":
I would like to have an option that causes Make to kill all jobs in flight as soon as one of the targets has failed because it does save people time. A worst case scenario that comes to mind is the Friday afternoon frenzy when there are 20 integration builds on the CI server queue. People are trying to get new code accepted for the long weekend regression tests for a Monday morning delivery. They are eager to get home. And then Make waits for all parallel in-flight jobs to complete despite an earlier failure in one of the parallel jobs in the build at the head of the integration queue, and the admin who can kill this build from outside Make is away.

I wish I had spent more time learning GNU Make's source code!

Thanks.

Patatti <patatti>
Wed 05 Mar 2014 06:00:22 PM UTC, comment #1:

There's some confusion in the reading material you mention, and in the bug report, about how GNU make actually works.

It's not true that enabling parallel builds will magically turn on the -k (keep-going) flag and that it can't be turned off. Make with parallel builds enabled, and without -k, handles failed builds in exactly the same way that it would otherwise: it stops building things "as soon as it can".

When a parallel make instance detects a failed build then it will not start any new jobs and it will wait for all the currently-running parallel jobs to complete, then it will exit with an error code. Make doesn't try to kill any running jobs, it waits for them to finish: killing jobs can lead to corrupted builds if the recipe doesn't expect to be killed (arguably this is a bug in the recipe since the user could always use CTRL-C but nevertheless it's not expected that make will kill things).

The problem you're seeing likely happens because you are using recursive make instances. Suppose you have a makefile which runs three recursive instances of make. You start it with -j2, so you have make instance A (the top one) spawning instance B and C (the children). The third recursive make instance is not started (yet) because there are only 2 jobs slots available.

Now say that during the run of make instance B a job fails. In that case, instance B detects that a failure happened and it won't start any new jobs and it will exit with an error as soon as all currently running jobs it started are completed. When B exits with an error, instance A (the root) sees the failure and it won't start up the final recursive make at all. However, already-running make instance C has no idea that B saw an error so C keeps building all its targets as usual, then exits.

If you have a large amount of parallelism and a recursive make environment, then you can get a large number of directories building happily along, unaware that one of their siblings has detected an error.

I've been thinking about this problem (which is not really the point of the patch you mention: that patch allows some finite number, less than "all" which is what you get with -k, errors to be ignored).

I think the right solution is that when a make instance gets an error, it notifies all the other make instances about the error. I think the right way to do that is to start putting back a different token into the jobserver queue, which means "error detected". Then when other make instances get that token they'll know one of their siblings had an error and they can stop building. This means that make will still not kill any existing jobs that are running, but it won't start any new ones either, even in recursive environments.

The big open question is, if a make instance detects that some other make instance failed and so it wants to stop early, what should its exit code be? I think it must exit with an error code, even though it, itself, did not detect any error, because it didn't completely build its target; we have to signal that to the parent in case the parent is relying on that for further processing. On the other hand that means you'll get a list of error messages for all the recursive make invocations, which might be unpleasant.

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Wed 05 Mar 2014 04:51:33 PM UTC, original submission:

In continuous integration systems, fast failure without recovery is a highly desirable behavior. As such, I would like to see GNU Make completely abort fast at the first failure.

During automated parallel builds with make -j <N>, make attemps to build as much as it possibly can, and even when some targets fail, GNU Make continues. This is the exact opposite of a fast failure.

In continuous integration systems, there is no value in doing incremental builds: each new build starts from scratch. This ensures stability and reproducibility.

A patch was proposed in 2009 http://lists.gnu.org/archive/html/bug-make/2009-01/msg00035.html, I wuold like to see this integrated in GNU Make.

There is some background reading material:
http://ninjaverification.wordpress.com/tag/gnu-make/
http://lists.gnu.org/archive/html/help-make/2009-02/msg00010.html

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by patatti (Posted a comment)
  • -unavailable- added by psmith (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 05 Mar 2014 06:00:22 PM UTCpsmithAssigned toNone=>psmith
      Triage StatusNone=>Medium Effort

    Back to the top


    Powered by Savane 3.1-cleanup