bugmake - Bugs: bug #41781, Provide a fast fail path when a...

 
 

bug #41781: Provide a fast fail path when a target is compromized during a parallel build

Submitter:  None
Submitted:  Wed 05 Mar 2014 04:51:33 PM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  psmith Open/Closed:  Open
Component Version:  4.0 Operating System:  Any
Fixed Release:  None Triage Status:  Medium Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 14 Mar 2024 02:21:43 PM UTC, comment #4: 

Definitely I don't anticipate making any change in error handling the default.

The idea I had 10yrs ago is still my best idea (perhaps with an option to allow "immediate kill" as suggested by Patatti) but it's gotten even more complex since then: that solution works well for POSIX systems which use a pipe to manage the jobserver, but on other systems like Windows we simply use a counting semaphore so there's no facility there to notify the other instances of make by using different tokens in the jobserver pipe.

We'd need to create some separate, different communication channel for those systems.

This has been hiding behind work I've been trying to do to clean up the way make dies, which is currently not reliable because it calls a complex, non-signal-safe method in a signal handler.  Unfortunately changing THAT requires reworking how we handle signals in general, most specifically as relates to the jobserver facility.  Sigh.

Paul D. Smith <psmith>
Group administrator
Thu 14 Mar 2024 07:09:53 AM UTC, comment #3: 

In Gentoo, we always do clean builds, so the issue of resumption / corrupt rules wouldn't be a problem for us.

I wouldn't expect this functionality to be on by default if made available, but having the option would be great.

The main motivation for us is to have better build logs when people hit a failure, rather than it being buried possibly hundreds of lines above in very-parallel flat builds.

See also https://github.com/ninja-build/ninja/issues/2083.

Anonymous
Mon 28 Apr 2014 02:46:58 PM UTC, comment #2: 

Hi Paul. Thanks for the clarifications. I do not use recursive make, I have one makefile and one DAG (I read the recursive make considered harmful paper), but I understand that for completeness it should be covered.

Finding the source of the failure that caused all the other targets to abort is not obvious. My users were struggling with this when I was usign that patch. Perhaps a different error message for the jobs that were aborted vs. the job that caused the abortion could help sort things out in post-processing. Could it be a different exit code? I think if someone actively choses to use a fast fail option, they may be willing to accept some amount of unpleasant aspects.

"This means that make will still not kill any existing jobs that are running":
I would like to have an option that causes Make to kill all jobs in flight as soon as one of the targets has failed because it does save people time. A worst case scenario that comes to mind is the Friday afternoon frenzy when there are 20 integration builds on the CI server queue. People are trying to get new code accepted for the long weekend regression tests for a Monday morning delivery. They are eager to get home. And then Make waits for all parallel in-flight jobs to complete despite an earlier failure in one of the parallel jobs in the build at the head of the integration queue, and the admin who can kill this build from outside Make is away.

I wish I had spent more time learning GNU Make's source code!

Thanks.

Patatti <patatti>
Wed 05 Mar 2014 06:00:22 PM UTC, comment #1: 

There's some confusion in the reading material you mention, and in the bug report, about how GNU make actually works.

It's not true that enabling parallel builds will magically turn on the -k (keep-going) flag and that it can't be turned off.  Make with parallel builds enabled, and without -k, handles failed builds in exactly the same way that it would otherwise: it stops building things "as soon as it can".

When a parallel make instance detects a failed build then it will not start any new jobs and it will wait for all the currently-running parallel jobs to complete, then it will exit with an error code.  Make doesn't try to kill any running jobs, it waits for them to finish: killing jobs can lead to corrupted builds if the recipe doesn't expect to be killed (arguably this is a bug in the recipe since the user could always use CTRL-C but nevertheless it's not expected that make will kill things).

The problem you're seeing likely happens because you are using recursive make instances.  Suppose you have a makefile which runs three recursive instances of make.  You start it with -j2, so you have make instance A (the top one) spawning instance B and C (the children).  The third recursive make instance is not started (yet) because there are only 2 jobs slots available.

Now say that during the run of make instance B a job fails.  In that case, instance B detects that a failure happened and it won't start any new jobs and it will exit with an error as soon as all currently running jobs it started are completed.  When B exits with an error, instance A (the root) sees the failure and it won't start up the final recursive make at all.  However, already-running make instance C has no idea that B saw an error so C keeps building all its targets as usual, then exits.

If you have a large amount of parallelism and a recursive make environment, then you can get a large number of directories building happily along, unaware that one of their siblings has detected an error.

I've been thinking about this problem (which is not really the point of the patch you mention: that patch allows some finite number, less than "all" which is what you get with -k, errors to be ignored).

I think the right solution is that when a make instance gets an error, it notifies all the other make instances about the error.  I think the right way to do that is to start putting back a different token into the jobserver queue, which means "error detected".  Then when other make instances get that token they'll know one of their siblings had an error and they can stop building.  This means that make will still not kill any existing jobs that are running, but it won't start any new ones either, even in recursive environments.

The big open question is, if a make instance detects that some other make instance failed and so it wants to stop early, what should its exit code be?  I think it must exit with an error code, even though it, itself, did not detect any error, because it didn't completely build its target; we have to signal that to the parent in case the parent is relying on that for further processing.  On the other hand that means you'll get a list of error messages for all the recursive make invocations, which might be unpleasant.

Paul D. Smith <psmith>
Group administrator
Wed 05 Mar 2014 04:51:33 PM UTC, original submission:  

In continuous integration systems, fast failure without recovery is a highly desirable behavior. As such, I would like to see GNU Make completely abort fast at the first failure.

During automated parallel builds with make -j <N>, make attemps to build as much as it possibly can, and even when some targets fail, GNU Make continues. This is the exact opposite of a fast failure.

In continuous integration systems, there is no value in doing incremental builds: each new build starts from scratch. This ensures stability and reproducibility.

A patch was proposed in 2009 http://lists.gnu.org/archive/html/bug-make/2009-01/msg00035.html, I wuold like to see this integrated in GNU Make.

There is some background reading material:
http://ninjaverification.wordpress.com/tag/gnu-make/
http://lists.gnu.org/archive/html/help-make/2009-02/msg00010.html

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by patatti (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-03-05 psmith Assigned toNone psmith
        Triage StatusNone Medium Effort

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code