bugmake - Bugs: bug #14853, make randomly fails with...

 
 

bug #14853: make randomly fails with interrupted system call on slower filesystems with parallel builds

Submitted by:  None
Submitted on:  Mon 24 Oct 2005 10:11:40 PM UTC  
Votes:  6  
 
Severity:  3 - Normal Item Group:  Bug
Status:  Duplicate Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Component Version:  3.80 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 13 Mar 2016 04:49:15 PM UTC, comment #10: 

In Git I've implemented the use of pselect() for systems which support it, which is most modern systems (it was added to POSIX in 2001 or so).  Also many  more system calls in GNU make have been changed to use the EINTR loop to avoid problems with SIGCHLD, for systems that don't support pselect().

I think I'll close this as a duplicate of bug #46261

Paul D. Smith <psmith>
Project Administrator
Fri 15 Dec 2006 04:51:08 PM UTC, comment #9: 

I have yet to test the patch.  I am about to.  But i am seeing this frequently on Linux on my T1000,  its a 8 core box and the buildsystem  uses -j32 all the storage is local on the build box. the box has 16gb ram, with no swap configured.

Dennis Gilmore <ausil>
Wed 11 Oct 2006 08:11:08 PM UTC, comment #8: 

It was indeed the same problem. Since Steve Williams patch does not
seem to be available, I used the description to create my own.
I have built an entire gnu/linux distro including gcc, glibc and the kernel using the patched make at -j12 on a 6 core sun T1000 without a single failure. Without the patch, I could only build gcc successfully about 1 in 3 times.

I highly recommend a fix such as this be applied ASAP since make is curently very broken at high -j# on multi-core machines.

My patch is attached for reference.

Andrew Walrond <awalrond>
Wed 11 Oct 2006 01:18:36 PM UTC, comment #7: 

I am getting intermittent symptoms of this nature when building things like the linux kernel or gcc under gnu/linux on a Sun T1000 (niagra with 6 cores, 24 threads) with make -j12 with version 3.81

E.g.

<snip>
/bin/sh: line 4: 28895 Segmentation fault      make all
make[1]: *** [all-build-libiberty] Error 139
make[1]: *** Waiting for unfinished jobs....
<snip>
...
<snip>
make[1]: Leaving directory `/tmp/gcc-4-1/work'
make: *** [all] Error 2
make: INTERNAL: Exiting with 10 jobserver tokens available; should be 12!

Does this look like the same problem?
Anything I can do to diagnose this further?

Andrew Walrond <awalrond>
Sat 05 Nov 2005 03:24:21 PM UTC, comment #6: 

I agree that from a CPU load perspective you probably won't see a huge hit, since you just wake up for a second to try the read.  There are factors that make it less ideal: if your make process is huge (as some are when you use non-recursive make paradigms) there could be swapping involved--although not that likely if the wakeup is short.  Then there is the tension between how long to sleep (using less CPU, etc.) and how quickly you notice something was finished (to minimize the latency between jobs).  Obviously 1 second, for example, is too long to wait.  I don't want to do any sort of hard loop, of course!  That's a non-starter.  I suppose for systems without an identifiable sub-second sleep we could just wait for 1 second... very unlikely that an operating system like that could do much with parallel jobs anyway.

Anyway, I acknowledge your point that even if it's not the most elegant solution, it does have the advantage of working regardless of system call restart facilities.

I'm still interested to know whether you are able to reproduce the busted system call problem with the latest beta version of GNU make, though.

Thanks.

Paul D. Smith <psmith>
Project Administrator
Fri 28 Oct 2005 07:52:17 PM UTC, comment #5: 

Here is the output of:
  prstat -mc -s cpu -U src_dev 1
for the command:
  smake -j6;make clean;pmake -j6

It doesn't show any major new cpu consumption for a polling make, it does show zero signals for polling make. Overall, performance seems to be very similar for both versions.

   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
  4200 src_dev  8.3 2.5 0.1 0.0 0.0 0.0  89 0.0  11   2  1K   0 smake/1
  4198 src_dev  6.9 1.6 0.1 0.0 0.0 0.0  91 0.0  14   2  1K   2 smake/1
  4206 src_dev  3.1 1.6 0.1 0.0 0.0 0.0  95 0.0  16   0 897   0 smake/1
Total: 20 processes, 20 lwps, load averages: 0.21, 0.28, 0.44
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
  4228 src_dev  7.4 1.9 0.0 0.0 0.0 0.0  88 2.6  28   3  1K   7 smake/1
Total: 29 processes, 29 lwps, load averages: 0.21, 0.28, 0.44
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
  4228 src_dev  0.5 0.3 0.0 0.0 0.0 0.0  99 0.0  12   0 156   6 smake/1
Total: 31 processes, 31 lwps, load averages: 0.22, 0.28, 0.44
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
  4228 src_dev  0.2 0.2 0.0 0.0 0.0 0.0  98 1.7  11   0 121   8 smake/1
  4198 src_dev  0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 smake/1
  4200 src_dev  0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 smake/1
Total: 16 processes, 16 lwps, load averages: 0.27, 0.29, 0.45
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
  4417 src_dev  8.9 2.7 0.1 0.0 0.0 0.0  88 0.0  12   2  1K   0 pmake/1
  4415 src_dev  7.4 1.6 0.1 0.0 0.0 0.0  91 0.0  15   2  1K   0 pmake/1
  4423 src_dev  3.4 1.8 0.1 0.0 0.0 0.0  95 0.0  17   2 958   0 pmake/1
Total: 20 processes, 20 lwps, load averages: 0.30, 0.30, 0.45
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
  4445 src_dev  7.7 2.2 0.0 0.0 0.0 0.0  59  31  82  11  1K   0 pmake/1
Total: 31 processes, 31 lwps, load averages: 0.33, 0.30, 0.45
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
  4445 src_dev  0.6 0.5 0.0 0.0 0.0 0.0  98 1.0 106   0 394   0 pmake/1
Total: 31 processes, 31 lwps, load averages: 0.37, 0.31, 0.45
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
  4445 src_dev  0.6 0.5 0.0 0.0 0.0 0.0  97 2.2 109   2 419   0 pmake/1
Total: 30 processes, 30 lwps, load averages: 0.42, 0.32, 0.45
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
  4445 src_dev  0.3 0.2 0.0 0.0 0.0 0.0  98 1.8  16   0 115   0 pmake/1
  4415 src_dev  0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 pmake/1
  4417 src_dev  0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 pmake/1
Total: 16 processes, 16 lwps, load averages: 0.48, 0.33, 0.46

Steve Williams <steve4williams>
Fri 28 Oct 2005 06:55:00 PM UTC, comment #4: 

Sorry, I forgot to mention that, yes, I set the pipe reads to non-blocking.

I agree, polling is less elegant, but in this case, the use of polling actually eliminates the root cause of the problem.

I also agree a portable sub-second sleep is tricky, maybe a loop with a yield could be substituted on systems that have no such facility.

I took some measurements on my machine here of the relative performance of polling make and signal make (called pmake and smake respectively). I built make-3.80 three times with each version of make using 6 parallel jobs on a SUNFire V440, 4xUltraSparc3i

pmake -j6
real    0m4.66s
user    0m11.82s
sys     0m0.70s

real    0m4.87s
user    0m11.88s
sys     0m0.73s

real    0m4.37s
user    0m11.88s
sys     0m0.69s

smake -j6
real    0m4.89s
user    0m11.87s
sys     0m0.70s

real    0m4.39s
user    0m11.89s
sys     0m0.69s

real    0m4.84s
user    0m11.84s
sys     0m0.69s

As can be seen the time taken is essentially the same

Steve Williams <steve4williams>
Fri 28 Oct 2005 06:15:07 PM UTC, comment #3: 

I'll look at the patch.  I'm assuming you've modified the read() so that it's non-blocking.  That should take care of any race conditions (even if a job ends just before the read, it will only be an extra X amount of time (where X is the time make sleeps) before it notices the job is done.

Note that finding a portable way to sleep for <1 second is tricky: older versions of UNIX may not support usleep() or nanosleep() etc.

Also, I just find polling a less elegant design :-).  However, I'll look at your changes and think about them.

I'd be very interested to know whether the changes in the beta fix your problem, however.

Paul D. Smith <psmith>
Project Administrator
Fri 28 Oct 2005 05:32:07 PM UTC, comment #2: 

I have also implemented a workaround.

I disabled the SIG_CHLD in main.c so that make receives no interrupts.

The job server (job.c) works as normal, except when there are no tokens left. When there are no tokens left, make sleeps for 100mS, then retries for a token. It keeps doing this until it gets a token.

The existing code already cleans up the children, so no new code was required to handle the reaping of child processes.

I have been using this version of make to build gdb, gcc, ACE plus our application. One side effect was that applications I have never been able to successfully build in parallel now build - for example gcc.

If you are interested I have attached a patch. I also added an option to configure (--enable-job-poll) to turn the feature on. (The change was made in configure.in, the patch includes updates to configure and Makefile.in).

Anonymous
Fri 28 Oct 2005 03:23:44 PM UTC, comment #1: 

The problem is that SA_RESTART is not universally implemented.  On Solaris in particular, any non-local filesystem (like NFS etc.) doesn't adhere to SA_RESTART characteristics: system calls are not restarted when they are interrupted even when SA_RESTART is set.

Linux, apparently, does work properly (I've not seen any Linux users have this problem).

Anyway, I added some code into GNU make to try to provide restart-ability on systems that don't handle SA_RESTART universally.  Try the latest beta:

  ftp://alpha.gnu.org/gnu/make/make-3.81beta3.tar.gz

and see if it works any better for you.

Paul D. Smith <psmith>
Project Administrator
Mon 24 Oct 2005 10:11:40 PM UTC, original submission:  

make-3.80 sparc64-sun-solaris2.10

We experience a lot of failures when using parallel builds over NFS with interrupted system calls.

The problem becomes especially bad when the NFS servers become sluggish under heavy load.

I see that make uses the SA_RESTART flag when it builds, so I am assuming that system calls should be restarted if interrupted by a signal, though my experience is to the contrary.

Has anyone else experienced this problem? If it is a general problem, I would have expected to already see a bug report here, but there appears to be none.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #10950:  make-3.81.apw1.patch added by awalrond (2KiB - text/x-patch - Patch against version 3.81 to fix Bug #14853)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by ausil (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 6 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-03-13 psmith StatusNone => Duplicate
        Open/ClosedOpen => Closed
    2006-12-15 ausil Carbon-Copy- => Added ausil
    2006-10-11 awalrond Attached File- => Added make-3.81.apw1.patch, #10950
        Carbon-Copy- => Added andrew@walrond.org
        Carbon-Copy- => Added awalrond
    2005-10-25 steve4williams Carbon-Copy- => Added steve4williams

    Back to the top


    Powered by Savane 3.6