bugmake - Bugs: bug #9478, massive parallel runs tend to...

 
 

bug #9478: massive parallel runs tend to deteriorate to single-processing

Submitted by:  None
Submitted on:  Mon 28 Jun 2004 09:15:42 AM UTC  
 
Severity: 3 - NormalItem Group: Bug
Status: FixedPrivacy: Public
Assigned to: Paul D. Smith <psmith>Open/Closed: Closed
Component Version: 3.80Operating System: POSIX-Based
Fixed Release: 3.81Triage Status: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Sat 07 May 2005 03:31:59 PM UTC, comment #3:

I've fixed this problem, although I used a different solution than the one in Grant's patch, as I didn't want to duplicate the "wait for child" processing in the shell function. I also added some sanity testing similar to Tom's idea, where the top-level make will read the pipe before exiting, and print a message if it detects the wrong number of tokens. It also prints a message during processing of one of the make's detects that it's not handling its tokens correctly.

I tested this on a compile of GCC with -j4, but on a single-CPU box. It would be great if folks with lots more CPUs could give it a go.

Paul D. Smith <psmith>
Project AdministratorIn charge of this item.
Wed 16 Feb 2005 09:27:37 PM UTC, comment #2:

I encountered a similar problem when switching from RedHat 7.3 to Fedora Core 2. I applied the following patch which fixed the problem:

http://lists.gnu.org/archive/html/bug-make/2003-05/msg00009.html

John Stuart <johns>
Tue 29 Jun 2004 11:51:51 AM UTC, comment #1:

I (the original sumbitter) have checked this again with further instrumentation; an fstat() on
the job fd shows the amount of readable chars in it (as st_size), and does this also whenever a new job gets started.

So, at the end, the topmost make invocation should print
'n-1 chars readable' when done.

My observation now is that the higher -j n is with respect to the number of actual jobs to do (i.e. 500 > 125), the more characters are in the pipe at the end (often even 499).
The smaller the difference, the more likely is 0 bytes at
the end (so, having 1000 jobs with only 100 allowed will result in <<100 chars readable at the end).

Tom Riedl <riedlt>
Mon 28 Jun 2004 09:15:42 AM UTC, original submission:

When invoking GNU make-3.80 (GCC V3.3.1-compiled) on 50+ processor Suns (running Solaris 5.8/5.9), with
-j 100 (150,200,250) -l 45 (50,55), the XLOAD fever curve
(and other procmon tools as well) indicate that after
a massively parallel start the parallelism decreases over time
to a almost sequential crawl. This is the worse the less the
load limit is hit (i.e. on an unloaded machine, it happens more often than on a loaded machine).

As a quickly patched version using semaphores instead of
the pipe mechanism does not exhibit this behaviour (starts,
peaks to the load limit, remains there, and drops when
done), neither on loaded nor on unloaded machines, I
suspect problems with the token release mechanism.

BTW: The involved make process runs recursively, like this:
top level -> 35 || sublevels -> 160 || subsub levels -> 4000
|| required dependency submakefiles, 4000 || compilations
<- 160 || links, <- 35 || links, 1 top level link.

Notice to Paul: This happens with the original V.3.80
as well as with my load-peak-avoidance version

Tom Riedl (Thomas Dot Riedl At siemens Dot com)

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

Do you think this task is very important?
If so, you can click here to add your encouragement to it.
This task has 0 encouragements so far.

Only logged-in users can vote.

 

Please enter the title of George Orwell's famous dystopian book (it's a date):

 

 

Follow 7 latest changes.

Date Changed By Updated Field Previous Value => Replaced By
Sat 01 Apr 2006 07:11:20 AM UTCpsmithFixed Release4.0=>3.81
Sat 07 May 2005 03:31:59 PM UTCpsmithStatusNone=>Fixed
  Assigned toNone=>psmith
  Open/ClosedOpen=>Closed
  Fixed ReleaseNone=>4.0
Wed 30 Jun 2004 08:31:33 AM UTCriedltCarbon-Copy-=>Added thomas --DOT-- riedl --AT-- siemens --DOT-- com
Mon 28 Jun 2004 09:21:18 AM UTCriedltCarbon-Copy-=>Added riedlt

Back to the top


Powered by Savane 3.1-cleanup1