bugmake - Bugs: bug #105, Parallel builds should order by...

 
 

bug #105: Parallel builds should order by slowest-first

Submitted by:  Paul D. Smith <psmith>
Submitted on:  Sat 20 Apr 2002 08:39:38 PM UTC  
 
Severity: 1 - WishItem Group: Enhancement
Status: NonePrivacy: Public
Assigned to: NoneOpen/Closed: Open
Component Version: 3.78Operating System: POSIX-Based
Fixed Release: NoneTriage Status: Major Effort

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 18 Apr 2006 04:47:06 PM UTC, comment #3:

Closed support #105307 as a duplicate of this bug. See the details on that support request because it's slightly different, but any solution for this problem should ideally be "hookable" so that a generic re-ordering of prerequisites is possible.

Paul D. Smith <psmith>
Project Administrator
Thu 09 Feb 2006 12:26:11 PM UTC, comment #2:

Closed bug #13229 as a duplicate of this bug.

Paul D. Smith <psmith>
Project Administrator
Wed 27 Jul 2005 03:17:51 PM UTC, comment #1:

Paul,

I've implemented this before, but not within make. We used a custom version of makedepend that calculated the size of the entire compilation unit by summing up the sizes of all included files. Make depend would drop the sizes into a sub-dir which contained 'touched' files of the form:

<size>-file.c

Then we could sort the sub-dir by decreasing size to determine the best compile order. This was accomplished by doing:

SRCS := $(shell order-files $(size-dir))

Then we were very careful to not $(sort $(SRCS)).

IMO, trying to base the decision solely on the size of the primary input file will not result in a correct solution. The included files must be factored in to correctly determine potential compute time. And that may not be good enough. If those included files use lots of templates the compiler could run even slower for a small file than for a large file containing few template references.

Good luck on this one...

srmadsen

Reid Madsen <srmadsen>
Sat 20 Apr 2002 08:39:38 PM UTC, original submission:

I run on a 4-way machine and make extensive use of `make -j N'. I would like it to be a bit smarter in how it picks the order fo //el jobs, specifically I would like it to schedule the compilation of .c file in slowest-first order. This might be tricky so I would settle for largest first.

A common problem I have with //el makes is that the makefile lists .c file in arbitrary order and make just follows it. If there are four .c files and I run `make -j 3' and the files sizes are 1K, 1K, 1K and 10K respectively make will schedule three compiles on the smallest file, wait for a completion and then schedule the 10K file. This means that the elapsed time is effectively what it would be to compile a 1K and 10K serially.

If make were to schedule based on sizes it would start 10K, 1k & 1K and fit the last 1K in after one of the others had finished. Elapsed time would be the time taken to compile 10K.

As I mentioned above predicting how long a file will take to compile may be tricky but I assume make knows how big each files is when it construct the list of files to compile, all I want is it to sort the list on files size.

Reported via GNATS by Tim Phipps <tim_phipps@bigfoot.com>

Paul D. Smith <psmith>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by psmith (Updated the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follows 1 latest change.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 03 Jul 2010 07:39:26 PM UTCpsmithTriage StatusNone=>Major Effort

    Back to the top


    Powered by Savane 3.1-cleanup