bugmake - Bugs: bug #105, Parallel builds should order by...

 
 

bug #105: Parallel builds should order by slowest-first

Submitter:  Paul D. Smith <psmith>
Submitted:  Sat 20 Apr 2002 08:39:38 PM UTC
   
 
Severity:  1 - Wish Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  3.78 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  Major Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 22 Jan 2018 10:42:03 AM UTC, comment #4: 

It might make sense to implement a few different dependency ordering modes.  For development I'd like to see a random mode to uncover missing dependencies, and to speed up builds I'd like to see make prioritise jobs with a high number of dependants.

 Andrew Stormont <andyjstormont>
Tue 18 Apr 2006 04:47:06 PM UTC, comment #3: 

Closed support #105307 as a duplicate of this bug.  See the details on that support request because it's slightly different, but any solution for this problem should ideally be "hookable" so that a generic re-ordering of prerequisites is possible.

 Paul D. Smith <psmith>
Group administrator
Thu 09 Feb 2006 12:26:11 PM UTC, comment #2: 

Closed bug #13229 as a duplicate of this bug.

 Paul D. Smith <psmith>
Group administrator
Wed 27 Jul 2005 03:17:51 PM UTC, comment #1: 

Paul,

I've implemented this before, but not within make.  We used a custom version of makedepend that calculated the size of the entire compilation unit by summing up the sizes of all included files.  Make depend would drop the sizes into a sub-dir which contained 'touched' files of the form:

<size>-file.c

Then we could sort the sub-dir by decreasing size to determine the best compile order.  This was accomplished by doing:

SRCS := $(shell order-files $(size-dir))

Then we were very careful to not $(sort $(SRCS)).

IMO, trying to base the decision solely on the size of the primary input file will not result in a correct solution.  The included files must be factored in to correctly determine potential compute time.  And that may not be good enough.  If those included files use lots of templates the compiler could run even slower for a small file than for a large file containing few template references.

Good luck on this one...

srmadsen


Reid Madsen <srmadsen>
Sat 20 Apr 2002 08:39:38 PM UTC, original submission:  

I run on a 4-way machine and make extensive use of `make -j N'. I would like it to be a bit smarter in how it picks the order fo el jobs, specifically I would like it to schedule the compilation of .c file in slowest-first order.  This might be tricky so I would settle for largest first.

A common problem I have with el makes is that the makefile lists .c file in arbitrary order and make just follows it. If there are four .c files and I run `make -j 3' and the files sizes are 1K, 1K, 1K and 10K respectively make will schedule three compiles on the smallest file, wait for a completion and then schedule the 10K file. This means that the elapsed time is effectively what it would be to compile a 1K and 10K serially.

If make were to schedule based on sizes it would start 10K, 1k & 1K and fit the last 1K in after one of the others had finished. Elapsed time would be the time taken to compile 10K.

As I mentioned above predicting how long a file will take to compile may be tricky but I assume make knows how big each files is when it construct the list of files to compile, all I want is it to sort the list on files size.

Reported via GNATS by Tim Phipps <tim_phipps@bigfoot.com>

 Paul D. Smith <psmith>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by andyjstormont (Posted a comment)
  • -email is unavailable- added by psmith (Updated the item)

    •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-07-03 psmith Triage StatusNone Major Effort

    Back to the top

    Copyright © 2024 Free Software Foundation, Inc.
    Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.
    The Levitating, Meditating, Flute-playing Gnu logo is a GNU GPL'ed image provided by the Nevrax Design Team.
    Page source code

    Powered by Savane 3.14-f13d.
    Corresponding source code