bugmake - Bugs: bug #14927, Fix for building archive members...

 
 

bug #14927: Fix for building archive members in parallel

Submitted by:  Reid Madsen <srmadsen>
Submitted on:  Tue 08 Nov 2005 12:19:27 AM UTC  
Votes:  45  
 
Severity: 3 - NormalItem Group: Bug
Status: NonePrivacy: Public
Assigned to: NoneOpen/Closed: Open
Component Version: 3.81Operating System: Any
Fixed Release: NoneTriage Status: None

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Mon 31 Dec 2007 07:19:44 PM UTC, comment #4:

Hi,

By treating the archive as if it was a file system - it is possible to leave Make 'clean' from specific implementations.
AVFS - http://sourceforge.net/projects/avf/

http://directory.fsf.org/project/AVFS :

"A Virtual File System lets programs look inside archived or compressed files, or access remote files without recompiling the programs or changing the kernel. ... "

ar is supported.

/Alon

Alon Blayer-Gat <alonbg>
Mon 25 Dec 2006 11:03:16 PM UTC, comment #3:

Following is a way to add objects in parallel with 3.80/81
The main idea -
*- depend files are used as sentinels of the object files
*- object files which do not have depend files are extracted from the library with their original modification date.

#FILES - a given list of source files no extension (e.g. .cc)
#L - The Archive
#DEP_FILES - depend files
#*_BASE - directories

DEP_FILES = $(patsubst %,$(DEP_BASE)/%.d,$(FILES))
vpath %.d $(DEP_BASE)

#make a list of existing depend files
existing_depend_files=$(if $(wildcard $L), \
$(foreach dep_file,$(FILES),$(wildcard $(DEP_BASE)/$(dep_file).d)),)

#make a list of non-existing depend files
non_existing_depend_files=$(filter-out $(existing_depend_files), \
$(patsubst %,$(DEP_BASE)/%.d,$(FILES)))

#If depend file does not exist then first try to take out object with original modification time
#Then just create the depend file
define make_nonexisting_depend
$(if $(wildcard $L),cd $(@D);$(AR) -xvo $(L) $*.o,)
$(DEPEND.cc) $< -MT '$1 $(OBJ_BASE)/$*.o' -MP -MF $1
endef

$(non_existing_depend_files): $(DEP_BASE)/%.d: %.cc
$(call make_nonexisting_depend,$@)

#If depend file exists compile as well.
define make_existing_depend
$(COMPILE.cc) $< -MM -MT '$1 $(OBJ_BASE)/$.o' -MP -MF $1 -o $(OBJ_BASE)/$.o
endef

$(existing_depend_files): $(DEP_BASE)/%.d: %.cc
$(call make_existing_depend,$@)

include $(patsubst %,$(DEP_BASE)/%.d,$(FILES))

#Library is made of:
#object files created by the existing depend files target
#object files from the non existing depend files list created by the object target itself
#If all object were updated remove library first
$L: $(existing_depend_files) $(patsubst $(DEP_BASE)/%.d,$(OBJ_BASE)/%.o,$(non_existing_depend_files))
$(if $(filter-out $?,$^),@$(RM) $@; echo $@ was removed,)
$(AR) $(ARFLAGS) $@ $(patsubst %.d,%.o,$(filter %.d,$?)) $(filter %.o,$?)
$(RM) $(OBJ_BASE)/*.o

$(OBJ_FILES): $(OBJ_BASE)/%.o: %.cc
$(COMPILE.cc) $< -o $@

/Alon

Alon Blayer-Gat <alonbg>
Thu 06 Apr 2006 06:00:04 PM UTC, comment #2:

The crux of the issue here is building objects and adding them to archives IN PARALLEL. Up until now, no version of GNUmake has ever supported this capability. Section 11.3 of the GNUmake manual describes the reasons why it doesn't work.

This bug, or enhancement, describes a way to fix GNUmake so that you can use the archive member syntax (e.g., lib(x.o)) in a parallel build environment. If you've never tried to build an archive in parrallel on a multi-processor machine using the archive member rules you would never experience the problem.

The default archive member rule is as follows:

(%.o): %.c
<TAB> $(CC) $(CFLAGS) $(CPPFLAGS) -c $< -o $*.o
<TAB> $(AR) r $@ $*.o
<TAB> $(RM) $*.o

The problem with the above rule is that when you build in parallel, you get multiple CC, and AR commands running simultaneously. The CC commands are okay, because the generate separate .o files. The AR commands fail because they are all updating the same library -- and the library is corrupted in the process.

The solution to this problem requires the following:

1. Disable the built in archive member rule so that the archives DO NOT occur in parallel. This is done as follows:

lib: lib(%.o): %.o ;

This keeps the AR command from occuring in parallel. However, since the default rule for compiles has not been overridden the CC commands still occur in parallel.

2. Add the AR command to the following rule:

lib: lib($(OBJS))
<TAB> $(AR) r lib $?
<TAB> $(RM) $?

Now the compiles will all occur in parallel follows by a single AR command that updates all the new objects.

3. And to get #2 to work a bug fix is required so that the modification time of the newly compiled object is propagated to the archive member.

With these changes in place archives can be built in parallel, and we can correct a longstanding weakness in GNUmake.

I presently build on a server farm with a few hundred available CPU's. When I build an archive with 60 members, the 60 CC commands are launched in parallel (distcc) across all available cpus. Once all the compiles complete, a single AR command is launched to update the library. Without these fixes the parallel aspect of this would be impossible -- I would be forced to build serially. Yuck.

srmadsen

Reid Madsen <srmadsen>
Tue 04 Apr 2006 10:14:17 AM UTC, comment #1:

I don't think this is a bug. You get what you deserved with the makefile you provided: you redefined the (%.o): %.c rule and as a result, you got no actual file in archive even though the rule claims to create one.

First, I checked your makefile with make 3.81 and 3.80. Both exhibit the following expected behavior:

- When the make is run for the first time, the library is built as you expect:

cc -c -o a.o a.c # to just compile the files
cc -c -o b.o b.c # to just compile the files
ar cru mylib.a a.o b.o # Add all new objects to the archive
rm -f a.o b.o # Remove the objects

- When a.c is touched and make is rerun, a.c is recompiled but not put into the archive:

cc -c -o a.o a.c # to just compile the files

This is what one would expect from your makefile: once a library is already created, its modification date is newer than any of the objects therein. Therefore, the rule that archives the newly built objects is not invoked.

Alexey Neyman <stilor>
Tue 08 Nov 2005 12:19:27 AM UTC, original submission:

In section 11.3 of the GNUmake manual entitled "Dangers When Using Archives" it correctly warns people about the pitfalls of using the archive member rule to build libraries in parallel (i.e. -j4). The purpose of this submission is to describe how to fix this problem along with the required one line change to GNUmake to support it.

For example, assume the following Makefile:

SRCS := a.c b.c
OBJS := $(SRCS:.c=.o)
LIB := mylib.a

%.o: %.c
$(CC) -c -o $@ $<

$(LIB): $(LIB)($(OBJS))
@echo Finished building $@

When the above is executed in parallel, the following usually occurs:

example> make -j4
gcc -c -o a.o a.c
gcc -c -o b.o b.c
ar cru mylib.a a.o
ar cru mylib.a b.o
<error msg from one of the ar commands>
rm a.o
rm b.o

The logical solution to the parallelism issue is to only do the compiles in parallel, and to defer the archiving until all members have been built. This is accomplished by changing the initial makefile as follows:

SRCS := a.c b.c
OBJS := $(SRCS:.c=.o)
LIB := mylib.a

%.o: %.c
$(CC) -c -o $@ $<

(%.o): %.c # Redefine archive member rule
$(CC) -c -o $% $< # to just compile the files

$(LIB): $(LIB)($(OBJS))
$(AR) cru $@ $? # Add all new objects to the archive
$(RM) $? # Remove the objects

Then when make is executed, the following occurs (from a clean state):

gcc -c -o a.o a.c
gcc -c -o b.o b.c
<The $(AR) and $(RM) commands are never executed>

Huh? What happened to the archive and delete commands? Well, as it turns out, make does not propagate the modification time of the object file to the archive member in the "(%.o): %.c" rule. The code that does this is in the "notice_finished_file()" function in remake.c. At about line 842 of that file/function the following code exists:

if ((ran && !file->phony) || touched)
{
struct file *f;
int i = 0;

/* If -n, -t, or -q and all the commands are recursive, we ran them so
really check the target's mtime again. Otherwise, assume the target
would have been updated. */

if (question_flag || just_print_flag || touch_flag)
{
for (i = file->cmds->ncommand_lines; i > 0; --i)
if (! (file->cmds->lines_flags[i-1] & COMMANDS_RECURSE))
break;
}

/* If there were no commands at all, it's always new. */

else if (
#ifdef TEKTRONIX
ar_name(file->name) ||
#endif
(file->is_target && file->cmds == 0))
i = 1;

file->last_mtime = i == 0 ? UNKNOWN_MTIME : NEW_MTIME;

Note the addition of the "ar_name(file->name) ||" in the TEKTRONIX section above. When this code is enabled, make then produces the following when build using -j4.

example> make -j4
gcc -c -o a.o a.c
gcc -c -o b.o b.c
ar cru mylib.a a.o b.o
rm a.o b.o

If a.c is touched, and make is re-executed, the following occurs:

example> make -j4
gcc -c -o a.o a.c
ar cru mylib.a a.o
rm a.o

You should also note that in the initial Makefile, the "@echo Finished building $@" text never appears. The fix for that is the same fix described above.

I've been using this fix since GNUmake 3.75 and have been building libraries in parallel with no issues. I will upload the file containing the fix for your review.

Instead of me carrying this fix around from version to version, how about incorporating this fix into the latest version and updating section 11.3 of the documentation with the above example. Then you don't have to worry about synchronizing archive commands in some future release.

srmadsen

Reid Madsen <srmadsen>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #2720:  remake.c added by srmadsen (44KiB - application/octet-stream - Version of remake.c with the parallel archive member fix)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by psmith (Updated the item)
  • -unavailable- added by alonbg (Posted a comment)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 45 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 5 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Wed 09 Oct 2013 04:49:04 AM UTCpsmithComponent Version4.0=>3.81
    Fri 27 Oct 2006 08:44:29 PM UTCicarusCarbon-Copy-=>Added icarus
    Tue 24 Oct 2006 08:11:22 PM UTCalonbgCarbon-Copy-=>Added alonbg
    Mon 22 May 2006 03:38:41 PM UTCsrmadsenCarbon-Copy-=>Added srmadsen
    Tue 08 Nov 2005 12:19:27 AM UTCsrmadsenAttached File-=>Added remake.c, #3088

    Back to the top


    Powered by Savane 3.1-cleanup1