Tue 08 Nov 2005 12:19:27 AM UTC, original submission:
In section 11.3 of the GNUmake manual entitled "Dangers When Using Archives" it correctly warns people about the pitfalls of using the archive member rule to build libraries in parallel (i.e. -j4). The purpose of this submission is to describe how to fix this problem along with the required one line change to GNUmake to support it.
For example, assume the following Makefile:
SRCS := a.c b.c
OBJS := $(SRCS:.c=.o)
LIB := mylib.a
%.o: %.c
$(CC) -c -o $@ $<
$(LIB): $(LIB)($(OBJS))
@echo Finished building $@
When the above is executed in parallel, the following usually occurs:
example> make -j4
gcc -c -o a.o a.c
gcc -c -o b.o b.c
ar cru mylib.a a.o
ar cru mylib.a b.o
<error msg from one of the ar commands>
rm a.o
rm b.o
The logical solution to the parallelism issue is to only do the compiles in parallel, and to defer the archiving until all members have been built. This is accomplished by changing the initial makefile as follows:
SRCS := a.c b.c
OBJS := $(SRCS:.c=.o)
LIB := mylib.a
%.o: %.c
$(CC) -c -o $@ $<
(%.o): %.c # Redefine archive member rule
$(CC) -c -o $% $< # to just compile the files
$(LIB): $(LIB)($(OBJS))
$(AR) cru $@ $? # Add all new objects to the archive
$(RM) $? # Remove the objects
Then when make is executed, the following occurs (from a clean state):
gcc -c -o a.o a.c
gcc -c -o b.o b.c
<The $(AR) and $(RM) commands are never executed>
Huh? What happened to the archive and delete commands? Well, as it turns out, make does not propagate the modification time of the object file to the archive member in the "(%.o): %.c" rule. The code that does this is in the "notice_finished_file()" function in remake.c. At about line 842 of that file/function the following code exists:
if ((ran && !file->phony) || touched)
{
struct file *f;
int i = 0;
/* If -n, -t, or -q and all the commands are recursive, we ran them so
really check the target's mtime again. Otherwise, assume the target
would have been updated. */
if (question_flag || just_print_flag || touch_flag)
{
for (i = file->cmds->ncommand_lines; i > 0; --i)
if (! (file->cmds->lines_flags[i-1] & COMMANDS_RECURSE))
break;
}
/* If there were no commands at all, it's always new. */
else if (
#ifdef TEKTRONIX
ar_name(file->name) ||
#endif
(file->is_target && file->cmds == 0))
i = 1;
file->last_mtime = i == 0 ? UNKNOWN_MTIME : NEW_MTIME;
Note the addition of the "ar_name(file->name) ||" in the TEKTRONIX section above. When this code is enabled, make then produces the following when build using -j4.
example> make -j4
gcc -c -o a.o a.c
gcc -c -o b.o b.c
ar cru mylib.a a.o b.o
rm a.o b.o
If a.c is touched, and make is re-executed, the following occurs:
example> make -j4
gcc -c -o a.o a.c
ar cru mylib.a a.o
rm a.o
You should also note that in the initial Makefile, the "@echo Finished building $@" text never appears. The fix for that is the same fix described above.
I've been using this fix since GNUmake 3.75 and have been building libraries in parallel with no issues. I will upload the file containing the fix for your review.
Instead of me carrying this fix around from version to version, how about incorporating this fix into the latest version and updating section 11.3 of the documentation with the above example. Then you don't have to worry about synchronizing archive commands in some future release.
srmadsen
|