bugmake - Bugs: bug #14927, Fix for building archive members...

 
 

bug #14927: Fix for building archive members in parallel

Submitter:  Reid Madsen <srmadsen>
Submitted:  Tue 08 Nov 2005 12:19:27 AM UTC
Votes: 25
 
Severity:  3 - Normal Item Group:  Bug
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  3.81 Operating System:  Any
Fixed Release:  4.4.1 Triage Status:  Medium Effort
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 21 Jan 2023 07:09:53 PM UTC, comment #14: 

Opened a defect report against posix
https://austingroupbugs.net/view.php?id=1631

Dmitry Goncharov <dgoncharov>
Tue 03 Jan 2023 02:58:45 PM UTC, comment #13: 

I don't see the point in that.  If you want to do that you should just stop using the special archive features of make and treat it like any other target.  The only reason to use the special archive feature is to allow the object files to be considered intermediate, and be deleted off the disk.

If you're going to extract them again anyway then there's no point in having them deleted and you can just write out your archive rule the same way you'd write a program or shared library rule: without special syntax.

One thing to note: it's not quite straightforward to treat the archive contents the same as files on the disk.  The archive timestamp is a time_t which means it doesn't support sub-second timestamps, while almost every (POSIX) filesystem these days does support sub-second timestamps.  This difference can make timestamp comparison rebuild things when not needed (the file on disk is the same as the file in the archive, but the sub-second portion of the timestamp is stripped so the file on disk "appears" newer than the one in the archive).

Paul D. Smith <psmith>
Group administrator
Tue 03 Jan 2023 12:29:41 PM UTC, comment #12: 

I think the main confusion arises from not clearly separating storage from target.
make does not (and should not) concern itself with the underlying structure that stores the file as long as that storage satisfies certain requirements. Similarly for the archive acting as storage here and not as target.
Whether that storage supports parallelism is a different question.

On the other hand, modern file systems do support all that is needed. So intermediate targets could also do the trick.

By extracting all objects out of the archive but retaining their update time, as it is stored, would be the first step to discover who actually needs to be recompiled.
Then their intermediacy would make sense as it relates to the the real file system acting as a temporary.



SRCS := a.c b.c
OBJS := $(SRCS:.c=.o)
_:= $(shell $(AR) xo $(OBJS) $(LIBS))

$(OBJS):
  # compile

$(LIB): $(OBJS)
   # reconstruct lib from scratch




     

Alon Blayer-Gat <alonbg>
Tue 03 Jan 2023 07:29:43 AM UTC, comment #11: 

I pushed a change to allow this to work.  I didn't change the default rules to enable it but I added documentation on how to make the change.

A quick attempt to make the new behavior the default showed some failures in the regression tests which will need to be considered if we want to do that.

Paul D. Smith <psmith>
Group administrator
Tue 03 Jan 2023 04:51:54 AM UTC, comment #10: 

The real question is, should we modify the default rules to allow parallel archive builds to work out of the box?  Or should we just make it possible for users to rewrite the rules to allow it if they want to?

In order to allow it by default we'd have to violate POSIX requirements, which say that the rules for building archives are:

.c.a:
        $(CC) -c $(CFLAGS) $<
        $(AR) $(ARFLAGS) $@ $*.o
        rm -f $*.o

which is what GNU make uses (basically) and is clearly incompatible with parallel builds.  And, if we do update the default rules, should we put them back of .POSIX: is given?

Paul D. Smith <psmith>
Group administrator
Tue 03 Jan 2023 04:46:58 AM UTC, comment #9: 

We don't need to worry about shared libraries because make's special syntax only works for archives (created by ar) not for shared libraries, which are constructed like programs (created by the linker).  An archive stores a time last modified for every object in it, and they can be updated/removed/etc. from the archive one at a time.  You can't do that with shared libraries.

It's very true that you can't use intermediate files for shared libraries, just like you can't use them for programs.

I have a fix that I think works for archives plus parallelism, although it's not the same one suggested by Reid originally.

Paul D. Smith <psmith>
Group administrator
Tue 03 Jan 2023 04:31:23 AM UTC, comment #8: 

comment #5:

> The disadvantage to this is that the object files are not considered intermediate, and so they are left behind after the build completes rather than being deleted.


If this makefile is modified to build a shared object, rather than an archive, then everybody expects the object files to be left behind. Which raises the question whether this is really such a disadvantage.
In fact, there are reasons to keep these object files. E.g. for this a.c i'd write a unit test a.t.c, which contains function main and tests code from a.o. Then a.o and a.t.o are linked to an executable.

However, this is somewhat awkward, that make has a special syntax for building archives and make supports parallel builds and these two features cannot be used together.

Dmitry Goncharov <dgoncharov>
Tue 03 Jan 2023 03:36:07 AM UTC, comment #7: 

I was actually thinking about Reid's original request.  I have no problem with using some external tool such as a pseudo filesystem, but that's outside the purview of GNU Make.

These days it seems to me that intermediate files are less and less useful: the cost of disk space is a tiny fraction of what it once was and it makes me wonder how important it really is to delete intermediate files.  But probably there are still situations where it's helpful.

Paul D. Smith <psmith>
Group administrator
Tue 03 Jan 2023 02:03:31 AM UTC, comment #6: 

Took me a bit to reconstruct my 17y old previous response but yes - this is the requirement - on the one hand treating object files as intermediates and on the other doing the minimum required - just the delta.

"avoid the library syntax altogether"  would also be my choice.
When dealing with a file-system, files and last update time - make is doing just fine. An archives of files is a special case then. If it could have been abstracted as a "file-system" then there would be a lot to gain - not just for objects but also for any kind of archive that supports the last modification time for it's members.

And I believe this enhancement is outside the capabilities of what gmake should provide. There are already tools which can abstract an archive into a file system. Hence making the work with objects library trivial/natural.
The preliminary action then should be mounting the library (as a user-space file system) and then the library "disappears" and there is no intermediate files problem once unmounted as the end.

Something like:

$(LIB): $(shell mount LIB)
   $(AR) -U cru $@ $?
   unmount ...

That could also work for jar, zip, tar etc.

On the other hand, since object archives have been supported by gmake from it's very beginning then intrinsic file-system abstraction support just for it might be considered ...

Here are the 345 lines for ar https://sourceforge.net/p/avf/git/ci/master/tree/modules/uar.c

Alon Blayer-Gat <alonbg>
Mon 02 Jan 2023 09:18:15 PM UTC, comment #5: 

Of course the simplest thing to do is just avoid the library syntax altogether and write the makefile as:

SRCS := a.c b.c
OBJS := $(SRCS:.c=.o)
LIB := mylib.a

$(LIB): $(OBJS) ; $(AR) -U cru $@ $?

This ensures that we build just object files and the library is updated at the end, and parallel builds work fine.

The disadvantage to this is that the object files are not considered intermediate, and so they are left behind after the build completes rather than being deleted.  And of course we can't just declare them .INTERMEDIATE because it means that any time we modify one source file, make will have to rebuild them all.

Is that the requirement here, that we be able to treat these object files as intermediate?

Paul D. Smith <psmith>
Group administrator
Mon 31 Dec 2007 07:19:44 PM UTC, comment #4: 

Hi,

By treating the archive as if it was a file system - it is possible to leave Make 'clean' from specific implementations.
AVFS - http://sourceforge.net/projects/avf/

http://directory.fsf.org/project/AVFS   :

"A Virtual File System lets programs look inside archived or compressed files, or access remote files without recompiling the programs or changing the kernel. ... "

ar is supported.

/Alon

Alon Blayer-Gat <alonbg>
Mon 25 Dec 2006 11:03:16 PM UTC, comment #3: 

Following is a way to add objects in parallel with 3.80/81
The main idea -
*- depend files are used as sentinels of the object files
*- object files which do not have depend files are extracted from the library with their original modification date.
 
#FILES - a given list of source files no extension (e.g. .cc)
#L - The Archive
#DEP_FILES - depend files
#*_BASE - directories

DEP_FILES = $(patsubst %,$(DEP_BASE)/%.d,$(FILES))
vpath %.d $(DEP_BASE)

#make a list of existing depend files
existing_depend_files=$(if $(wildcard $L), \
$(foreach dep_file,$(FILES),$(wildcard $(DEP_BASE)/$(dep_file).d)),)

#make a list of non-existing depend files
non_existing_depend_files=$(filter-out $(existing_depend_files), \
$(patsubst %,$(DEP_BASE)/%.d,$(FILES)))

#If depend file does not exist then first try to take out object with original modification time
#Then just create the depend file
define make_nonexisting_depend
$(if $(wildcard $L),cd $(@D);$(AR) -xvo $(L) $*.o,)
$(DEPEND.cc) $< -MT '$1 $(OBJ_BASE)/$*.o' -MP -MF $1
endef

$(non_existing_depend_files): $(DEP_BASE)/%.d: %.cc
        $(call make_nonexisting_depend,$@)

#If depend file exists compile as well.
define make_existing_depend
$(COMPILE.cc) $< -MM -MT '$1 $(OBJ_BASE)/$*.o' -MP -MF $1 -o $(OBJ_BASE)/$*.o
endef

$(existing_depend_files): $(DEP_BASE)/%.d: %.cc
        $(call make_existing_depend,$@)

include $(patsubst %,$(DEP_BASE)/%.d,$(FILES))

#Library is made of:
#object files created by the existing depend files target
#object files from the non existing depend files list created by the object target itself
#If all object were updated remove library first
$L: $(existing_depend_files) $(patsubst $(DEP_BASE)/%.d,$(OBJ_BASE)/%.o,$(non_existing_depend_files))
        $(if $(filter-out $?,$^),@$(RM) $@; echo $@ was removed,)
        $(AR) $(ARFLAGS) $@ $(patsubst %.d,%.o,$(filter %.d,$?)) $(filter %.o,$?)     
        $(RM) $(OBJ_BASE)/*.o

$(OBJ_FILES): $(OBJ_BASE)/%.o: %.cc                                                                      
        $(COMPILE.cc) $< -o $@


/Alon

Alon Blayer-Gat <alonbg>
Thu 06 Apr 2006 06:00:04 PM UTC, comment #2: 

The crux of the issue here is building objects and adding them to archives IN PARALLEL.  Up until now, no version of GNUmake has ever supported this capability. Section 11.3 of the GNUmake manual describes the reasons why it doesn't work.

This bug, or enhancement, describes a way to fix GNUmake so that you can use the archive member syntax (e.g., lib(x.o)) in a parallel build environment. If you've never tried to build an archive in parrallel on a multi-processor machine using the archive member rules you would never experience the problem.

The default archive member rule is as follows:

(%.o): %.c
<TAB> $(CC) $(CFLAGS) $(CPPFLAGS) -c $< -o $*.o
<TAB> $(AR) r $@ $*.o
<TAB> $(RM) $*.o

The problem with the above rule is that when you build in parallel, you get multiple CC, and AR commands running simultaneously.  The CC commands are okay, because the generate separate .o files.  The AR commands fail because they are all updating the same library -- and the library is corrupted in the process.

The solution to this problem requires the following:

1. Disable the built in archive member rule so that the archives DO NOT occur in parallel. This is done as follows:

lib: lib(%.o): %.o ;

This keeps the AR command from occuring in parallel.  However, since the default rule for compiles has not been overridden the CC commands still occur in parallel.

2. Add the AR command to the following rule:

lib: lib($(OBJS))
<TAB> $(AR) r lib $?
<TAB> $(RM) $?

Now the compiles will all occur in parallel follows by a single AR command that updates all the new objects.

3. And to get #2 to work a bug fix is required so that the modification time of the newly compiled object is propagated to the archive member.

With these changes in place archives can be built in parallel, and we can correct a longstanding weakness in GNUmake.

I presently build on a server farm with a few hundred available CPU's.  When I build an archive with 60 members, the 60 CC commands are launched in parallel (distcc) across all available cpus.  Once all the compiles complete, a single AR command is launched to update the library.  Without these fixes the parallel aspect of this would be impossible -- I would be forced to build serially.  Yuck.

srmadsen

Reid Madsen <srmadsen>
Tue 04 Apr 2006 10:14:17 AM UTC, comment #1: 

I don't think this is a bug. You get what you deserved with the makefile you provided: you redefined the (%.o): %.c rule and as a result, you got no actual file in archive even though the rule claims to create one.

First, I checked your makefile with make 3.81 and 3.80. Both exhibit the following expected behavior:

- When the make is run for the first time, the library is built as you expect:

cc -c -o a.o a.c # to just compile the files
cc -c -o b.o b.c # to just compile the files
ar cru mylib.a a.o b.o # Add all new objects to the archive
rm -f a.o b.o # Remove the objects

- When a.c is touched and make is rerun, a.c is recompiled but not put into the archive:

cc -c -o a.o a.c # to just compile the files

This is what one would expect from your makefile: once a library is already created, its modification date is newer than any of the objects therein. Therefore, the rule that archives the newly built objects is not invoked.

Alexey Neyman <stilor>
Tue 08 Nov 2005 12:19:27 AM UTC, original submission:  

In section 11.3 of the GNUmake manual entitled "Dangers When Using Archives" it correctly warns people about the pitfalls of using the archive member rule to build libraries in parallel (i.e. -j4). The purpose of this submission is to describe how to fix this problem along with the required one line change to GNUmake to support it.

For example, assume the following Makefile:

SRCS := a.c b.c
OBJS := $(SRCS:.c=.o)
LIB := mylib.a

%.o: %.c
    $(CC) -c -o $@ $<

$(LIB): $(LIB)($(OBJS))
    @echo Finished building $@

When the above is executed in parallel, the following usually occurs:

example> make -j4
gcc -c -o a.o a.c
gcc -c -o b.o b.c
ar cru mylib.a a.o
ar cru mylib.a b.o
<error msg from one of the ar commands>
rm a.o
rm b.o

The logical solution to the parallelism issue is to only do the compiles in parallel, and to defer the archiving until all members have been built.  This is accomplished by changing the initial makefile as follows:

SRCS := a.c b.c
OBJS := $(SRCS:.c=.o)
LIB := mylib.a

%.o: %.c
    $(CC) -c -o $@ $<

(%.o): %.c             # Redefine archive member rule
    $(CC) -c -o $% $<  # to just compile the files

$(LIB): $(LIB)($(OBJS))
    $(AR) cru $@ $?    # Add all new objects to the archive
    $(RM) $?           # Remove the objects

Then when make is executed, the following occurs (from a clean state):

gcc -c -o a.o a.c
gcc -c -o b.o b.c
<The $(AR) and $(RM) commands are never executed>

Huh? What happened to the archive and delete commands? Well, as it turns out, make does not propagate the modification time of the object file to the archive member in the "(%.o): %.c" rule. The code that does this is in the "notice_finished_file()" function in remake.c. At about line 842 of that file/function the following code exists:

if ((ran && !file->phony) || touched)
    {
      struct file *f;
      int i = 0;

      /* If -n, -t, or -q and all the commands are recursive, we ran them so
         really check the target's mtime again.  Otherwise, assume the target
         would have been updated. */

      if (question_flag || just_print_flag || touch_flag)
        {
          for (i = file->cmds->ncommand_lines; i > 0; --i)
            if (! (file->cmds->lines_flags[i-1] & COMMANDS_RECURSE))
              break;
        }

      /* If there were no commands at all, it's always new. */

      else if (
#ifdef TEKTRONIX
       ar_name(file->name) ||
#endif
       (file->is_target && file->cmds == 0))
i = 1;

      file->last_mtime = i == 0 ? UNKNOWN_MTIME : NEW_MTIME;

Note the addition of the "ar_name(file->name) ||" in the TEKTRONIX section above.  When this code is enabled, make then produces the following when build using -j4.

example> make -j4
gcc -c -o a.o a.c
gcc -c -o b.o b.c
ar cru mylib.a a.o b.o
rm a.o b.o

If a.c is touched, and make is re-executed, the following occurs:

example> make -j4
gcc -c -o a.o a.c 
ar cru mylib.a a.o
rm a.o

You should also note that in the initial Makefile, the "@echo Finished building $@" text never appears.  The fix for that is the same fix described above.

I've been using this fix since GNUmake 3.75 and have been building libraries in parallel with no issues.  I will upload the file containing the fix for your review.

Instead of me carrying this fix around from version to version, how about incorporating this fix into the latest version and updating section 11.3 of the documentation with the above example.  Then you don't have to worry about synchronizing archive commands in some future release.

srmadsen

Reid Madsen <srmadsen>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #2720:  remake.c added by srmadsen (44KiB - application/octet-stream - Version of remake.c with the parallel archive member fix)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dgoncharov (Posted a comment)
  • -email is unavailable- added by psmith (Updated the item)
  • -email is unavailable- added by alonbg (Posted a comment)
  •  

    There are 25 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 10 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-03 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.4.1
        Triage StatusNone Medium Effort
    2013-10-09 psmith Component Version4.0 3.81
    2006-10-27 icarus Carbon-Copy- Added icarus
    2006-10-24 alonbg Carbon-Copy- Added alonbg
    2006-05-22 srmadsen Carbon-Copy- Added srmadsen
    2005-11-08 srmadsen Attached File- Added remake.c, #3088

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code