bugmake - Bugs: bug #20394, vpath directive drops entries

 
 

bug #20394: vpath directive drops entries

Submitter:  None
Submitted:  Sat 07 Jul 2007 08:09:38 AM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  Fixed Privacy:  Public
Assigned to:  psmith Open/Closed:  Closed
Component Version:  3.81 Operating System:  POSIX-Based
Fixed Release:  4.4 Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 02 Jan 2023 07:38:18 PM UTC, comment #9: 

This was fixed with the implementation of the cache outdating facility in GNU make 4.4.

Paul D. Smith <psmith>
Group administrator
  Spam posted by sevanath
Sat 14 Jul 2007 12:58:47 PM UTC, comment #7: 

As to when to use the cache vs. avoid it, I actually meant the opposite from what you wrote. Maybe it's more clear with a practical example:

$ time make -C fltk-1.1.7, up-to-date, on a (slow) Ubuntu box:
make (3.81b4, as is):
- 3.8 sec
make (cvs, no default_terminal_rules, no cache):
- 2.0 sec

Anonymous
Thu 12 Jul 2007 07:57:48 AM UTC, comment #6: 

I haven't looked into it carefully, but it's not immediately clear to me that it's a simple thing to avoid the directory cache for chained/intermediate rules, vs. any other kind of rule.

The disadvantage with the "is_stale" boolean is that every time something goes stale you have to walk through the entire cache and reset all the boolean values.

If you use a counter then you don't have to touch anything in the cache until and unless you need to look at that cached data, then you compare (and update) the counter kept with that set of data.

It may be that our caches are not large enough to make a difference, practically speaking; I'm not sure.

Paul D. Smith <psmith>
Group administrator
Wed 11 Jul 2007 06:15:46 PM UTC, comment #5: 

Timestamps are a different matter, gmake doesn't use the cache to check timestamps of files. Also, what I meant was chained rules with intermediate files, not first-level implicit rules. Basically it is so that gmakes spends all win from the cache on it's own built-in chained and match-anything rules. For the user the cache brings no benefit, only the known inconsistencies.

However it is maybe right that two different behaviours is too confusing. Probably it would be better to use no cache at all and get rid of some exotic built-in rules instead.

As to empty the cache after each subprocess, well it could work too. I don't quite see the deal with the counter though, shouldn't a boolean 'is_stale' suffice?

Anonymous
Tue 10 Jul 2007 06:50:59 AM UTC, comment #4: 

Something like that could be tricky to accomplish, since make is highly recursive: when make starts checking for timestamps on files it doesn't know whether that file was found using implicit or explicit rules, etc.

Plus, I think it would be even more confusing to have some kinds of targets (implicit) behave unexpectedly while others (explicit) behaved as expected, but more slowly.

To fix this problem we need to add a way to determine if the cache is stale.

My idea on how to do this is for make to keep a counter that is incremented every time make invokes a subprocess.  Or maybe, from a correctness point of view, it should be updated every time a subprocess finishes.  Then when a directory was cached it would store the current value of that counter.  The next time the cached data was consulted, if the current counter was greater than the stored counter the cache would be considered stale and new data would be read.

This would significantly reduce the efficiency of the cache, but it would increase correctness and the cache would still be useful while make is looking for things to build--for situations where there's very little or nothing to do the cache would still be a big win.

Paul D. Smith <psmith>
Group administrator
Mon 09 Jul 2007 08:25:24 PM UTC, comment #3: 

It would maybe make sense to rely on the directory cache only for intermediate search and match-anything rules. Because these two alone increase the number of file-queries by a factor of about 10. That is just from the 5 built-in rules (which moreover likely nobody ever uses).

Anonymous
Sat 07 Jul 2007 09:01:14 PM UTC, comment #2: 

It's not exactly correct to say that GNU make caches directories from the 10th on, but you're on exactly the right track; thanks for the note.

What make actually does is cache EVERY directory... BUT it caches them "lazily", AND it only allows 10 directories to be open at any one time.  Basically, when make wants to check a file in a directory it opens the directory for reading and looks for that file.  As it reads the directory it caches the contents.  Once the file is found, it leaves the directory open but stops reading.  If make looks for another file in that directory and it's not in the cache, make continues reading (and caching) where it left off.  If it gets to the end of the directory it closes the directory.

In order to avoid too many directories being open by make at one time, if make gets to a point where more than 10 directories are open, it will completely read and cache the current (latest) directory, then close it.

The issue of make's caching is well-known (see bug #443 for example).  In your case it takes an odd twist but this is definitely the issue.  A completely correctly-written makefile shouldn't run across cache problems, but I recognize there are situations where it's not possible to write that makefile.  However, in your case I think you can do what you want.  Change your makefile like this:


LIBS := $(patsubst %,-l%,$(SUBDIRS))
LIBTARGETS := $(foreach L,$(LIBS),$(L)/lib$(L).a)

all: $(TARGET)

$(TARGET): $(LIBTARGETS)
        $(CC) -o $@ $(LIBDIRS) $(LIBS)

$(LIBTARGETS): FORCE
        $(MAKE) -w -C $(@D) -f Makefile $(MAKECMDGOALS)
FORCE:


This has some other advantages over the method you were using (for one thing, your example does not work properly if any of the sub-makes fail).

Paul D. Smith <psmith>
Group administrator
Sat 07 Jul 2007 06:53:28 PM UTC, comment #1: 

gmake caches directories from the 10th on. So it will not see a file in those directories if it didn't exist initially (except under windows NT, where it checks the directory timestamps).

Just looks like foo1..8 plus lib, include and . were one more than 10.

Anonymous
Sat 07 Jul 2007 08:09:38 AM UTC, original submission:  

With GNU make v3.81 (as well as earlier versions, eg. 3.81beta4 and v3.80), the vpath directive for libraries seems to "forget" entries under certain circumstances.

Example:

A project with the following directory layout:

Makefile
foo1/
 foo1/Makefile
foo2/
 foo2/Makefile
foo3/
 foo3/Makefile
foo4/
 foo4/Makefile
foo5/
 foo5/Makefile
foo6/
 foo6/Makefile
foo7/
 foo7/Makefile
foo8/
 foo8/Makefile
include/
lib/

and this top-level Makefile:


#####################
SUBDIRS  := foo1 foo2 foo3 foo4 foo5 foo6 foo7 foo8
TARGET   := app
LIBDIRS  := $(patsubst %,-L%,$(SUBDIRS))
LIBS     := $(patsubst %,-l%,$(SUBDIRS))


vpath %.h include
vpath %.a lib $(SUBDIRS)


all: subdirs $(TARGET)

subdirs:
        @(for module in $(SUBDIRS); \
                do \
                        $(MAKE) -w -C $$module -f Makefile $(MAKECMDGOALS); \
                done)

$(TARGET): $(LIBS)
        $(CC) -o $@ $(LIBDIRS) $(LIBS)
#####################


and this as each of the subdir-Makefiles:


LIBNAME := $(shell basename `pwd`)
LIBFILENAME := lib$(LIBNAME).a
LIBTARGET := $(LIBFILENAME)

all: $(LIBTARGET)

$(LIBTARGET):
        touch $@





Output:


$ make
make[1]: Entering directory `/home/luna/tmp/testmake3/foo1'
touch libfoo1.a
make[1]: Leaving directory `/home/luna/tmp/testmake3/foo1'
make[1]: Entering directory `/home/luna/tmp/testmake3/foo2'
touch libfoo2.a
make[1]: Leaving directory `/home/luna/tmp/testmake3/foo2'
make[1]: Entering directory `/home/luna/tmp/testmake3/foo3'
touch libfoo3.a
make[1]: Leaving directory `/home/luna/tmp/testmake3/foo3'
make[1]: Entering directory `/home/luna/tmp/testmake3/foo4'
touch libfoo4.a
make[1]: Leaving directory `/home/luna/tmp/testmake3/foo4'
make[1]: Entering directory `/home/luna/tmp/testmake3/foo5'
touch libfoo5.a
make[1]: Leaving directory `/home/luna/tmp/testmake3/foo5'
make[1]: Entering directory `/home/luna/tmp/testmake3/foo6'
touch libfoo6.a
make[1]: Leaving directory `/home/luna/tmp/testmake3/foo6'
make[1]: Entering directory `/home/luna/tmp/testmake3/foo7'
touch libfoo7.a
make[1]: Leaving directory `/home/luna/tmp/testmake3/foo7'
make[1]: Entering directory `/home/luna/tmp/testmake3/foo8'
touch libfoo8.a
make[1]: Leaving directory `/home/luna/tmp/testmake3/foo8'
make: *** No rule to make target `-lfoo8', needed by `app'.  Stop.



Interestingly, it works again if any one of these actions is taken:

  • the lib/ directory is removed
  • the include/ directory is removed
  • one of the entries of SUBDIRS is removed
  • the line with vpath %.h include is removed
  • the line with vpath %.h include is put after the vpath %.a ...
  • in the line with vpath %.a ..., the entry lib is moved to the end of the list


Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #13261:  make_vpath_error.tar.gz added by None (845B - application/x-tgz - Example directory structure, as described above)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by sevanath (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-02 psmith StatusNone Fixed
        Assigned toNone psmith
        Open/ClosedOpen Closed
        Fixed ReleaseNone 4.4
    2007-07-07 None Attached File- Added make_vpath_error.tar.gz, #13261

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code