bugmake - Bugs: bug #55560, .SECONDEXPANSION and eval in...

 
 

bug #55560: .SECONDEXPANSION and eval in target-specific prerequisite lists

Submitter:  None
Submitted:  Thu 24 Jan 2019 04:33:32 PM UTC
   
 
Severity:  3 - Normal Item Group:  None
Status:  Not A Bug Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Component Version:  4.2.1 Operating System:  POSIX-Based
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Mon 13 May 2019 08:04:11 PM UTC, comment #4: 

I'm afraid I can't really understand the issue here.  I recommend that you discuss it on the -email is unavailable- mailing list until the issue becomes clear enough to file a bug about (or it's explained clearly).  It's much simpler and faster to use the mailing list for things like this.  I'm going to close this but we can re-open if/when the problem being reported becomes clear.

In particular you say it worked the first time but not the next time: what was the difference there?  We need a repro case (as simple as you can get it).

Some notes on your example:

First, you need to escape the : because make doesn't skip over escaped variable references.  If the : appears in a normal variable reference like $(foo:.c=.o) make knows this : can't be a separator, but if the variable reference is escaped like $$(foo:.c=.o) make thinks this is just some normal text and will treat this : as if it were a rule separator.  Consider a rule like: some$$file:prereq ; echo hi

Second, this line:


all: $$(eval LIBRARY_OBJECTS \:= $$(OBJECTS)) $$(LIBRARY)


does NOT define a target-specific variable LIBRARY_OBJECTS (maybe you know that but it wasn't clear from your question).  This defines a global variable LIBRARY_OBJECTS, once the prerequisites of all are expanded (but not before.).

Paul D. Smith <psmith>
Group administrator
Mon 28 Jan 2019 04:53:58 AM UTC, comment #3: 

Please note: I made an error in the last post, the following line should be removed:

$(WORKDIR)/%/$(LIBRARY) ;


Quinn Mikelson <quinn_mikelson>
Mon 28 Jan 2019 04:49:25 AM UTC, comment #2: 

I noticed that running my library builds in parallel seemed to break things more often that running them serially. Looking at the previous examples there is indeed some ambiguity in the prerequisites parsing order. I found some old email threads discussing static patterns with patterned subdirectories to accesse similarly patterned variables and I'm fairly certain I've come up with a solution:

=== Makefile ===

TARGET := libfoo

all: SOURCES := foo.cc bar.cc
tests: SOURCES := foo.cc bar.cc test.cc
include library.mk


=== library.mk ===

WORKDIR := $(CURDIR)/.work
LIBRARY := $(LIBDIR)/$(TARGET).so
# This macro uses the stem of a pattern to export target-specific variables
OBJECT_PATTERN = $(eval $*-OBJECTS := $(SOURCES:%.cc=%.o))

LIBRARY_TARGETS := all tests
PHONY: $(LIBRARY_TARGETS)
.SECONDEXPANSION: $(LIBRARY_TARGETS)
# Static pattern forces the preprocessor to evaluate LIBRARY_TARGETS first, then LIBRARY
$(LIBRARY_TARGETS): %: $$(OBJECT_PATTERN) $(WORKDIR)/%/$(LIBRARY) ;

# We have to write the LIBRARY as a pattern to get our preprocessor trick to work correctly.
## Here we nest it in a subdirectory and have it access the variable we exported earlier.
$(WORKDIR)/%/$(LIBRARY): $$(%-OBJECTS)
    $(LD) $^ -o $@


Writing the phony library targets as static pattern rules and modifying the Library target to contain a pattern-specific variable seems to work. Is my understanding of "forcing" the pattern parsing order correct here? Anyway, I've tested this on a number of different source trees and all of my usage tests have been working. Coincidentally I have noticed builds are marginally slower, so this technique may imply a performance penalty.

If there's anyone out there that can chime in, I'd be very grateful.

Quinn Mikelson <quinn_mikelson>
Sat 26 Jan 2019 10:55:05 PM UTC, comment #1: 

I think I have an understanding of what is happening here. Even though make is processing a separate Target, it's still already performed a first pass on all of the mentioned target'prerequisite lists s when entering the second expansion phase. This is evident when attempting to generate new recipes from within the context of a prerequisite list as you'll get errors about generating new prerequisites.

Although not a bug, this seems like a heavy restriction. It would be great to be able to leverage the target-specific variable system to generate prerequisites to avoid bloating the global namespaces. Are there any metaprogramming techniques that I could use to get around this limitation?

Quinn Mikelson <quinn_mikelson>
Thu 24 Jan 2019 04:33:32 PM UTC, original submission:  

Host

GNU Make 4.1
x86_64-pc-linux-gnu
Package Version: 4.1-9.1ubuntu1

Background

I am attempting to export a target-specific variable in the context of a prerequisite list; I would like to use this variable to update the prerequisite list of another target mentioned in the prerequisite list (which should satifsy the  requirements for target-specific variable inheritance).
 
The techniques in the articles Deferred Simple Variable Expansion and Target-specific and Pattern-specific GNU Make macros got me thinking that this might be possible. I believe the code below is a valid solution, but I'm seeing inconsistent behavior and I'd like to understand why.

Build Files


Makefile


TARGET := libfoo

all: SOURCES := foo.cc bar.cc bin.cc

include library.mk



library.mk


SOURCES :=
WORKDIR := $(CURDIR)/.work-lib
LIBRARY := $(WORKDIR)/$(TARGET).so
OBJECTS = $(patsubst %,$(WORKDIR)/%.o,$(basename $(SOURCES)))

.SECONDEXPANSION:
.PHONY: all
all: $$(eval LIBRARY_OBJECTS \:= $$(OBJECTS)) $$(LIBRARY)
        @echo Built $(LIBRARY) with target $@

$(LIBRARY): $$(LIBRARY_OBJECTS)
        $(LD) $(LIBRARY_OBJECTS) -o $@

$(WORKDIR)/%.o: %.cc
$(WORKDIR)/%.o: %.cc %.d | create_dirtree
        $(CC) $*.cc -o $@


Behavior

I was actually pleasantly surprised; on the first library I ran this on everything compiled and linked. Prerequisites were properly marked as out-of-date when a source changed, etc.

The second library I attempted this on didn't work. LIBRARY_OBJECTS was not getting picked up in the $(LIBRARY) prerequisite list, but were available for linking which I was promptly notified with a whole bunch of g++ No such file or directory errors.

I understand this is a very specific case, and I'd like to get to the bottom of this issue, is there any more information I can provide?

Followup Question
  1. Why must the colon be escaped in the following snippet (multiple target errors otherwise)?


.SECONDEXPANSION:
all: $$(eval LIBRARY_OBJECTS \:= $$(OBJECTS))


Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by quinn_mikelson (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-05-13 psmith StatusNone Not A Bug
        Open/ClosedOpen Closed

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code