bugmake - Bugs: bug #59154, Multiline environment variables...

 
 

bug #59154: Multiline environment variables handled poorly

Submitter:  Dan Kegel <dankegel>
Submitted:  Mon 21 Sep 2020 02:58:39 PM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.2.1 Operating System:  Any
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 02 Nov 2020 11:39:00 AM UTC, comment #16: 

Hi Dan,

using BSD make as an example in general is not a good idea since BSD make implements plenty of non-POSIX compliant "features".

Since this command however works fine with SunPro Make and smake and produces the same output as bmake, it must be seen as a gmake bug that prevents gmake from being used as a make alternative.

Jörg Schilling <schily>
Wed 23 Sep 2020 12:49:35 AM UTC, comment #15: 

Also: I have great respect for the difficulty of getting stuff like this right.  (I recently stared down the maw of commandline issues in response files in meson... https://github.com/mesonbuild/meson/pull/7245 ) so while I may have sounded glib, I didn't mean to imply a nonbreaking fix would be easy.

Anonymous
Wed 23 Sep 2020 12:08:07 AM UTC, comment #14: 

comment #10:

> GNU makefiles will allow the following makefile:
>


> define SOMECOMMAND
> cd foo && echo one
> cd foo && echo two
> endef
>
> all: ; $(SOMECOMMAND)


>
> to print both "one" and "two" when there is a subdirectory "foo" which is empty.
>
> If the variable is passed to a single shell then it will print an error at the second "cd".


Thanks for the example!  I looked briefly in tests/, but didn't immediately see a regression test that verifies this behavior.  I imagine there is one. 
Anyway, so the task for the intrepid complainer would be to
support the BSD usage while not breaking the usage described
above.  If that's impossible, well, that's that.

Anonymous
Tue 22 Sep 2020 11:22:46 PM UTC, comment #13: 

Yes, I said:

> when there is a subdirectory "foo" which is empty.


Your subdirectory "foo" is not empty, it contains a sub-subdirectory "foo" :).


Basically people use define/endef to create recipes which can span multiple lines, and they expect that to work identically to explicitly writing multiple lines in the recipe itself, including resetting the environment on each new line.

I'm not saying that I think this is the best way it could be done, but that's how it works and how it has has always worked.  Maybe if a different choice had been made when define was invented we would have less need for .ONESHELL.  But maybe other things would be harder.

In any event, for sure it will break a lot of things to change it now so it's kind of a non-starter IMO.  I don't see any straightforward way to do something different: even treating environment variables specially is not a solution due to recursive makefiles and the export command.

Paul D. Smith <psmith>
Group administrator
Tue 22 Sep 2020 11:05:33 PM UTC, comment #12: 

comment #10:

> If the variable is passed to a single shell then it will print an error at the second "cd".


Eh?


martind@sirius:~/tmp/make-59154$ bash -c 'cd foo && echo one
cd foo && echo two'
one
two
martind@sirius:~/tmp/make-59154$


Oh, you meant if there wasn't a foo/foo/.  I'm liking that easy way to pass a multi-line script to a single shell without having .ONESHELL applied to everything.  Devil's advocacy aside, perhaps https://www.gnu.org/software/make/manual/make.html#One-Shell would be the best place to explain that variable expansion takes place before the splitting into lines.

Martin Dorey <mdorey>
Tue 22 Sep 2020 05:02:04 PM UTC, comment #11: 

comment #9:

> I don't think this would require Make to parse double quotes.
>
> My guess is it would mean passing along expanded variables to the shell without parsing the variable contents for newlines.
>
> If I have time, maybe I'll have a look at whether my guess is right.


This absolutely will break tons of makefiles, including my company's 20 year old build automation suite and the Linux kernel makefile, for the reason Paul stated. I recommend resolving WONTFIX.

Kyle Rose <squarooticus>
Tue 22 Sep 2020 04:35:07 PM UTC, comment #10: 

I already explained below why it works the way it does... maybe my mic is not on?  Am I on mute?  I do that all the time on my video calls.

BSD make doesn't support multiline variable definitions the way GNU make does, so it makes sense that they do something different when they expand variables containing newlines.

GNU makefiles will allow the following makefile:


define SOMECOMMAND
cd foo && echo one
cd foo && echo two
endef

all: ; $(SOMECOMMAND)


to print both "one" and "two" when there is a subdirectory "foo" which is empty.

If the variable is passed to a single shell then it will print an error at the second "cd".

Paul D. Smith <psmith>
Group administrator
Tue 22 Sep 2020 04:21:21 PM UTC, comment #9: 

I don't think this would require Make to parse double quotes.

My guess is it would mean passing along expanded variables to the shell without parsing the variable contents for newlines.

If I have time, maybe I'll have a look at whether my guess is right.

Anonymous
Tue 22 Sep 2020 04:02:22 PM UTC, comment #8: 

comment #7:

> If the string isn't quoted, the first newline would naturally signal a new command to the shell, wouldn't it?
>
> Got an example of a real-world Makefile that would break?


Nope. But that doesn't mean there isn't one.

There are already existing ways to do what you want. I, for instance, use structures like:

echo -e $(call quote-sh,$(subst $(NL),$(backslash)n,$(mystring)))

to do this (in part because "$(mystring)" doesn't work as expected when $(mystring) contains a double quote).

Finally, does make even parse double quotes in the recipes currently? Doubtful, and I'm not sure I'd want to add that functionality.

Kyle Rose <squarooticus>
Tue 22 Sep 2020 03:56:05 PM UTC, comment #7: 

If the string isn't quoted, the first newline would naturally signal a new command to the shell, wouldn't it?

Got an example of a real-world Makefile that would break?

Anonymous
Tue 22 Sep 2020 03:26:47 PM UTC, comment #6: 

comment #5:

> (To be precise, the line
>        echo "${BLAH}"
> works in all the shells I've tested, and in BSD Make, but not GNU Make.  If BSD Make can get this right, seems like gnu make should be able to, too.)


This could break a ton of software that relies on multiple lines in an environment variable becoming multiple commands in a recipe. If you want this behavior, it should be selectable on a target-by-target basis with a .MULTILINECMD dependency or some other mechanism.

Kyle Rose <squarooticus>
Tue 22 Sep 2020 03:13:39 PM UTC, comment #5: 

(To be precise, the line
       echo "${BLAH}"
works in all the shells I've tested, and in BSD Make, but not GNU Make.  If BSD Make can get this right, seems like gnu make should be able to, too.)

Anonymous
Tue 22 Sep 2020 03:03:27 PM UTC, comment #4: 

"Newlines in variable expansions are treated as real newlines in the rule."

Well, yes, of course.  What I didn't expect is for the command to be truncated at the first newline.  The shell script

BLAH="hi
there"
export BLAH

echo "$BLAH"

properly echoes out both lines of BLAH... yet a Makefile with the same line it it fails.  But only in Gnu make; in BSD make, it works.  I fail to understand how this could be by design.

Anonymous
Tue 22 Sep 2020 01:12:05 AM UTC, comment #3: 

Since, IN THIS CASE, the value should really only be processed once, isn't the solution to defer the expansion to the shell by using
        echo PUB is "$${PUB}"

as the recipe?

Anonymous
Tue 22 Sep 2020 12:45:23 AM UTC, comment #2: 

This is done on purpose.  Newlines in variable expansions are treated as real newlines in the rule.  If they weren't, then multiline definitions would not work at all:


define MYRULE
echo hello
echo world
endef

all: ; $(MYRULE)


I suppose there may be some way to special-case things, I'm not sure.

Paul D. Smith <psmith>
Group administrator
Mon 21 Sep 2020 10:54:53 PM UTC, comment #1: 

I thought this might be related to https://en.wikipedia.org/wiki/Shellshock_(software_bug) but make is exec()ing the wrong command:


martind@sirius:~/tmp/make-59154$ export PUB="$(echo hello; echo world)"
martind@sirius:~/tmp/make-59154$ cat Makefile
foo:; echo PUB is "$(PUB)"
martind@sirius:~/tmp/make-59154$ strace -f make 2>&1 | grep execve
execve("/usr/bin/make", ["make"], [/* 209 vars */]) = 0
[pid 17978] execve("/bin/sh", ["/bin/sh", "-c", "echo PUB is \"hello"], [/* 212 vars */]) = 0
martind@sirius:~/tmp/make-59154$


As is suggested by the make output:


martind@sirius:~/tmp/make-59154$ make
echo PUB is "hello
/bin/sh: 1: Syntax error: Unterminated quoted string
Makefile:1: recipe for target 'foo' failed
make: *** [foo] Error 2
martind@sirius:~/tmp/make-59154$


make -p says PUB has an appropriate-looking definition:


define PUB
hello
world
endef


I got the same error from make-3.81, so not a recent regression, which makes me more surprised that I haven't tripped on this.

Martin Dorey <mdorey>
Mon 21 Sep 2020 02:58:39 PM UTC, original submission:  

I accidentally used a multiline envrionment variable thusly:

$ export PUB="$(base64 < ~/.ssh/id_rsa.pub)"
$ cat Makefile
foo:
       echo PUB is "$(PUB)"

This worked fine on mac where base64 defaults to not wrapping
there.

But on Linux where base64 defaults to wrapping,
this explored multiline variable values.
With gnu make, it fails with
/bin/sh: 1: Syntax error: unterminated quoted string

With bsd make, it succeeds.

The shell is happy to handle multiline environment variables.
Given that, IMHO, Makefiles that contain properly quoted shell references to an environment variable ought to continue working even if that environment variable happens to contain a multiline value.  One other implementation gets it right, so it seems reasonable to enhance gnu make to handle this case as well.

Dan Kegel <dankegel>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by schily (Posted a comment)
  • -email is unavailable- added by squarooticus (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by mdorey (Posted a comment)
  • -email is unavailable- added by dankegel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code