bugmake - Bugs: bug #62595, Add ability to read .env files...

 
 

bug #62595: Add ability to read .env files into make

Submitter:  Chigozirim Chukwu <firstairbender>
Submitted:  Wed 08 Jun 2022 08:43:10 AM UTC
   
 
Severity:  3 - Normal Item Group:  Enhancement
Status:  None Privacy:  Public
Assigned to:  None Open/Closed:  Open
Component Version:  4.3 Operating System:  None
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 13 Jun 2022 09:27:58 PM UTC, comment #8: 


> Of course you can also use the re-exec feature of make to write a rule to generate the include file then you have the full power of a recipe.

I like the idea, especially because of the re-exec feature. However, my goal was to avoid having to make this external to make. Also given that the rule for recipe is not likely to change from project to project, I would like to suggest an alternative (and possibly simpler) abstraction:

What if we have a special target rule, which has dependencies on any number of ".env" files:


.ENVFILES: .env production.env


This will have the job of parsing the env files and including them into make.

> I'm also not sure I agree that the right behaviour is to always export every variable in the file. But these are details:...

I suggested the always export option because that's usually the way I've seen .env files being used.

For example, when working with docker-compose yaml files, docker-compose reads a .env file and any other included yaml files can also see the variables. See the syntax rules for .env

The same is true for most people who use .env files with shell scripts. Some will do something like this to automatically export and load the variables into the current process:


set -a; source .env; set +a


Pipenv which is used to manage a python project dependencies will also usually load the .env file into the current process, and every program that is started sees the environments.

In any case, I don't think lack of automatic export is a deal breaker. If the current plan is to create an external program, or to use the special target, then decision to export or not can be made later rather than now.

Chigozirim Chukwu <firstairbender>
Sat 11 Jun 2022 02:36:32 PM UTC, comment #7: 

Thanks for the pointer to the spec, that's very helpful.  I agree that this would be possible to implement.

I'm not interested in a flag like --envfile, that doesn't sound very useful.  I would anticipate that most build systems would want to always include the content of the file, and not require the user to add an option to the command line.

I'm also not sure I agree that the right behavior is to always export every variable in the file.  But these are details: the first thing is to decide whether this needs to be added at all.

I was thinking about alternatives to adding new syntax, and the problem is that there's not a very straightforward way in GNU make today to evaluate the output of a program as a makefile.

We can run a program using $(shell ...), but the results are modified by removing newlines and so this is not useful.

The only way to do it is to run $(shell ...) and have it write to a temporary file, then include the temporary file.  Of course you can also use the re-exec feature of make to write a rule to generate the include file then you have the full power of a recipe.

Is that sufficient?  It would be something like:


include env.mk

env.mk: env.env
        env2make < $< > $@


for some putative env2make conversion tool.

An alternative would be to invent some new type of function which would either preserve newlines in the result, which could then be passed to the eval function, or create an alternative to include which would run a program and "include" the output, or something like that.

Paul D. Smith <psmith>
Group administrator
Fri 10 Jun 2022 02:03:13 PM UTC, comment #6: 

 > ⚠ External Email
 >
 > Follow-up Comment #4, bug #62595 (project make):
 >
 > No, no, that's not something one should expect from a .env file.
 >
 > I said that some dotenv parsers take it a step further to do all that fancy
 > interpolation, but really .env files are just files containing a bunch of
 > key=value lines, no functions or shell parsing necessary. It's just a
 > configuration file. Variable interpolation is a nice-to-have feature of most
 > dotenv parsers, not a requirement.

<snip>

You're asking for a specialized parser, then.


 > So the goal is not to write a parser, but more of a translator that converts
 > .env syntax to make syntax.
 >

The write a translator of arbitrary shell syntax, you need a parser.

Rather than update Make for this special case, I think it would be
better to write a small shell wrapper for Make that reads each line of
the 'env' file, and either writes a post-processed file that Make can
read, or just export the variables into the current shell before
invoking make.

Here's a simple example to show you the concept.


#!/bin/bash

FILE=$(pwd)/example.env;

while read LINE ; do
    IFS='=' read -r key value <<<${LINE};
    eval ${LINE};
    echo "input      : ${LINE}";
    echo "key/value  : '${key}' / '${value}'";
    echo "evaluated  : ${!key}";
    echo "Make output: ${key}=${!key}";
    echo "";
done <${FILE};

Escaping special characters (like '#') before exporting to make is not
implemented, but should be trivial to do before evaluating 'LINE' into
the environment.

Anonymous
Fri 10 Jun 2022 06:29:41 AM UTC, comment #5: 

I'd never heard of these env file things, so appreciated the well-written posts that those links point to.  Why would this feature be better in Make rather than in a wrapper?  If you wanted, for some reason that I'm struggling to guess, to keep "make" as the interface, then I think it'd be straightforward to write a Makefile that would spot that an expected environment variable wasn't available, then started a shell that sourced the environment file, exported all its variables and reran make.

Martin Dorey <mdorey>
Fri 10 Jun 2022 05:46:19 AM UTC, comment #4: 

No, no, that's not something one should expect from a .env file.

I said that some dotenv parsers take it a step further to do all that fancy interpolation, but really .env files are just files containing a bunch of key=value lines, no functions or shell parsing necessary. It's just a configuration file. Variable interpolation is a nice-to-have feature of most dotenv parsers, not a requirement.




Unfortulately I don't think there is a definite spec I can point to, but I found this one which explains the format. And another one which talks about the origin

To bring this back on topic, I believe "make" already has the capacity to do what most dotenv parsers are doing. As you can see from the stackexchange page I linked earlier, make can already read some simple .env files when included using the "include" directive, but for others it just needs to be guided a little. For example:


TEST_VAR='8X46Yj*aek3GQeiW7#bK!Eg@#6vjV'


should be translated into


export TEST_VAR := 8X46Yj*aek3GQeiW7\#bK!Eg@\#6vjV


And something like:


EMAIL=${USER}@example.org


is to be translated to


export EMAIL := $(USER)@example.org


So the goal is not to write a parser, but more of a translator that converts .env syntax to make syntax.

This feature can be exposed either via a directive within a Makefile (includeenv) or a flag (--envfile) to make.

Chigozirim Chukwu <firstairbender>
Thu 09 Jun 2022 07:28:01 PM UTC, comment #3: 

Right, so that's just a shell script that doesn't happen to do anything except set variables.

Your comment 'The format of a .env file is simply "key=value"' is not the case, as you yourself point out when you say "the single quotes surrounding the value are to be ignored".

That means that the value needs to be parsed/interpreted somehow.  And if it's really a shell script then the value could contain any valid shell syntax: it might be quoted with single quotes, or double quotes, and the contents could be escaped with "\" and how that is handled depends on the type of quoting.  The value could contain a simple variable that needs to be expanded, or a variable of the form ${...}.  It could use fancy shell variable syntax like ${foo:-bar} or any of the other things.

It could even contain backticks or $(....) to run subcommands and retrieve the output.

This is what I meant by, I don't really want to put an entire shell parser into make.

If there's some limit to the actual syntax of these ".env files", where there's some standard specification that reduces their scope from "any valid shell variable assignment" to something more tractable, can you point to such a spec?

Paul D. Smith <psmith>
Group administrator
Wed 08 Jun 2022 04:15:00 PM UTC, comment #2: 

Hi Paul

.env files are commonly used for specifying secrets for an application or a build.

Please see this question to get an idea of how people are currently trying to use .env files in their Makefile:
https://unix.stackexchange.com/questions/235223/makefile-include-env-file

The format of a .env file is simply "key=value". From the make.env project I linked to, here is what a typical file may look like

APP_PASSWORD='8oyy!r#vNpRy2TT'
FOO_BAR='4dabfs#a%d73w$Z2TBN4!nYD4Y$TW'
TEST_VAR='8X46Yj*aek3GQeiW7#bK!Eg@#6vjV'
ANOTHER_VAR='"hello world'

---

In the above case, the single quotes surrounding the value are to be ignored, but everything else is to be taken literally. Some .env parsers take it a step further and also expand environment variables within the .env file, so values that are surrounded by double quotes and have $ symbol inside are expanded and the result is what is stored as part of the value. ex. user="$USER" expands to user="username".

The solution I settled on for the make.env project is to convert each of those lines to "export KEY:='$(value VALUE)'", then load them into make using the --eval flag. This was the only way I found that reliably loads the value into make without make attempting to interpret the value.

I hope that clears things up

Chigozirim Chukwu <firstairbender>
Wed 08 Jun 2022 12:50:32 PM UTC, comment #1: 

I don't know what an ".env" file is.

Do you mean, a shell script?  It's not possible to embed a shell parser into make.  Or, I guess it would be possible but it's not something I think is a good idea.

If you don't mean a shell script, is there some definition of exactly what syntax an ".env" file supports and where such things are used?

Paul D. Smith <psmith>
Group administrator
Wed 08 Jun 2022 08:43:10 AM UTC, original submission:  

I would like the ability to allow make to read .env files. I kinda already got started here with a python wrapper around make, called `make.env`: https://github.com/smac89/make.env

I believe this is something many people will benefit from, and it can be exposed via a flag of some sort.

I don't mind working on it, but I just need to know if it's something that has been considered before and discarded, or if there are any concerns with giving `make` the ability to read environment files

Chigozirim Chukwu <firstairbender>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by mdorey (Posted a comment)
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by firstairbender (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-01-08 psmith SummaryAdd ability to read .env files into make and Add ability to read .env files into make

    Back to the top

    Powered by Savane 3.13-c373.
    Corresponding source code