Three months ago we stopped using ENV files as the default export option in the Doppler CLI. This change led to a number of benefits including supporting multi-line variables and a deterministic schema. Before going too deep on the technical choices we made, let's first go over what ENV files are and how they're used. This article originally appeared on doppler.com and was written by CEO and founder, Brian Vallelunga. What are ENV files? ENV files are plain text files that store variables and secrets that you would not want hardcoded in your codebase. These variables could be a port number or a database url, and may change depending on where your code is deployed. For example, when developing locally you may use port 3000, but when deployed to Heroku your application will need to use the port it's dynamically assigned. An example ENV file when developing locally could look something like this with the schema of : KEY=VALUE PORT= DATABASE_URL= AWS_KEY= BUGSNAG_API_KEY= INTERCOM_KEY= MAILGUN_KEY= NEW_RELIC_LICENSE_KEY= NEW_RELIC_LOG= SALESFORCE_TOKEN= STRIPE_KEY= "3000" "psql://postgres@localhost/db_name" "rfiunb34fu93n49iufgn3o2o0ini2ef" "33dasdk34bsf23f35871as0fa27" "ndeiovnkofrnij30490398u39RIBF39IOENIOWF3ENWKEWENVIO" "fdjnbvjoenvlmswsdnjoelfv" "b583d4eab955872122843a067faca9db5d4202af" "stdout" "feojkvndfjkovnskfn3eni32one" "sk_test_ejidsfnvoiuebnfvoi3enjkdNCJSB" Having the file is not enough though, you would also need a tool like to parse the file and inject those variables into the environment. foreman Benefits So what are some of the benefits of using an ENV file? Well these files live on your local machine which means you do not need a network connection to fetch your secrets. The schema is also quite simple so it's easy go into a file and add a new variable. Lastly, everyone knows this format so there is a ton of support by the open source community for parsers and managers. Downsides From our time working with ENV files and adding support for various use cases, we have found that there isn't a standardized schema all libraries use. For example, take a look at the sample ENV file below: PORT = 3030 Notice that this ENV file that contains a spaces between the and the . If we were to use bash to inject the variable into the environment with the command we would get an error. KEY VALUE source source spaces.env # >> bash: PORT: command not found Now if we use another tool like we would see it parse without an error. This is because each library is deciding the schema of a ENV file instead of strictly following an open standard. These inconsistencies causes other problems to arise as well, such as parsing multi-line secrets. In this example a variable uses encoded newlines through : foreman \n CERT= "-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE\nlw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk\nqDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr\nR3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr\n9S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG\nBO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP\nUvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm\n4KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO\n2UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY\ndqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf\nzq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf\nSidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE\nCCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB\nVLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6\ndBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU\nOOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL\ndX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva\n/YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh\n2splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K\n2iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV\nZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM\nUyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc\nO/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F\n6kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa\nNxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=\N-----END RSA PRIVATE KEY-----\n" This newline is treated differently depending on which tool you use. Using the bash command the in the string would not be converted to newline characters. On the other hand, using Python's most popular ENV library will convert the to newlines automatically. Now let's look at the inverse: source \n dotenv \n CERT= "-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE
lw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk
qDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr
R3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr
9S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG
BO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP
UvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm
4KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO
2UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY
dqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf
zq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf
SidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE
CCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB
VLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6
dBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU
OOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL
dX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva
/YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh
2splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K
2iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV
ZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM
Uyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc
O/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F
6kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa
NxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=
-----END RSA PRIVATE KEY-----" In this example, we have the same cert but with newline characters. Surprisingly the bash command respects the newline character but the Node library does not. More interestingly is how the Node library breaks. It parses the value as and disregards all the other lines. I also find it funny that because it is a multi-line variable the quote detection algorithm broke, which can be seen by the first character being a quotation mark. If the quote detection algorithm was working correctly, you would see the value be stripped of it's quotation marks at the beginning and end of the string. source dotenv "-----BEGIN RSA PRIVATE KEY----- Alternatives After realizing ENV files are problematic, we started looking at alternatives. We wanted something that has a universally accepted schema with no room for interpretation and a large community for support. The two data formats we focused on were and . YAML JSON Let's start off with YAML. One of the primary advantages of YAML is that it is incredibly easy to read and write. It uses indentation and nesting as a way to designate structure. Let's look at a sample YAML file: PORT: DATABASE_URL: AWS_KEY: CERT: |
  -----BEGIN RSA PRIVATE KEY-----
  MIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE
  lw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk
  qDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr
  R3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG
  BO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP
  UvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY
  dqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf
  zq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf
  SidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+ yj+vdpOmE
  CCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB
  VLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6
  dBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU
  OOq+ AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL
  dX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva
  /YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV
  ZWB5zElKCAWwVJkQCZc+ dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM
  Uyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc
  O/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+ HZlb0U2nnTFCTNfjhcD2+Xa
  NxB4RWiMLTgeDmGICV4U+ qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=
  -----END RSA PRIVATE KEY----- 3000 "psql://postgres@localhost/db_name" "rfiunb34fu93n49iufgn3o2o0ini2ef" 9 4 2 317 3 2 2 4 6 6030 1 At first glance, the syntax looks very similar to the ENV format but when we look closer we see subtle differences. The YAML syntax uses colons instead of equal signs and has native support for multi-line strings. The one downside when using multi-line secrets is that indentation really matters. The fabled debates of how many spaces equals a tab come into play. With developers each having their own style, it can make YAML files prone to parsing errors when sharing. JSON on the other hand has a wildly different syntax then YAML. Wikipedia has an accurate description of the language: JavaScript Object Notation is an open standard file format, and data interchange format, that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and array data types (or any other serializable value). Let's take a look at the same config of variables in JSON format: { : , : , : , : } "PORT" "3000" "DATABASE_URL" "psql://postgres@localhost/db_name" "AWS_KEY" "rfiunb34fu93n49iufgn3o2o0ini2ef" "CERT" "-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEA6ONkkK5eT0wUIjV4CyeO5yQ4AMmCTUyfahKq3gOto4UVhtHE\nlw6GnZwbvRUSwpqGi1X8iTo1GKjcYBVNvRf6Hw5zk9wGTImwNBAlEF7K1aYnelMk\nqDLJ7T0vHAVEvAq2Wz24SljMWgdv9d83KOvuTjZE04H7YlBS4w3OeRu7D2+kgkAr\nR3fqCNEUOvafikwqThHV27xSMaj7uvvm+eMv9ztNb8VauSnZ9zPXtLOPSNy7HGQr\n9S3rqwg7Hif9yLQ2iWVa9R6ACc2I9oK27Olq8AvyHsIz4gktBqLpV3rfBc5muReG\nBO+kdsSpCxpQBQ1W4gU8gTi7Qgr9+bEeaN2bfwIDAQABAoIBAG+J2PRiTtDzwwDP\nUvskqxCRDDF0UW/sLr2Cy0shv9v9NV4owVsHnfmGdtKMcTu6/o1lVVn0AtIYrdNm\n4KCcBzMwnLJIQswNddK5mMbKX6MLvQSdJYVZLdTt5M4qx8y35La2TLlu5hCIV1sO\n2UBEHxJec4BJVLi1d70/M5BVc7Xj/ImqPgHtJhNv5gaej3s/vS1j5YmtCHwGnwbY\ndqVLiY9NgHKO3EOFa0vJplxwR0sIj0WumtkLLwjAfEmt0ivZ3D1fJ9hCFrfpJYwf\nzq9Nv1RL0Jry4SfnWTpXKPlF4N+ateXkNhrZILRg8xmOJSQduYt0wo2KxkAbgxtf\nSidoWIECgYEA+/Ggv0LsqxwmsiR991BA2aurYlJwzEFHL/YUc/j+317yj+vdpOmE\nCCV3mAa9tAgMf+BJvQS1RGS2bnnVe5CcjuoEJ1gQ3LdU9LA1H14880TjMsuxEKkB\nVLHkhiS1yG4lo01H8Aml2EAn1Hz84BazubxMy8vWu6xqm6wT0LIxuI8CgYEA7KM6\ndBtrkWSbj1lpLR8zeLhQkcQP94biLcrH0xEONpphNdTy2DW/Ne6qQWQ9y171iMvU\nOOq+3AcyNf/hZxhRAcTN5Qb3qGUqZn4tRXuVzhKd3CQ5Ijiq7EAfSUI+NBKGPChL\ndX7unhIgJVgcuuo/qg6J5vOV+FGGpm5Zbu9zBhECgYBBcAruYnWSI+exEWVeXQva\n/YmwKfV+N95DiMjbLmsUnVanJv4UnUpby096vxV6szR76kd8vsJOF1KC80YNqAvh\n2splZaxLh5qbS0Eg+pseHGBeiyVcTGk6FFJkvRgyDNndxm7O29KljlRKDoSnt33K\n2iugKzuE102BTXqAFChx5QKBgEyJeuWE3OTYwou54o/KkK5SBxUuce+ge9VNyhXV\nZWB5zElKCAWwVJkQCZc+4dG+c/H74zdJjdPCrBXVHkVnEwRccC/MchvQJMejtebM\nUyak1NQYDzanV3k0QCpEt7PF7g7VBZsKJAmSWT1a42f9Tfwl2aqOTIpVbBS2ikyc\nO/rRAoGARmMBi0jfi1m3DpRt35QyCWJXd8YNGxsaB1cc/NorBPOX5cIP3YGn1b6F\n6kS0HEz1SOpENczi+C5hJiyldVIkek9sjoW7+6030HZlb0U2nnTFCTNfjhcD2+Xa\nNxB4RWiMLTgeDmGICV4U+1qIFLyiuZxabLxw0q5O2kkyGGKlpeQ=\n-----END RSA PRIVATE KEY-----\n" One of the main beauties of JSON is that it is strictly enforced and there is only one way of accomplishing each task. For example, when we look at the variable , we can see the value is wrapped in quotes to state it is a string. Unlike YAML, which will guess if the line should be cast to a string or number, JSON only has one way of notating strings and numbers. One other stark difference between YAML and JSON is how they handle multi-line variables. In JSON we can see it uses the encoded newline characters which we think is a safer bet than trusting humans with indentation. PORT \n Choosing JSON We ended up going with JSON because it has a far stricter schema and has strong native support in most languages. After making the switch, we saw our customers' issues with parsing downloaded config files flat line. Since the creates a fallback of your secrets by default when running your application, we decided to go one step further by enabling encryption by default. Doppler CLI We strongly believe that you are always going to be worse off having secrets on disk, but if you are going to, it is imperative that they are encrypted.

Fetch

Heroku

It's Free

Read My Stories

Too Long; Didn't Read

Publish Your First Brand Story for FREE. Click Here.

What are ENV Files? - Pros, Cons, Alternatives

Too Long; Didn't Read

Companies Mentioned

@doppler

RELATED STORIES