A-Z Of DevOps: Managing Multiple Environments With The Help Of These Tools

In most DevOps settings you’ll find that there are multiple environments in the pipeline. You might have conditions that change the environment based on which branch was merged or when a branch is tagged for release. There are a number of reasons you want to have more than just a production environment, the biggest reason being testing.

Keep in mind that every organization does things slightly different. You might see more environments than the ones covered here or they might not have the same names. The important part is to know what purpose each environment serves. They will have different access levels to services and varying public side access.

Environments you might see

Development

The development environment is like a step up from your local environment. It’s where you can deploy little changes to see if they work somewhere other than locally. Your code doesn’t have to work perfectly to deploy to this environment. You might need to test how a new endpoint integrates with your application and you need to get around CORS issues.

You might work on a large feature with other developers in this environment when you all are testing approaches to a feature or bug. It’s like an advanced sandbox that gives you slightly more access to other services than your local environment might allow.

Staging

In most cases, this is where your code goes before it gets shipped to production. You should have gotten code reviews and tried some individual testing to make sure that all of the functionality is there and there haven’t been any regressions. The changes that get deployed to staging should be working with little to no issues.

This is where your integration tests run and any third party services get the secrets they need. Some companies will set up completely separate services for staging because this is the environment closest to the production environment that we can get.

Once you have your code on staging, you can verify that all of the data is loading like you expect and that you are getting the correct responses from services. This is a good place to run chaos engineering tests to check how sturdy your system is with the resources it has available. The available resources the staging server has is usually lower than what production has so keep that in mind for performance testing.

QA

You’ll see the QA (quality assurance) environments in different forms. They are usually set up for software testing engineers to systematically hunt bugs and issues with the app from a user perspective. This is where a lot of edge-case scenarios come up.

Many times you’ll see multiple QA environments and they all serve different purposes. One might be specifically set up for edge-case scenarios, another might run automated UI tests, and another might have newly released features that are being experimented with.

These environments take on a number of different names, so you might not see a pipeline with specific commands to deploy to QA. Get to know the structure of these environments early if they are used on any projects you work on.

Production

This is the final environment for your code. Once you’ve reached this server, all of your changes are live for customers to interact with. There’s not much testing you can do here without effecting user experience and at this point you have done everything you can to test that your code is strong.

Production will likely be the most resource intensive because uptime and performance are typical metrics for how well an application works for users. There are very few cases you should ever deploy directly to production. One specific case that you might would be getting a hotfix out for a major bug that’s effecting users.

Otherwise it’s always best to send your code changes to the other environments that are in place. Most CI/CD pipelines deploy the code to multiple environments in parallel so that no one is waiting for every environment to deploy before they can start testing.

Handle data across environments

Keeping parity across all of the environments is important for them to remain useful. The main area that you need consistency in is the data. If you can get the data from production and obfuscate it, that will be the best data to develop against. It’s exactly what users are seeing, but with all of the sensitive information changed or filtered out.

It also helps to have data that recreates edge-cases if possible. If an issue has come up in the QA environment then it’s worth the time to make the data for that issue readily available. This makes sure that there aren’t any regressions getting through certain environments and not the others.

When you have a plan for how to handle your data across these environments, it makes development and testing easier which will help speed up the deploy process. There is a little upfront time investment because copying and modifying production data can take a while. If you can, try to import just enough data to get your app working like you expect.

Maintaining each environment

Since you know what each environment is there for, there are just a few more things you need to consider. In the long term maintenance of these environments, make sure you know who has access. While you might be using censored data, there still might be processes and code that not everyone needs access to.

Another thing that comes up is how resource allocation is handled on pre-production environments. It’s unlikely that the development and QA environments will have the same amount of server resources as production. This introduces a different set of limitations that you don’t have to deal with in production.

When you’re trying to figure out performance issues or you’re trying to monitor services, be aware of your environment. That might be what you’re trying to debug instead of the actual issue. Keep the cache clear, delete any unused buckets, and watch for anything else that could bog your running code down.

Deploying to the different environments in the CD phase

When you want to deploy to all of these environments, it’ll probably be a part of the continuous delivery process. Just so you’ll have an idea of what this might look like in your pipeline, here’s an example.

import conducto as co

def cicd() -> co.Serial:
    image = co.Image("node:current-alpine", copy_dir=".")

    cra_node = co.Exec("npx create-react-app random-demo")
    installation_node = co.Exec("cd random-demo; npm i; CI=true; npm test")
    build_node = co.Exec("npm build")
    staging_node = co.Exec("echo sent build artifact to staging on an AWS server")
    qa_node = co.Exec("echo sent build artifact to QA on a Heroku server")
    prod_node = co.Exec("echo sent build artifact to prod on a Netlify server")
    start_node = co.Exec("npm start")

    pipeline = co.Serial(image=image, same_container=co.SameContainer.NEW)

    pipeline["Make CRA..."] = cra_node
    pipeline["Install dependencies..."] = installation_node
    pipeline["Build project..."] = build_node
    pipeline["Deploy to envs..."] = co.Parallel()
    pipeline["Deploy to envs..."]["Staging"] = staging_node
    pipeline["Deploy to envs..."]["QA"] = qa_node
    pipeline["Deploy to prod..."] = prod_node
    pipeline["Start project..."] = start_node

    return pipeline

if __name__ == "__main__":
  co.main(default=cicd)

Other thoughts

It might seem like a lot to keep track of, but once you understand what’s expected from each environment it’s just a part of the process. As long as credentials and permissions are kept up to date and expectations are set, working with multiple environments will help you get better code to users.

You’ll be able to get feedback from different people faster and you’ll be able to track down the bugs and security holes more consistently. It makes all of the testing and integrations go smoother when it’s really time to go to production.

Make sure you follow me on Twitter because I post about stuff like this and other tech topics all the time!

If you’re wondering which tool you should check out first, try Conducto for your CI/CD pipeline. It’s pretty easy to get up and running and it’s even easier to debug while it’s running live.

Previously published at https://www.flippedcoding.com/articles/2020/sept/how-to-manage-multiple-envs-devops/