An account for every color of the rainbow
Having seperate AWS accounts is the latests Dev Ops craze to grace our toolkit (not including kubernetes of course) and with good reason! Seperate AWS accounts to manage environments, or even products is a great idea. If done right itâs something else that will help you sleep at night. The benefits are:
- Security! Separation of concerns across AWS accounts means completely different set of IAM Users, Roles & API Keys. Now your Production account will finally, not be exposed to every single one of your Developers.
- Development Speed, Developer ease! Developers will only be scared of touching things in the Production account. Meaning less bloat in your other accounts, whilst being able to tear other environments apart. This is underrated, developers will be able to play & build more! Make Dev Ops For Everyone!
- Billing. Youâll be able to stop paying stax 2% of your AWS bill, for dissecting your bills. You can use inbuilt consolidated billing, to definitively understand the financial aspect of every environment.
Iâm going to show you how Avoss tackled their multi account setup, with a large focus on deployments, security & developer access.
But be warned! The silver bullet doesnât exist and multi accounts comes with its own trade offs! Learning IAM is going to take time, migrating to multi accounts will be a headache & managing everything is going to take careful consideration.
Side note* Running a multi AWS account setup would be very hard & tedious, if you are not using some sort of Infrastructure as Code solution. In this blog post & at Avoss we are using CloudFormation. If you want to start investigating Infrastructure as Code, I covered a comparison in my last blog post CloudFormation Vs Terraform.
IAM & Deployments
Your first issue is going to be IAM. Who has access to what? How does everything get deployed? How can we reduce access! ARHH DONâT COMMIT THE ACCESS KEYS (yes we have all done it)! Itâs overwhelming.
The Goals for our Multi environment accounts are:
- There should be no loose API keys for your Production Account.
- Developers should still be able to have full read access to production, but changes should take place through your Continuous Integration.
We can achieve both. Or at the very least only two API keys, if your continuous integration is running outside of AWS infrastructure.
Lets get started
I know this is going to make you cringe, but first we are going to have to create a User or Role manually. Unless there is something I donât know about there has to be some manual User or Role creation, to create that first initial User or Role that will kick of your deployments. The only other scenario is using your root credentials, and that is definitely a no go â ď¸. This will have to be done over each environment.
The manually created User or Role, is going to be used to create the rest of the IAM Roles & Permissions for all of your deployments in each environment. This gives us a few advantages, all of your IAM roles & permissions, will be easily visible & subject to code review. And coupled with continuous integration, the roles & permissions will roll across all off your accounts nicely.
But this may sound scary, giving one manually created User or Role permission to execute all the permissions & roles for your whole account! Wow that is scary, but we can take the appropriate steps to make sure these set of permissions are locked down as tight as possible. Secondly if you are using a continuous integration solution inside of AWS, you can have an EC2 instance assume the deployment Role & have the zero API keys I promised for your Production account. Making your deployments incredibly secure.
Below is the manually created User or Roleâs attached policy.
You will notice that
- It can only execute a specific stack, that you specifically name
- It can only create a specific User & specific Roles that you specify
This means that even if this Role or User was compromised that attacker would need to know your exact stack name, roles and/or IAM User. If you wanted to be extremely hardcore, you could go as far as to salt these names with a bit of random junk. Now thats secure!
Okay now we have a user, that can create more roles, where are we going? To a place where API keys donât exist & we have maximum security. Nirvana.
This scenario repeats over every AWS account you wish to use for deployments
At the heart of this diagram is the ${env}-roles-stack.yml which you can checkout here. This is where your rest of your IAM roles & permissions come from.
This ${env}-roles-stack.yaml creates a specific deployment User or Role & specific roles, that can only execute against specific stacks. This is very similar to our manually created User or Role, except each future deployment role also specifies what exactly can assume it, whether itâs CloudFormation or the Deployment User or Role. This works incredibly well, when your infrastructure is decomposed.
You will notice the relationship between the Role & the User is two ways. The Role is explicitly told it can only be assumed by the User. Whilst we tell the User it is allowed to assume the role & also allowed to execute the specific CloudFormation stack, that requires the role. The alternative to assuming the role from the User, is to pass the role to the CloudFormation stack & specifying on the Role only CloudFormation is allowed to assume that role.
This means that if an attacker compromises our deployment CI at this point they again they would need the exact name of a Role, with the exact stack name & whether that role was being passed to a certain service or assumed.
Assuming the deployment role, when passing the role isnât available
Passing roles to CloudFormation via command line when available
This is great & a very secure deployment, but what happened to the multiple AWS accounts!
The deployment to Multiple AWS Accounts or Environments
The deployment of each one of these pipelines is replicated over your AWS environments. For each AWS account their is a set of deployment Roles & a shared CloudFormation template. Your continuous integration should handle the input parameters of deployments like account details, environment name, region etc. And there is no reason why this diagram canât involve as many accounts as you wish! This is easy to scale, as per each environment there is only one deployment user.
Another caveat of this deployment. If you are passing roles to CloudFormation stacks to be assumed when executed, you MUST make sure that the roles exist for any future modifications of the said stacks. Including deletion, if it does not exist CloudFormation doesnât have a role to assume to execute the stack!
Tell me again why all this effort is worth it?
Itâs very secure! Firstly your first line of defence must be compromised whether that is the manually created User/Role, or the deployment User/Role. Then the attacker, must figure out the permissions are CloudFormation related and can only execute for certain stacks, against certain roles. And if they happen to know the stack name, & role the deployment role should be locked down so the blast radius only affects that piece of infrastructure.
Now if you used only roles during this process, you will have literally zero API keys the deployment process. The only attack could from someone taking over your compromised EC2 instance that inherits the role. This is unlikely, especially if you are taking advantage of Virtual Private Networks and protecting your instance!
Can I access production though?
Yes! You can access it like normal through the web console & command line! And you guessed it, you grant access by [switching roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_permissions-to-switch.html) in your production account from your development account.
In your production account you create a role with all the permissions your heart so desires (but keep it strict! this is production remember!). Then you must edit the Trust Relationship to allow your Development account to assume the role. In your Development account you create and attach a policy to your desired Users or Groups. This policy will only have the function of being able to assume the role, in your production account.
Then you can actually fill this handy link out, that will prompt to a switch role screen https://signin.aws.amazon.com/switchrole?roleName=<role_name>&account=<target_account>Â . After you have switched roles, you will be able to browse the UI like normal, granted you have the permissions.
Console UI after switching roles
This is seamless & itâs easy to switch back and forth. Except for itâs easier, you didnât even have to type another password in! Better yet, you can use the CLI to assume-role and use the command line like normal too!
Consolidated Billing
Last but not least, for those who havenât seen it. If you set up your accounts as an organisation, you get free access to consolidated billing. Consolidated billing offers a per account price split, which gives you a clear indication of your financial AWS health! Not only that but the cost explorer, also understands each one of your linked accounts & you can apply this to your filters, rather than just having to rely on tags.
An example of what consolidated billing looks like
Thanks for reading! If you got this far, be sure to leave a clap!