This post originally appeared on the . , Hacker Noon’s weekly sponsor, is currently offering a 90 day free trial — email friends@segment and mention Hacker Noon to redeem. Segment blog Segment Segment receives billions of events from our customers daily and has grown in to dozens of accounts. Expanding in to many more accounts was necessary in order to best align with our , but it comes at a large complexity cost. In order to continue scaling gracefully we are investing in building tooling for employees to use with many accounts, and centrally managing employee access to AWS with terraform and our identity provider. AWS GDPR and security initiatives Segment began in a single AWS account and last year finished our move to a dev, stage, prod, and “ops” accounts. For the past few months we’ve been growing at about one new AWS account every week or two, and plan to continue this expansion in to per-team and per-system accounts. Having many “micro-accounts” provides superior security isolation between systems, and reliability benefits by limiting the blast radius of AWS rate-limits. When Segment had only a few accounts, employees would log in to the AWS “ops” account using their email, password, and 2FA token. Employees would then connect to the role in the dev, stage, and prod accounts using the api. ops-admin AssumeRole Segment now has a few dozen AWS accounts and plans to continue adding more! In order to organize this expansion we needed a mechanism to control our accounts, which accounts employees have access to, each employee’s permissions in each account. and We also using AWS API keys when we don’t absolutely have to so . Instead, employees only access AWS through our identity provider. Today we have zero employees with AWS keys and there is no future need for employees to have a personal AWS key. This is a massive security win! hate we moved to a system where no employees have any AWS keys Designing a scalable IAM architecture Segment uses Okta as an identity provider, and consulted their for managing multiple AWS accounts, but improved it with a minor change for better employee experience. The integration guide recommends connecting the identity provider to each AWS account but this breaks AWS’ built in support for account switching and was more complicated to audit which teams had access to which roles. integration guide Instead, employees use our identity provider to connect to our “ops” account and then use the simple token service API to connect to each account they have access to. Using our identity provider, each team is assigned to a different role in our hub account, and each team role has access to different roles in each account. This is the classic “hub-and-spoke” architecture. assume-role In order to make maintaining our hub-and-spoke architecture simple, we built a terraform module for creating a role in our spoke accounts, and a separate terraform module for creating a role in our hub account. Both modules simply create a role and attach a policy ARN to it, which is part of the module’s input. The difference between the modules are their trust relationships. The hub role module allows access from our identity provider while the spoke module only allow access from the hub account. Below is module we use for allowing access to a hub role from our Identity provider. only resource "aws_iam_role" "okta-role" { name = "${var.name}" assume_role_policy = <<EOF{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "${var.idp_arn}" }, "Action": "sts:AssumeRoleWithSAML", "Condition": { "StringEquals": { "SAML:aud": "https://signin.aws.amazon.com/saml" } } } ]}EOF}resource "aws_iam_role_policy_attachment" "okta-attach" { policy_arn = "${var.policy_arn}" role = "${aws_iam_role.okta-role.name}"} In order to provide each team with granular access to only the resources the teams need we create a role for each team in the hub account using our hub role terraform module. These roles contain IAM policies for in to other accounts but it is also possible to give granular access in our hub role too. mostly sts:AssumeRole One concrete and simple example of a granular policy is our Financial Planning and Analysis team’s role, . Our FP&A team has access to billing information and information about our reserved capacity. who keeps close watch on our AWS spend only module "fpa" { source = "git@github.com:segmentio/access//modules/okta-role" name = "fpa" idp_arn = "${module.idp.idp_arn}" policy_arn = "arn:aws:iam::aws:policy/job-function/Billing"}resource "aws_iam_policy" "fpa_reserved_policy" { name = "fpa_reserved_policy" description = "FP&A team needs ability to describe our reserved instances." policy = <<EOF{ "Version": "2012-10-17", "Statement": [ { "Action": [ "ec2:GetHostReservationPurchasePreview", "ec2:DescribeReservedInstancesModifications", "ec2:DescribeReservedInstances", "ec2:DescribeHostReservations", "ec2:DescribeReservedInstancesListings", "ec2:GetReservedInstancesExchangeQuote", "ec2:DescribeReservedInstancesOfferings", "ec2:DescribeHostReservationOfferings", "ec2:CreateReservedInstancesListing" ], "Effect": "Allow", "Resource": "*" } ]}EOF}resource "aws_iam_role_policy_attachment" "fpa_reserved_attach" { role = "fpa" policy_arn = "${aws_iam_policy.fpa_reserved_policy.arn}"} The FP&A team does not have access to our spoke accounts, though. One team that needs full access to much of our infrastructure and all of our accounts is our Foundation and Reliability team, who participate in our on-call rotation. We provide both a role, and an role to our foundation team in all of our accounts. ReadOnly Administrator module "foundation" { source = "git@github.com:segmentio/access//modules/okta-role" name = "Foundation" idp_arn = "${module.idp.idp_arn}" policy_arn = "${aws_iam_policy.foundation-policy.arn}"}resource "aws_iam_policy" "foundation-policy" { name = "foundation-policy" description = "A policy for foundation to access all AWS accounts, as on-call" policy = <<EOF{ "Version": "2012-10-17", "Statement": [ { "Action": [ "sts:AssumeRole" ], "Effect": "Allow", "Resource": [ "arn:aws:iam::${var.ops_account}:role/AdministratorAccess", "arn:aws:iam::${var.ci_account}:role/AdministratorAccess", "arn:aws:iam::${var.dns_account}:role/AdministratorAccess", "arn:aws:iam::${var.customerdata_account}:role/AdministratorAccess", "arn:aws:iam::${var.customerdata_account}:role/ReadOnly", "arn:aws:iam::${var.dns_account}:role/ReadOnly", "arn:aws:iam::${var.ci_account}:role/ReadOnly", "arn:aws:iam::${var.ops_account}:role/ReadOnly", ... ] } ]}EOF} After per-team roles are created for each team in the hub account, employees are assigned to groups that represent their teams in Okta, and each team can then be assigned to their associated role in the hub account. Okta allows each group to be assigned different IAM roles in the hub account, and using their UI we can assign the FP&A team to our “Amazon Web Services” app, and restrict their access to the fpa role that we created for them in the hub account. After building this, we needed the tooling to provide our employees with an amazing experience. Even though this system is far more secure, we wanted it to be just as usable and efficient as our setup with only a handful of AWS accounts. engineering Maintaining usability with aws-okta One great thing about our old IAM setup was each employee with AWS access could use AWS APIs from their local computer using . Each employee had their IAM user credentials securely stored in their laptop’s keychain. However, accessing AWS entirely through Okta is a massive breaking change for our old workflows. aws-vault Our tooling team took up the challenge and created a (near) drop in replacement for aws-vault which our engineering team used extensively, called . aws-okta is now . aws-okta open-source and available on github The quality of is the principal reason that Segment engineers were able to smoothly have their AWS credentials revoked. Employees are able to execute commands using the permissions and roles they are granted, exactly like they did when using aws-vault. aws-okta $ aws-okta exec hub -- aws s3 ls s3://<some-bucket> 2018/02/08 15:40:22 Opening keychain /Users/ejcx/Library/Keychains/aws-okta.keychainINFO[0004] Sending push notification... There is a lot of new complexity handled with aws-okta that is is not able to be handled in aws-vault. While aws-vault uses IAM user credentials to run commands, aws-okta uses your Okta password (stored in your keychain) to authenticate with Okta, waits for a response to a push notification for 2FA, and finally provides AWS with a SAML assertion to retrieve temporary credentials. In order to authenticate with Okta, aws-okta needs to know your Okta “application id”. We took the liberty of extending the ini file to add in the necessary id. ~/.aws/config [okta]aws_saml_url = home/amazon_aws/uE2R4ro0Gat9VHg8xM5Y/111 When Segment had only a few AWS accounts and the role, Segment engineers all shared the same . Once each team had access to different accounts and systems, we needed a better system to manage each team’s . Our system also needed a way to update the access that employees had quickly, when new accounts and roles are created. ops-admin ~/.aws/config ~/.aws/config We decided to integrate this solution closely with prior art that Segment had built. Each team’s config is stored in a git repo that has our company dotfiles in it. Each team can initialize their aws config by using our internal tool called , which is a tool to share helpful commands between employees. robo $ SEGMENT_TEAM=foundation robo config.aws✔️ : Your old aws config has been backed up in /tmp/awsconfig-318c16acc2b25bed2eb699e611462744✔️ : Your aws config was successfully updated.$ shasum ~/.aws/config c2734b78e470c51a26d8c98e178d4ec2ed1b1b06 /Users/ejcx/.aws/config $ SEGMENT_TEAM=platform robo aws.config✔️ : Your old aws config has been backed up in /tmp/awsconfig-d5688401634de0e8b2f48b11377d0749✔️ : Your aws config was successfully updated. $ shasum ~/.aws/config 283053d6f5a23ca79f16c69856df340b631d3cdf /Users/ejcx/.aws/config This was only possible to add because all Segment engineers already had an environment variable called called , which denotes the team the engineer is a part of. Running will clone the dotfiles repo, save the old , and initialize the most recent config for their team. SEGMENT_TEAM robo aws.config ~/.aws/config were the primary way that engineers navigated our environment when we utilized fewer accounts. When we got rid of the role, the engineers sign-in bookmarks stopped working. Additionally, AWS bookmarks only support up to five different AssumeRole targets and we now have many more than five accounts. AWS bookmarks ops-admin In order to support having many more accounts, we mostly abandoned bookmarks and instead ensured that supports engineers who needed to switch AWS accounts often. Our previous use of meant many of us were familiar with the command. We found that adding a login command to aws-okta helped engineers who switched accounts often. aws-okta aws-vault aws-vault login After responding to the Duo push notification aws-okta will open a browser and log in to the specified role in only a couple of seconds. This feature is supported by the AWS feature, but feels more like magic when using it. It makes logging in a breeze. Custom Federated Login Beyond 100 accounts We expect to be near 50 AWS accounts by the end of this year. The security of having an account be completely closed by default, and the reliability benefits of having isolated per-account rate-limits are compelling. This system we have built is plenty robust and usable enough to scale our AWS usage to many hundreds of AWS accounts and many more engineering teams. from a security perspective, and this alone is a compelling enough reason to integrate your identity provider with your AWS hub account. Deleting all employee AWS keys was extremely satisfying This post originally appeared on the . , Hacker Noon’s weekly sponsor, is currently offering a 90 day free trial — email friends@segment and mention Hacker Noon to redeem. Segment blog Segment
