Connecting Securely to Private AWS Resources by Using SSH Tunnels and Bastion Hosts

The Problem with Publicly Accessible AWS Resources When you first develop infrastructure for a new project, you naturally optimize for rapid development. You want to get something - anything - out the door, and you therefore want to be able to write code and debug issues quickly. Because of that, it's awfully tempting to spin up servers and databases in public subnets so that you can readily connect to them for debugging sessions. It's nice to be able to to do some live code debugging, or to assess the current state of your database. ssh my-user@my-web-server psql -U my-user -h my-database-instance Sure, AWS allows you to do that. You expose SSH on your webservers and you expose your AWS Relational Database Service (RDS) and Redis/ElastiCache servers to the internet for easy access. But should you? Exposing databases and servers publicly yields a pretty obvious security issue: your sensitive data is directly accessible to anyone on the internet. can can You might argue "it's fine; we lock down our VPC security groups so that only people from our office IP can access our EC2, RDS, or ElastiCache instances. Other people on the internet are blocked because of their IP addresses". While that's true - locking down your resources this way is better than nothing - this strategy yields two pitfalls: Allowing anyone from your office IP address to access your AWS resources is still a security issue. Should guests on your wifi, or John the junior developer really have access to production databases? By locking down your resources by public IP, it's pretty hard to discriminate between DevOps users and the rest of your organization - everyone likely uses the same public IP address. Plus, as your organization grows, more attack vectors are created. A single compromised user on your network can wreak havoc on your production systems. Your remote workers are going to struggle to hit resources. If your employees are working from home (which, in this COVID-riddled year, they very likely are), you will either need your employees to maintain a VPN connection to your office, or you will need to maintain a whitelist of all of your employees' home IP addresses. If your employees happen respond to an outage from an airport or coffee shop, they might spend minutes (hours?) manually monkeying with VPC security groups before they can even get access to address the outage. This is a problem that I've had to solve everywhere that I've worked, so I imagine it's a problem others have had to face, too. Let's look at how to do it right, optimizing for both security and accessibility at the same time. Enter AWS Systems Manager Let's suppose you've followed security protocols, and have placed your various AWS resources (EC2, RDS, ElastiCache, etc.) into private subnets and removed public access to them. How can you securely grant access to members on your team (like your senior devs and DevOps) who need them? AWS provides to let you inspect and access your AWS resources - even those residing in private subnets. With Systems Manager, users exchange their AWS credentials for temporary shell access to EC2s. They can get that access both on the CLI, and through the AWS web console. Systems Manager To enable access to your EC2s, simply install the onto the servers that you spin up. Good news - if you're building your servers off of Amazon Linux, macOS, Ubuntu Server, or some Amazon-provided Windows Server images, the SSM agent is already installed for you - you just need to turn on a machine and grant it a specific role defined in AWS's . AWS SSM agent What's the deal with the extraneous "S" in "SSM", you might ask? It's historical - Systems Manager used to be called "Simple Systems Manager" and the extra "S" stuck - just go with it. docs Once you have an EC2 with the SSM agent installed, you can open a shell into your instance with the CLI tool: aws $ aws ssm start-session --target i-0b6c737cc21dc01a9 Starting session with SessionId: treece-0bf6ff366c16d651f sh-4.2$ whoami ssm-user sh-4.2$ cat /etc/system-release Amazon Linux release 2 (Karoo) When you have the SSM agent installed, you don't need to worry about juggling IP whitelists. Your systems management teams can access resources from anywhere (the office, at home, or even a coffee shop) using their AWS credentials. And, without AWS credentials John the junior dev has no access to production databases. Sorry, John! What About Accessing RDS or ElastiCache? With AWS Systems Manager you can remote into EC2s, but what about databases like PostgreSQL/RDS or Redis/ElastiCache? AWS doesn't allow you to directly SSH into the systems running RDS or ElastiCache. Instead, I suggest spinning up a minimal EC2 instance called a in your VPC that you can remote into with Systems Manager. You can remote into the and once there you can access your databases. bastion bastion, Make sure that you assign your bastion a VPC security group, and create ingress rules so that your RDS and ElastiCache security groups allow access from your bastion security group. Once you have all that set up, you can access your databases from the command line on the bastion: $ aws ssm start-session --target i-0b6c737cc21dc01a9 Starting session with SessionId: treece-048a3bf2269ad7caf sh-4.2$ psql -h 10.0.2.88 -U testuser postgres Password user testuser: SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256) Type . postgres=> CREATE TABLE users (id INT, email VARCHAR(40)); CREATE TABLE for "help" for help You might ask, "can't I just spin up a bastion in a public subnet, and SSH into it?" Yes and no. Yes, you can totally do that. No, it's not a good idea - that opens up the same security vulnerabilities that leaving your servers and databases in public subnets does. Tunnels... Tunnels Everywhere At this point, we have options for remoting in to EC2s directly through Systems Manager, and we can access our databases indirectly via a bastion EC2. The last thing I want to look at is leveraging SSH tunnels through the bastion to access database resources as though they're locally accessible. That way, you can connect your favorite database GUI, like to your RDS instances. pgAdmin First, we need to get an SSH public key into our bastion. I'll assume you already have a key in . We'll use the CLI to temporarily load up that key for user into our bastion: $HOME/.ssh/id_rsa.pub aws ssm-user $ aws ec2-instance-connect \ send-ssh-public-key \ --availability-zone us-east-1a \ --instance-id i-0b6c737cc21dc01a9 \ --instance-os-user ssm-user \ --ssh-public-key file:// /.ssh/id_rsa.pub { : , : } $HOME "RequestId" "f8f3db2a-3107-4c0a-ae3d-94e2d7fff620" "Success" true Next, I'm going to modify my laptop's file so that servers starting with will proxy their connections through the command we used earlier. I'll add: $HOME/.ssh/config i- aws ssm start-session host i-* ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'" Now, I can simply to access my instance: ssh ssm-user@INSTANCE-ID $ ssh ssm-user@i-0b6c737cc21dc01a9 Last login: Tue Dec 15 20:27:34 2020 from localhost __| __|_ ) _| ( / Amazon Linux 2 AMI ___|\___|___| https://aws.amazon.com/amazon-linux-2/ 10 package(s) needed security, out of 22 available Run to apply all updates. [ssm-user@ip-10-0-0-71 ~]$ ls file.txt system.log [ssm-user@ip-10-0-0-71 ~]$ for "sudo yum update" Now that I have SSH access to my bastion (even though it's in a private network that's not accessible from the internet!) I can spin up SSH tunnels like I would with any other SSH connection. To create a tunnel to my RDS instance, for example, I can simply run: $ ssh ssm-user@i-0b6c737cc21dc01a9 -NL 5000:10.0.2.88:5432 And then from a separate shell I can access my database "locally" on port 5000: $ psql -h localhost -p 5000 -U testuser postgres Password user testuser: SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off) Type . postgres=> \dt List of relations Schema | Name | Type | Owner --------+-------+-------+---------- public | users | table | testuser (1 row) for "help" for help Cool, huh? Using tunnels through your bastion you can access any resources that reside in your private network without needing to expose them to hooligans on the internet... or John the junior dev. Previously published at https://prismatic.io/blog/connect-to-aws-resources-with-ssh/