Ditch your SSH keys and enable AWS SSM!

If you manage AWS for an organization, big or small, chances are you have several Secure Shell (SSH) keys laying around you hardly use, OR WORSE, you don’t recall the account the key was made for. SSH key management is a rabbit hole in itself and most people understand the security concerns that arise with improper SSH key hygiene. Luckily for us, there is a way to bid farewell to the the cumbersome practice of using SSH to remote into an EC2 instance. Allow me to introduce you the AWS service, Systems Manager (SSM).

I will teach you the following in this guide:

• Identify SSM Remote Session Manager requirements-including for an enterprise
• Enable Remote Session Manager for all EC2 instances
• Enable Remote Session Manager logging
• Lock down Remote Session Manager through IAM User permissions 🔐
• Debugging Remote Session Manager

Enable SSM Remote Session Manager

The AWS managed service, SSM, comes with a neat feature called Session Manager. Session Manager allows us to connect into an instance and get a shell session through the usage of HTTPS TLS1.2/ port 443, without having to use SSH keys. It’s important to understand that this is NOT an SSH connection but rather an HTTPS connection. The Session Manager allows us to use a terminal session from our web browser directly OR by using the AWS CLI. It’s really that easy…..assuming you have everything configured correctly.

Here are the core requirements to get SSM’s Remote Session Manager to work:

• The SSM Agent is installed on the instance(s). By default the SSM agent is installed on Amazon Linux, Amazon Linux2, Ubuntu Server 16.04, Ubuntu Server 18.04, and all Windows Server AMIs. The agent can also be installed on your instance (if lacking) or be pre-baked into your AMI through the usage of Packer.
• The EC2 instance requires an IAM Instance Profile — an instance profile role of the type EC2. To create an IAM instance profile for SSM follow this link.
• The IAM Instance Profile requires proper SSM permissions. This is a step that often causes confusion or that is missed. In order for the SSM agent to communicate with the AWS SSM API endpoints, it needs the proper IAM permissions. AWS provides a default SSM policy for your convenience named, amazonEC2RoleforSSM . That being said, I recommend you make your own policy based of the default provided policy that does not include the following permission s3:* (more on this later).

When you have the three requirements checked of for your instance(s), you are ready to take off the training wheels and start using SSM’s Remote Session. Ta da!

What you should see if all requirements have been met!

What a session looks like in the browser

However, as everything else in life there are less obvious requirements than the three I mentioned above. Additionally, if you’re an enterprise user with Direct Connect — things can get a bit trickier. Let’s dive into the “less common” requirements.

• The VPC has access to the internet. This is not a luxury everyone has available. If you are in an organization that leverages Direct Connect with your AWS Infrastructure, chances are that you may have VPCs that have intentionally been created without an Internet Gateway (IGW) and NAT Gateways — thus your only access to the internet is through your corporate data center. This makes things more difficult, but luckily for us there is a solution — I’ll talk more about this later.
• The AWS CLI requires an SSM plugin. That’s right, if you want to use the remote session function from your workstation then you need to install an additional plugin to the AWS CLI. It’s a simple install and all directions can be found here.

Addressing the VPC Internet challenge

If you find yourself in the situation I described above — a VPC without internet connectivity, you need to enable six VPC Endpoints. Simple.

If you’re not familiar with VPC endpoints, it’s a way of routing destined internet traffic to AWS public API endpoints through the AWS internal networking infrastructure. Traditionally, if you needed to make a call to the EC2 service API, the REST call would be routed through a NAT, or IGW and go through the public internet provided by an internet service provider (ISP) and then hit the public facing EC2 API. However, with VPC Endpoints, the REST call goes to the API endpoint but- gets routed from the VPC directly to the public facing API though AWS network backbone, thus never going out to the public internet and staying internal to the AWS infrastructure the entire time. Pretty cool right? This is what allows a VPC without internet access to be able to leverage SSM’s Remote Session manager.

Note: When creating the endpoint -MAKE SURE YOU ENABLE PRIVATE DNS NAME!

• com.amazonaws.region.ssm
• com.amazonaws._region._ssmmessages
• com.amazonaws._region._ec2
• com.amazonaws._region._ec2messages
• com.amazonaws._region._logs
• com.amazonaws.region. s3

These endpoints will allow you to leverage SSM’s Remote Session Manager and all it’s functionality

If you’re wondering why logs and s3 is part of the endpoint list, just know they are needed if logging capabilities are enabled. The logs endpoint is for the CloudWatch logs. I’m gonna emphasize this again, make sure you enable Private DNS name while creating the endpoints. The reason for that is because we are lacking internet access and and we need our VPC Endpoints to have DNS names that can be resolved without internet access.

Enable Logging

Enabling logging in SSM’s Remote Session Manager is straight forward. Simply navigate to SSM Remote Session screen, click on the preferences tab, and click ‘edit’. This will take you to a panel with options for enabling KMS, S3 logging and CloudWatch logging. Let’s enable both S3 and CloudWatch.

Note: The KMS option is if you desire to encrypt the SSM connection data in addition to the default TLS 1.2 encryption. The connection encryption would include encryption that leverages your custom KMS key or the AWS managed KMS key for SSM. Both the user starting the Session Manager session and the instance that the session connects to must have permission to use the specified KMS key. Be aware that custom KMS keys will incur addition cost.

S3

I created a private S3 bucket named karls-ssm-log. I then simply selected the bucket in the selection option (see below), and then specified a folder/prefix named logs. I also selected for the logs to be encrypted.

I want to highlight here, it is important that you allow your IAM Instance Role attached to the instance proper access to the S3 bucket. The default SSM policy provided by AWS, amazonEC2RoleforSSM has s3:* so that would cover all scenarios, however I don’t recommend providing this much access but rather restrict the policy to have access only to the SSM designated log bucket you create.

Also, if you choose to encrypt the logs, make sure that the IAM instance role has the proper KMS access. If your bucket is configured at the global level to encrypt everything using a custom KMS key, the IAM instance role will need access to use the specified custom KMS key.

Cloudwatch

I created an unencrypted Cloudwatch log group named ssm-log. I simply navigate to the SSM and follow the same steps mentioned above to get to the Session Manager preferences tab. The screenshot below illustrates how to enable the Cloudwatch logs.

Note: If you want your Cloudwatch logs to be encrypted, ensure the Cloudwatch log group is created with enabled encryption — otherwise writing logs to the log group will fail.

Simply check the CloudWatch checkbox and select the desired log-group

Access logs

In order to access SSM Remote Session Manager logs, you can either go directly to the S3 bucket and/or the Cloudwatch log group specified. However, there is an easier option. In the Session Manager selection screen, the middle tab is for viewing history. With logging enabled, the Session History screen will contain a shortcut to the respective session’s log (see below).

Logs are easily consolidated for your convenience

Note: Logs initiated from the aws cli will not have the user’s/role’s name, instead it will contain the prefix botocore-session-#######. However, using CloudTrail, you can use the botocore session id to hunt down the temporary access key granted through STS, and from there work backwards and identify what users/role was provided with the temporary access key. It’s not an easy nor a fun process but it’s certainly possible to identify the end user 🔎.

User Permissions

One of the most common concerns administrators have with enabling SSM’s Remote System Manager is the ability for everyone to remote into any instance that meets the SSM requirements. After all, the barrier of entry is now greatly reduced and simplified. Thankfully we can leverage IAM user permission to control who may use SSM’s Remote Session Manager to remote into an instance. We have the ability to lock down the access by either leveraging tags or go as granular as instanceID .

Take a peek at the following IAM policy example. This example policy restricts a user’s ability to use the Remote Session manager through the usage of a tag with the key value pair of ssm:enabled . If the instance has the tag with the correct value then the user can start a remote session otherwise, access is denied. The policy also ensures that the user is only allowed to terminate their own respective sessions thus avoiding the ability to troll other users. Visit this link for more example policies.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:StartSession"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:instance/*"
            ],
            "Condition": {
                "StringLike": {
                    "ssm:resourceTag/ssm": [
                        "enabled"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:TerminateSession"
            ],
            "Resource": [
                "arn:aws:ssm:*:*:session/${aws:username}-*"
            ]
        }
    ]
}

Debugging SSM

Sometimes you might run into situations where the Remote Session Manager is not working. Debugging SSM can be a challenge, especially if you cannot remote into the instance. I recommend following debugging steps:

1. Ensure the AMI has the agent installed
2. Verify an EC2 instance role is attached
3. Verify the EC2 instance role has the proper SSM permissions
4. Restart the instance

The four steps above will solve majority of all SSM related problems. However, there are trickier situations.

Q: I enabled logging but I can’t connect to my instance anymore?

A: This can either be a network issue or lack of permissions. Ensure your VPC has internet access, if it does not then enable the VPC endpoints required for SSM, and that Private DNS is enabled. If S3 logging was enabled with encryption then ensure both bucket and IAM instance role has access to the KMS key, if custom KMS key is used in the S3 bucket.

Q: I do not have logging enabled and still can’t connect to my instance?

A: Assuming you have covered the basics; agent installed, instance role, IAM permissions. Take a peek at your network security group and network ACLs. Ensure you allow inbound and outbound tcp 443 traffic within your VPC. In other words you trust traffic from your VPC CIDR, not to be confused with traffic external to the VPC.

Q: Where can I view the logs generated from the agent?

A: You can view the SSM Agent logs on Linux instances in the following locations

• /var/log/amazon/ssm/amazon-ssm-agent.log
• /var/log/amazon/ssm/errors.log

In Windows

• %PROGRAMFILES%\Amazon\SSM\Logs\amazon-ssm-agent.log
• %PROGRAMFILES%\Amazon\SSM\Logs\errors.log

Q: Is it possible to enable more debugging?

A: You bet — simply modify the seelog.xml.template ( I recommend you make a copy).

In Linux the file can be found

• /etc/amazon/ssm/seelog.xml.template

In Windows

• %PROGRAMFILES%\Amazon\SSM\seelog.xml.template

Change the value of minlevel=info to minlevel=debug

<seelog type="adaptive" mininterval="2000000" maxinterval="100000000" critmsgcount="500" minlevel="debug">

Q: Any other useful links I should know about?

A: Yes, ask questions in the AWS SSM forum!

Ready to ditch your SSH keys?

I hope you find this guide helpful in getting started with SSM’s Remote Session Manager. AWS does a great job of providing documentation for SSM but there really isn't a holistic job aid on how to set this functionality up, not to mention the few “gotchas”, hence the reason for this article. Now that you have all this knowledge available to you, start using SSM today and let SSH keys be a thing of the past.