9,878 reads

SSH key rotation

by mike fettisAugust 19th, 2017

Too Long; Didn't Read

SSH Key rotation! More Saturday morning fun.

Companies Mentioned

SSH Key rotation! More Saturday morning fun.

In some circles ssh key rotation is terrifying and considered a massive headache. In security circles the lack of key rotation is a ripe target to compromise and hack ALL the things.

How do we solve this? Well you can write a bash script that is going to go through a list of hosts and then do an `ssh-copy-id` to each host, but this is not going to invalidate the old keys it is just going to dd the new ones. Only solving half of the issue. None the less for the initial setup of the solution this command is pretty nice so here it is.

ssh-copy-id -i ~/.ssh/mykey user@host https://www.ssh.com/ssh/copy-id

Soultion: Anisible Ansible is nothing new, it has been around for a while but it is surprising the number of folks that do not use it. Here is some info for ansible http://docs.ansible.com/. The feature that we are going to focus on is the authorized_key module http://docs.ansible.com/ansible/latest/authorized_key_module.html

Lab setup:

Needed are, SSH keys already setup on each of the hosts that you are going to change.Ansible running on your laptop, which is where we are going to ssh from.Ansible inventory of the hosts with the shared ssh key. — for my lab Iam using a bunch of physical raspberry pis, because I like them and it makes me feel like a tiny mad scientist having a mini cluster of pi’s running on my desk. There were days that I would rather have a beowulf cluster of machines running but hey you gotta make do.

https://www.youtube.com/watch?v=8LsxmQV8AXk : switch to linux

Here is an inventory example — simple enough

[pi]192.168.1.6 user=pi192.168.1.7 user=pi192.168.1.8 user=pi192.168.1.9 user=pi

Ansible is going to generate a new ssh key at a different location than my primary key because right now I am labbing this out and don’t want to break all the things. It is easy enough to use your central location, once we are out of lab stage. A future implementation is going to have a KMS of sorts so I can securely store all the keys but right now lets do this insecurely and we can discuss the best practice later. Vault: https://www.vaultproject.io/docs/secrets/ssh/index.html

Great we have an inventory and our ssh key on each of the hosts. Lets say hello world to our hosts to make sure the lab is configured

ansible pi -a “/bin/echo hello world” -i hosts -u pi — private-key=”~/.ssh/id_rsa

192.168.1.6 | SUCCESS | rc=0 >>hello world 92.168.1.7 | SUCCESS | rc=0 >>hello world 92.168.1.8 | SUCCESS | rc=0 >>hello world 92.168.1.9 | SUCCESS | rc=0 >>hello world

Great it worked, next lets change the key and hope to not break everything.We are going to create an ansible playbook with a few tasks in it. First task is to generate a new ssh-key locally at our new location.

name: “Set up authorized_keys for the root user” hosts: pi user: pi

tasks: — name: Create new ssh key-pair local_action: command ssh-keygen -t rsa -N “” -q -f ~/test/id_rsa

Next we will have a task that takes our new keyfile and pushes it to the hosts, using an exclusive property. The exclusive property is important because without it, we can retain the old keys. If we wanted to have multiple keys deployed we can do multiple steps, the first being the exclusive push and then appending our keys as we go without having exclusive set.

— name: Set up authorized_keys for the pi user authorized_key: user=pi key=”{{ item }}” state=present exclusive=yes with_file: — ~/test/id_rsa.pub

Finally we have two steps to move the newly generated keys to our super secret archive, I have not worked this one out yet but cleanup is needed somewhere. Currently it is trying to run multiple times, and its a bug right now.

--name: move key-pair local_action: command mv ~/test/id_rsa ~/test/id_rsa.bak run_once: true

— name: move key-pair local_action: command mv ~/test/id_rsa.pub ~/test/id_rsa.pub.bak run_once: true

Putting it all together we get.I realize that this formatting is terrible so here is a gist: https://gist.github.com/ridingintraffic/a89b8a27cb6b9148c5bf7f9dd4294c35

- name: “Set up authorized_keys for the root user” hosts: pi user: pi

tasks: — name: Create new ssh key-pair local_action: command ssh-keygen -t rsa -N “” -q -f ~/test/id_rsa — name: Set up authorized_keys for the pi user authorized_key: user=pi key=”{{ item }}” state=present exclusive=yes with_file: — ~/test/id_rsa.pub

- name: move key-pair local_action: command mv ~/test/id_rsa ~/test/id_rsa.bak run_once: true

- name: move key-pair local_action: command mv ~/test/id_rsa.pub ~/test/id_rsa.pub.bak run_once: true

Lets fire it up and change some keys!

$ ansible-playbook -i hosts playbook/ssh-key-push-revoke-old.yml

PLAY [Set up authorized_keys for the root user] ********************************

TASK [setup] *******************************************************************ok: [192.168.1.9]

TASK [Create new ssh key-pair] *************************************************changed: [192.168.1.9 -> localhost]

TASK [Set up authorized_keys for the pi user] **********************************changed: [192.168.1.9] => (item=ssh-rsa xxxxx xxx@xxx)

TASK [move key-pair] ***********************************************************changed: [192.168.1.9 -> localhost]

TASK [move key-pair] ***********************************************************changed: [192.168.1.9 -> localhost] to retry, use: — limit @playbook/ssh-key-push-revoke-old.retry