SSH Key rotation! More Saturday morning fun.
In some circles ssh key rotation is terrifying and considered a massive headache. In security circles the lack of key rotation is a ripe target to compromise and hack ALL the things.
How do we solve this? Well you can write a bash script that is going to go through a list of hosts and then do an `ssh-copy-id` to each host, but this is not going to invalidate the old keys it is just going to dd the new ones. Only solving half of the issue. None the less for the initial setup of the solution this command is pretty nice so here it is.
ssh-copy-id -i ~/.ssh/mykey user@host https://www.ssh.com/ssh/copy-id
Soultion: Anisible Ansible is nothing new, it has been around for a while but it is surprising the number of folks that do not use it. Here is some info for ansible http://docs.ansible.com/. The feature that we are going to focus on is the authorized_key module http://docs.ansible.com/ansible/latest/authorized_key_module.html
Lab setup:
Needed are, SSH keys already setup on each of the hosts that you are going to change.Ansible running on your laptop, which is where we are going to ssh from.Ansible inventory of the hosts with the shared ssh key. — for my lab Iam using a bunch of physical raspberry pis, because I like them and it makes me feel like a tiny mad scientist having a mini cluster of pi’s running on my desk. There were days that I would rather have a beowulf cluster of machines running but hey you gotta make do.
https://www.youtube.com/watch?v=8LsxmQV8AXk : switch to linux
Here is an inventory example — simple enough
[pi]192.168.1.6 user=pi192.168.1.7 user=pi192.168.1.8 user=pi192.168.1.9 user=pi
Ansible is going to generate a new ssh key at a different location than my primary key because right now I am labbing this out and don’t want to break all the things. It is easy enough to use your central location, once we are out of lab stage. A future implementation is going to have a KMS of sorts so I can securely store all the keys but right now lets do this insecurely and we can discuss the best practice later. Vault: https://www.vaultproject.io/docs/secrets/ssh/index.html
Great we have an inventory and our ssh key on each of the hosts. Lets say hello world to our hosts to make sure the lab is configured
ansible pi -a “/bin/echo hello world” -i hosts -u pi — private-key=”~/.ssh/id_rsa
192.168.1.6 | SUCCESS | rc=0 >>hello world 92.168.1.7 | SUCCESS | rc=0 >>hello world 92.168.1.8 | SUCCESS | rc=0 >>hello world 92.168.1.9 | SUCCESS | rc=0 >>hello world
Great it worked, next lets change the key and hope to not break everything.We are going to create an ansible playbook with a few tasks in it. First task is to generate a new ssh-key locally at our new location.
name: “Set up authorized_keys for the root user” hosts: pi user: pi
tasks: — name: Create new ssh key-pair local_action: command ssh-keygen -t rsa -N “” -q -f ~/test/id_rsa
Next we will have a task that takes our new keyfile and pushes it to the hosts, using an exclusive property. The exclusive property is important because without it, we can retain the old keys. If we wanted to have multiple keys deployed we can do multiple steps, the first being the exclusive push and then appending our keys as we go without having exclusive set.
— name: Set up authorized_keys for the pi user authorized_key: user=pi key=”{{ item }}” state=present exclusive=yes with_file: — ~/test/id_rsa.pub
Finally we have two steps to move the newly generated keys to our super secret archive, I have not worked this one out yet but cleanup is needed somewhere. Currently it is trying to run multiple times, and its a bug right now.
--name: move key-pair local_action: command mv ~/test/id_rsa ~/test/id_rsa.bak run_once: true
— name: move key-pair local_action: command mv ~/test/id_rsa.pub ~/test/id_rsa.pub.bak run_once: true
Putting it all together we get.I realize that this formatting is terrible so here is a gist: https://gist.github.com/ridingintraffic/a89b8a27cb6b9148c5bf7f9dd4294c35
- name: “Set up authorized_keys for the root user” hosts: pi user: pi
tasks: — name: Create new ssh key-pair local_action: command ssh-keygen -t rsa -N “” -q -f ~/test/id_rsa — name: Set up authorized_keys for the pi user authorized_key: user=pi key=”{{ item }}” state=present exclusive=yes with_file: — ~/test/id_rsa.pub
- name: move key-pair local_action: command mv ~/test/id_rsa ~/test/id_rsa.bak run_once: true
- name: move key-pair local_action: command mv ~/test/id_rsa.pub ~/test/id_rsa.pub.bak run_once: true
Lets fire it up and change some keys!
$ ansible-playbook -i hosts playbook/ssh-key-push-revoke-old.yml
PLAY [Set up authorized_keys for the root user] ********************************
TASK [setup] *******************************************************************ok: [192.168.1.9]
TASK [Create new ssh key-pair] *************************************************changed: [192.168.1.9 -> localhost]
TASK [Set up authorized_keys for the pi user] **********************************changed: [192.168.1.9] => (item=ssh-rsa xxxxx xxx@xxx)
TASK [move key-pair] ***********************************************************changed: [192.168.1.9 -> localhost]
TASK [move key-pair] ***********************************************************changed: [192.168.1.9 -> localhost] to retry, use: — limit @playbook/ssh-key-push-revoke-old.retry
PLAY RECAP *********************************************************************192.168.1.9 : ok=1 changed=1 unreachable=0 failed=0
It worked! We rotated keys and we are more secure than we were an hour ago!!!
There are a ton of links i used diggning through this so I will include them all here:https://derpops.bike/2014/06/07/ssh-key-rotation-with-ansible/http://vicendominguez.blogspot.kr/2014/05/ansible-generating-pub-key-file-and.html
ansible docs:http://docs.ansible.com/ansible/latest/playbooks_intro.htmlhttp://docs.ansible.com/ansible/latest/authorized_key_module.html
manual ssh-copy-idhttps://www.ssh.com/ssh/copy-id