SSH Key rotation! More Saturday morning fun. In some circles ssh key rotation is terrifying and considered a massive headache. In security circles the lack of key rotation is a ripe target to compromise and hack ALL the things. How do we solve this? Well you can write a bash script that is going to go through a list of hosts and then do an ` to each host, but this is not going to invalidate the old keys it is just going to dd the new ones. Only solving half of the issue. None the less for the initial setup of the solution this command is pretty nice so here it is. ssh-copy-id` ssh-copy-id -i ~/.ssh/mykey user@host https://www.ssh.com/ssh/copy-id Soultion: Anisible Ansible is nothing new, it has been around for a while but it is surprising the number of folks that do not use it. Here is some info for ansible . The feature that we are going to focus on is the authorized_key module http://docs.ansible.com/ http://docs.ansible.com/ansible/latest/authorized_key_module.html Lab setup: Needed are, SSH keys already setup on each of the hosts that you are going to change.Ansible running on your laptop, which is where we are going to ssh from.Ansible inventory of the hosts with the shared ssh key. — for my lab Iam using a bunch of physical raspberry pis, because I like them and it makes me feel like a tiny mad scientist having a mini cluster of pi’s running on my desk. There were days that I would rather have a beowulf cluster of machines running but hey you gotta make do. : switch to linux https://www.youtube.com/watch?v=8LsxmQV8AXk Here is an inventory example — simple enough [pi]192.168.1.6 user=pi192.168.1.7 user=pi192.168.1.8 user=pi192.168.1.9 user=pi Ansible is going to generate a new ssh key at a different location than my primary key because right now I am labbing this out and don’t want to break all the things. It is easy enough to use your central location, once we are out of lab stage. A future implementation is going to have a KMS of sorts so I can securely store all the keys but right now lets do this insecurely and we can discuss the best practice later. Vault: https://www.vaultproject.io/docs/secrets/ssh/index.html Great we have an inventory and our ssh key on each of the hosts. Lets say hello world to our hosts to make sure the lab is configured ansible pi -a “/bin/echo hello world” -i hosts -u pi — private-key=”~/.ssh/id_rsa 192.168.1.6 | SUCCESS | rc=0 >>hello world 92.168.1.7 | SUCCESS | rc=0 >>hello world 92.168.1.8 | SUCCESS | rc=0 >>hello world 92.168.1.9 | SUCCESS | rc=0 >>hello world Great it worked, next lets change the key and hope to not break everything.We are going to create an ansible playbook with a few tasks in it. First task is to generate a new ssh-key locally at our new location. name: “Set up authorized_keys for the root user” hosts: pi user: pi tasks: — name: Create new ssh key-pair local_action: command ssh-keygen -t rsa -N “” -q -f ~/test/id_rsa Next we will have a task that takes our new keyfile and pushes it to the hosts, using an exclusive property. The exclusive property is important because without it, we can retain the old keys. If we wanted to have multiple keys deployed we can do multiple steps, the first being the exclusive push and then appending our keys as we go without having exclusive set. — name: Set up authorized_keys for the pi user authorized_key: user=pi key=”{{ item }}” state=present exclusive=yes  with_file: — ~/test/id_rsa.pub Finally we have two steps to move the newly generated keys to our super secret archive, I have not worked this one out yet but cleanup is needed somewhere. Currently it is trying to run multiple times, and its a bug right now. --name: move key-pair  local_action: command mv ~/test/id_rsa ~/test/id_rsa.bak run_once: true — name: move key-pair  local_action: command mv ~/test/id_rsa.pub ~/test/id_rsa.pub.bak run_once: true Putting it all together we get.I realize that this formatting is terrible so here is a gist: https://gist.github.com/ridingintraffic/a89b8a27cb6b9148c5bf7f9dd4294c35 - name: “Set up authorized_keys for the root user” hosts: pi user: pi tasks: — name: Create new ssh key-pair local_action: command ssh-keygen -t rsa -N “” -q -f ~/test/id_rsa   — name: Set up authorized_keys for the pi user authorized_key: user=pi key=”{{ item }}” state=present exclusive=yes  with_file: — ~/test/id_rsa.pub - name: move key-pair  local_action: command mv ~/test/id_rsa ~/test/id_rsa.bak run_once: true - name: move key-pair  local_action: command mv ~/test/id_rsa.pub ~/test/id_rsa.pub.bak run_once: true Lets fire it up and change some keys! $ ansible-playbook -i hosts playbook/ssh-key-push-revoke-old.yml PLAY [Set up authorized_keys for the root user] ******************************** TASK [setup] *******************************************************************ok: [192.168.1.9] TASK [Create new ssh key-pair] *************************************************changed: [192.168.1.9 -> localhost] TASK [Set up authorized_keys for the pi user] **********************************changed: [192.168.1.9] => (item=ssh-rsa xxxxx xxx@xxx) TASK [move key-pair] ***********************************************************changed: [192.168.1.9 -> localhost] TASK [move key-pair] ***********************************************************changed: [192.168.1.9 -> localhost] to retry, use: — limit .retry @playbook/ssh-key-push-revoke-old PLAY RECAP *********************************************************************192.168.1.9 : ok=1 changed=1 unreachable=0 failed=0 It worked! We rotated keys and we are more secure than we were an hour ago!!! There are a ton of links i used diggning through this so I will include them all here: https://derpops.bike/2014/06/07/ssh-key-rotation-with-ansible/ http://vicendominguez.blogspot.kr/2014/05/ansible-generating-pub-key-file-and.html ansible docs: http://docs.ansible.com/ansible/latest/playbooks_intro.html http://docs.ansible.com/ansible/latest/authorized_key_module.html manual ssh-copy-id https://www.ssh.com/ssh/copy-id

BUNCH

Super

Target

Twitter

YouTube

what did i learn?

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

SSH key rotation

Too Long; Didn't Read

Companies Mentioned

mike fettis

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

SSH key rotation

Too Long; Didn't Read

Companies Mentioned

mike fettis

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES