In the past few weeks, I have been doing a lot of different tests and deployments on kubernetes. Therefore, I had to create and destroy Kuberentes clusters like a lot (some days — a few times in an hour). I really needed a fresh cluster due to specific things I was testing, so simply deleting all pods/services/secrets/etc to make the cluster “fresh like” wouldn’t help.
Another requirement (again due to some of the specific tools I was testing) was to have the most “real like cluster” as possible so all local solutions (Minikube, vagrant, etc) wouldn’t work either.
At the beginning, I used DigitalOcean Kubernetes because it’s super simple to deploy and as soon the cluster is up I could just download kubectl config with a click of a button. But there were two problems with it:
1. It takes sooooo much time… around 10 minutes to deploy a cluster — if I had to deploy&destroy it few times per day — it becomes a lot of wasted time
2. Downloading and loading kubectl config — while it’s simple — it’s still a manual process
So I decided to create something which would allow me to deploy&destroy kubernetes cluster in the cloud faster and easier… and I end up with a simple bash script which creates virtual machines on Google Cloud, deploys 4-node Kuberentes cluster (1 master + 3 workers), downloads kubectl config and loads it into my system… and it does all of it in under 60 seconds! So from nothing (not even having VMs) to being able to do kubectl apply -f any_deployment.yaml in under 1 minute. Let’s talk about how was it possible.
Requirements:
One of the important factors for me was to make the solution as portable as possible. So I tried to make as fewer tools required as possible (so no terraform, no ansible, no whatever which require installation and configuration). That’s why I wrote it in bash and the only dependency I kept is to have gcloud installed and configured (preferably set default region and project).
VMs:
We start with VMs. Creating VMs normally takes around 45–60 seconds. For example on DigitalOcean VMs are up (meaning ping starts responding) after around 40 seconds, but You need another ~15 seconds for all the systems services to be up (and most importantly — ssh server being able to accept connections). So first we need to make that step faster… like 2 times faster :)
Well… we can achieve that by using a slimmer OS image. That’s why currently I stick to Google Cloud because they provide Ubuntu minimal (which is less than 200MB in size). In the meantime, I tried many other minimalistic distributions but some of them were missing necessary kernel modules, some even tho were slim — they boot up quite long.
Creating and booting up Ubuntu minimal VMs on Google Cloud takes around 30 seconds (from the gcloud API call till ssh server ready) so we got our first step.
Now we have another 30 seconds for…
Kubernetes:
How to deploy kubernetes cluster in around 30 seconds… there is one answer — k3s. For those who haven’t heard about k3s — similar to Minimal OS images it’s simply slim k8s solution:
By using k3s we don’t actually have to do much to get kubernetes up and running because k3s does everything for you. So my script only downloads k3s installer and executes it.
Connect the pieces:
So we found a way to boot up VMs in less than 30 seconds by using slim OS images, we have k3s.io which allows us to run kubernetes in around 20 seconds and now we just need to connect all the pieces together, so — prepare gcloud commands (to deploy VMs) in the script, download and execute k3s installer on master node (simply curl piped to sh), then, get the token (cat command over ssh) generated by k3s which is used for adding nodes to the cluster and download and execute (with token) k3s on worker nodes.
The only real challenge was to get kubectl config generated properly — public IP address on google VMs is not visible on the VMs so when k3s generates certificates and then config — it’s not valid for accessing the cluster from the outside. But after a lot of searching, I found the parameter “ — tls-san=” which allows providing additional IP addresses for certificate generation.
Then, if k3s is deployed on all nodes, workers are registered properly with the master — our cluster is ready. The only thing left is to simply download the kubectl config (using scp from master node). Everything takes 55–58 seconds usually (Google Cloud isn’t really consistent — sometimes it takes a bit more than 60 seconds, but it doesn’t happen often).
So as you can see there is nothing extraordinary about the solution — just a few gcloud and curl commands glued together in one bash script. But it does the job and does it quickly.
Room for improvement?
Definitely! First of all — currently it’s hardcoded to have 4 node cluster (1 master + 3 workers) — to make it configurable it’s rather simple — I just never needed bigger cluster for the tests I did.
Secondly — kubectl config is now either being only downloaded (so you can pass it yourself to kubectl command as a parameter) or it overwrites your existing kubectl config (which works just fine for myself — since I don’t have any long-running clusters).
But definitely, it would be beneficial to add an option to append the config to your existing config (and then change context). If there is anything else you can think of feel free to let me know in the comments or open GitHub issue.