Customise your EC2 to your deployment needs Picture the scene. You’ve got a few small, simple microservices. They all run beautifully on 200mb of ram and scale instantly. Engineering bliss. But wait… in this blue sky, a scar in the scenery, lies an omen. A storm cloud in the shape of a monolith. You need to get this beast running in your Kubernetes cluster but it’s dramatically larger than your other applications. In the words of the internet, “wut do?” Is your Microservice EC2 Instance Type enough? Your Microservices can be packed into small boxes, but your monolith needs some juice. There simply isn’t enough horsepower in a single instance to run your application as a pod. The purists are going to tell you “Well then, refactor it into microservices”. Reality isn’t often so black and white. So do you deploy everything onto big boxes? Your first port of call might be to reassess everything — drain the existing nodes and deploy onto bigger boxes. That way you don’t need to run multiple node pools. This will work! But it comes at a cost. There is a greater chance of wasted resources in your cluster. Bigger boxes mean you’re more likely to have very underutilized instances, costing you money. Okay, so can I have multiple node types? Yes you can, you lucky devil. It’s straightforward and requires only a few steps. For the sake of this tutorial, I’m going to assume you’re familiar with Kubernetes concepts like or . If you’re not, you should read through the . Additionally, I’ll assume you’re comfortable with AWS terminology like Autoscaling Group (ASG) and EC2. If you’re not, . For the sake of brevity and clarity, I won’t be including changes to IAM roles and security groups. The includes these details. pod node Kubernetes documentation have a read through the AWS documentation first official documentation Okay! The Cluster Autoscaler The first thing you’ll need to hook up is the . This has an AWS specific installation mode. It will detect when there is no free space in your EC2 instances to deploy pods, at which point it will increase the number of EC2 instances in your ASG. It works by adjusting the “desired” field in the ASG, up to a predefined maximum that you’ve set. I won’t go over the installation in detail here because . When you’re following these instructions, be sure to set it up for “Autodiscovery”. That way, when you add new ASGs, it will detect them and you won’t need to reconfigure your cluster. cluster autoscaler the documentation is pretty good Next, your Autoscaling Group Spinning up an ASG is straight forward. For the sake of example, we’ll be using Terraform. In the following code snippets, we will create an auto scaling group and a run configuration. The run configuration is the template that your ASG should use when it’s spinning up new EC2 instances. There are some key things to note here. We’re ignoring changes to . This is because the Cluster autoscaler will be using this value to scale the cluster up and down. We don’t want to reset it back to 2 every time we do a . desired_capacity terraform apply There are two highlighted tag keys. is to ensure that the cluster autoscaler can detect your Auto scaling groups. The second, isn’t mandatory but is a good idea if you’re running multiple clusters. This will prevent two clusters picking up the same ASG. k8s.io/cluster-autoscaler/enabled mandatory k8s.io/cluster-autoscaler/cluster_name resource "aws_autoscaling_group" "worker_node_asg" {desired_capacity = 2max_size = 10min_size = 1name_prefix = "worker-node-"lifecycle {ignore_changes = ["desired_capacity"]}tag {key = "Name"value = "worker-node"propagate_at_launch = true}tag {key = "k8s.io/cluster-autoscaler/enabled"value = "true"propagate_at_launch = true}tag {key = "k8s.io/cluster-autoscaler/cluster_name"value = "true"propagate_at_launch = true}} And the launch configuration (template) for your ASG. Note the field highlighted in bold. instance_type resource "aws_launch_configuration" "worker_launch_configuration" {image_id = "${data.aws_ami.worker_ami.id}"instance_type = "t2.medium"name_prefix = "worker-node-"security_groups = ["${some_security_group_ID}"]create_before_destroy = true} Create this infrastructure for each of the EC2 instance types you would like to create. We’ll assume, from here on out, you’ve done it twice for two different instance types. Let’s make the Pods more picky Your pods are pretty laid back at the moment — they sigh and say “oh just put me anywhere” as they’re flung at the k8s api. We need to give them a bit of choice and we can do that with . This will allow the pod to specify the type of node it would like to run on. node selectors Find your label If you run you’ll see some EC2 instances. Look for the label, that’ll let you know what the EC2 instance type is. You can use this in your resource yaml to pick which box you want your applications to deploy to. There are other ways to go about this that don’t pollute your yaml with AWS specific stuff —as always, be mindful. A simple example yaml might look something like this: kubectl get node -o yaml beta.kubernetes.io/instance-type apiVersion: v1kind: Podmetadata: name: nginx labels: env: testspec: containers: - name: nginx image: nginx nodeSelector: beta.kubernetes.io/instance-type: t2.medium Using the flag has given this pod a preference — it’s picky. Once you run and push it to the k8s API, kubernetes will find a box with the correct instance type and deploy it. If it doesn’t exist, it will trigger the autoscaling group to create an instance for you. nodeSelector kubectl apply -f myfile.yaml And there you have it! You’ve got a cluster that supports multiple EC2 instance types. You can add as many ASGs as you like. Go and take over the world you little scamp! If you enjoyed this tutorial, I’m regularly writing and throwing other articles . about on twitter
