Kubernetes in Docker: Adding Nodes Dynamically to a Kind Cluster

How to dynamically add nodes to a kind cluster

Kind allows you to run a Kubernetes cluster inside Docker. This is incredibly useful for developing Helm charts, Operators, or even just testing out different k8s features in a safe way.

I've recently been working on an operator (built using the operator-sdk) that manages cluster node lifecycles. Kind allows you to spin up clusters with multiple nodes, using a Docker container per node and joining them using a common Docker network. However, the kind executable does not allow you to modify an existing cluster by adding or removing a node.

I wanted to see if this was possible using a simple shell script, and it turns out that it's actually not too difficult!

Creating the node

Using my favorite diff tool, DiffMerge, docker inspect to compare an existing kind node's state to a new container's, I experimented with various docker run flags until I got something that's close enough to the kind node.

docker run \
--restart on-failure \
-v /lib/modules:/lib/modules:ro \
--privileged \
-h $NODE_NAME \
-d \
--network kind \
--network-alias $NODE_NAME \
--tmpfs /run \
--tmpfs /tmp \
--security-opt seccomp=unconfined \
--security-opt apparmor=unconfined \
--security-opt label=disable \
-v /var \
--name $NODE_NAME \
--label io.x-k8s.kind.cluster=kind \
--label io.x-k8s.kind.role=worker \
--env KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER \
kindest/node:v1.25.2@sha256:9be91e9e9cdf116809841fc77ebdb8845443c4c72fe5218f3ae9eb57fdb4bace

Joining the cluster

You can join new nodes to a k8s cluster by using the kubeadm join command. In this case, we can use docker exec to execute this command on our node after its container has started up.

This command won't work out of the box because kind uses a kubeadm.conf that does not exist in the node docker image. It is injected into the container by the kind executable.

Again, using my trusty DiffMerge tool, I compared two /kind/kubeadm.conf files in existing kind nodes and found very few differences. This allowed me to just grab one from any worker node to use as a template.

docker exec --privileged kind-worker cat /kind/kubeadm.conf > $LOCAL_KUBEADM

From here, I needed to set the node's unique IP in its kubeadm.conf. We can use docker inspect to grab any node IP address we need. Since I'm working in bash, I just decided to use a simple sed replacement to replace the template node's IP address with my new node's IP in my local copy of kubeadm.conf.

TEMPLATE_IP=$(docker inspect kind-worker | jq -r '.[0].NetworkSettings.Networks.kind.IPAddress')
NODE_IP=$(docker inspect $NODE_NAME | jq -r '.[0].NetworkSettings.Networks.kind.IPAddress')

ESCAPED_TEMPLATE_IP=$(echo $TEMPLATE_IP | sed 's/\./\\./g' )
ESCAPED_NODE_IP=$(echo $NODE_IP | sed 's/\./\\./g')

sed -i.bkp "s/${ESCAPED_TEMPLATE_IP}/${ESCAPED_NODE_IP}/g" $LOCAL_KUBEADM

Now that our kubeadm.conf is prepared, we need to copy it to the new node:

docker exec --privileged -i $NODE_NAME cp /dev/stdin /kind/kubeadm.conf < $LOCAL_KUBEADM

Finally, we can join our node to the cluster:

docker exec --privileged $NODE_NAME kubeadm join --config /kind/kubeadm.conf --skip-phases=preflight --v=6

Node Tags

Since you have complete control of the new node's kubeadm.conf, it is possible to configure many of its properties for further testing. For example, to add additional labels to the new node, you can run something like this:

sed -i.bkp "s/node-labels: \"\"/node-labels: \"my-label-key=my-label-value\"/g" $LOCAL_KUBEADM

This will add the my-label-key=my-label-value label to the node once it joins the cluster.

Future Work

Based on this script, I believe it's possible to add a kind create node subcommand to add a node to an existing cluster. Stay tuned for that...

Also published here.