A Guide to Taints and Tolerations, Node Affinity and Node Selector

Taint and affinity in Kubernetes are two mechanisms that allow users to specify how pods should be scheduled on nodes in a cluster. Node selectors are another mechanism that allows users to specify which nodes a pod should be scheduled on. In this article, we will explain the difference between taint, affinity, and node selector in Kubernetes and how they can be used to control pod scheduling in a cluster.

Taint

Taint is a feature in Kubernetes that allows users to mark a node as "tainted" with a specific key and value. Pods that do not have toleration for the taint will not be scheduled on the tainted node. Taint can be used to mark nodes as unavailable for certain pods, for example, if the node is undergoing maintenance or if it has a specific hardware or software configuration that is not suitable for certain workloads.

To taint a node, use the kubectl taint command. For example, to taint a node with key "disktype" and value "ssd", use the following command:

kubectl taint nodes <node-name> disktype=ssd:NoSchedule

To tolerate a taint, you can specify a toleration in the pod specification. For example, the following pod specification tolerates the taint "disktype=ssd" with a toleration period of 3600 seconds:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  tolerations:
  - key: "disktype"
    operator: "Equal"
    value: "ssd"
    effect: "NoSchedule"
    tolerationSeconds: 3600
  containers:
  - name: my-container
    image: nginx

📋Important Note: Taints and Tolerations do not guarantee that the pods only prefer these nodes. So the pods may be scheduled on the other untainted nodes.

Affinity

Affinity in Kubernetes is a mechanism that allows users to specify rules for pod scheduling based on node attributes. There are two types of affinity: node affinity and pod affinity.

Node Affinity

Node affinity allows users to specify which nodes a pod should or should not be scheduled on based on node labels. For example, you can use node affinity to specify that a pod should be scheduled on a node with a specific label, such as disktype=ssd.

To specify node affinity in a pod specification, use the affinity field and set the nodeAffinity field to the desired node affinity rules. For example, the following pod specification specifies that the pod should be scheduled on a node with the label disktype=ssd:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd
  containers:
  - name: my-container
    image: nginx

📋Important Note: ** Node Affinity by itself does not guarantee that other pods are not also scheduled on the same nodes as well.

Pod Affinity

Pod affinity allows users to specify which pods a pod should or should not be scheduled with based on labels. For example, you can use pod affinity to specify that a pod should be scheduled on the same node as other pods with a specific label, such as app=database.

To specify pod affinity in a pod specification, use the affinity field and set the `podAffinity

Node selector

Node selector is a feature in Kubernetes that allows users to specify which nodes a pod should be scheduled on based on node labels. Node selector works by adding labels to nodes and then specifying the desired node labels in the pod specification.

To add labels to a node, use the kubectl label command. For example, to add a label disktype=ssd to a node, use the following command:

kubectl label nodes <node-name> disktype=ssd

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  nodeSelector:
    disktype: ssd
  containers:
  - name: my-container
    image: nginx

Node selector is similar to node affinity in that both allow users to specify which nodes a pod should be scheduled on based on node labels. However, node selector is a simpler and more primitive mechanism compared to node affinity, which provides more fine-grained control over pod scheduling.

In summary, taint and affinity in Kubernetes allow users to specify rules for pod scheduling based on node attributes, while node selector allows users to specify which nodes a pod should be scheduled on based on node labels. All three mechanisms can be used to control pod scheduling in a cluster and ensure that pods are scheduled on the appropriate nodes based on the requirements of the workload.

Guidelines and Best Practices

Here are some guidelines for when to use taint, affinity, and node selector in Kubernetes:

A combination of Taints & Tolerations and Node Affinity rules can be used together to completely dedicate nodes for specific pods. We use Taints & Tolerations to prevent other pods from being scheduled on the desired nodes and then we use Node Affinity to have our pods to be scheduled on the desired nodes.
Taint should be used when you want to mark a node as unavailable for certain pods. For example, you can use taint to mark a node as "maintenance" and prevent pods from being scheduled on the node while it is undergoing maintenance.
Node affinity should be used when you want to specify which nodes a pod should or should not be scheduled on based on node labels. Node affinity provides more fine-grained control over pod scheduling compared to node selector and allows you to specify complex rules for pod scheduling based on multiple node labels.
Pod affinity should be used when you want to specify which pods a pod should or should not be scheduled with based on labels. Pod affinity can be used to ensure that certain pods are co-located on the same node or to ensure that certain pods are separated from each other.
Node selector should be used when you want to specify which nodes a pod should be scheduled on based on node labels, but do not need the fine-grained control provided by node affinity. Node selector is a simpler and more primitive mechanism compared to node affinity and is sufficient for many use cases.

In general, you should use taint and affinity when you need to specify complex rules for pod scheduling based on node attributes, and use node selector when you only need to specify simple rules based on node labels.

Hope you enjoyed reading this article, follow for more DevOps related content 👾🦄

Additional Resources:

Also published here.