Understanding Kubernetes Node Taints and Tolerations

2 years ago   •   4 min read

By Anurag
Table of contents

In Kubernetes, we know that the smallest deployable unit is pods and these pods gets deployed on some nodes. Nodes are the virtual machines that run your containerized workloads with the help of kubelet and container runtime. With different types of nodes available out there, the obvious question that comes to our head is which nodes are good for our pods? Which nodes are perfect for our pods?

Even if it’s not coming to your mind now, you should keep a note in your mind that Kubernetes is a highly dynamic orchestrator. So, even if your pods gets scheduled on some nodes for the moment, chances are Kubernetes will kill the pod if the node is facing disk pressure, memory pressure etc.

One more aspect where taints is required

So, scheduling your workloads on nodes is one of the most important thing that you should configure correctly in your Kubernetes cluster. Your initial attention to careful scheduling of your workloads will save you from troubleshooting the issues in staging/production environments.

What are taints & tolerations?

Think of taint as something that we apply on Kubernetes nodes and Kubernetes nodes will repel a set of pods. It’s a way of telling Kubernetes that don’t schedule these pods on the nodes under consideration.

Please note that we are applying the taints on the nodes. Taints are the property of the nodes. You really should understand that it’s node who’s telling that I don’t like these set of pods. It’s not like the node is saying that I have some love for certain types of pods.

Taints on a Kubernetes node looks like this

key=value:effect 

Notice the effect in the last, we use effect to describe the behaviour of what would happen if the pods are not fulfilling the criteria of nodes. What will happen if pods are not tolerating the taints by nodes?

Essentially there are three effect which you need to keep in mind

  1. NoSchedule - NoSchedule means do not schedule the pods that cannot tolerate the taints.

2. preferNoSchedule - It basically means that try not scheduling the pods on the nodes if the pods don’t tolerate the taints but in case there’s no more nodes left then you can schedule.

3. NoExecute - This is the most aggressive one and it basically means that pods which were scheduled earlier on the nodes and out of those pods that don’t satisfy the taints will be evicted.

In starting this may sound confusing but if you will read it twice or thrice then it would make more sense.

Node Controller

The node controller automatically taints a node based on certain conditions.

As per documentation,

node.kubernetes.io/not-ready: Node is not ready. This corresponds to the NodeCondition Ready being "False".

node.kubernetes.io/memory-pressure: Node has memory pressure.

You can see the more comprehensive list here

These taints are applied to the nodes by the nodeController and while scheduling, scheduler check these taints and schedule the workloads accordingly.

Adding & removing taints from node

  • How to taint a Node?
  • kubectl taint nodes <node-name> key=value:NoSchedule
  • How to remove taint from a node?
  • Just append the previous command with a - thing. kubectl taint nodes <node-name> key=value:NoSchedule-
  • For example purpose let’s taint a node with key-value foo=bar and the effect will be PreferNoSchedule

Adding tolerations to the pod

This tolerations you have to configure in your pod.spec section of the manifest.

tolerations:
  - key: "foo"
    operator: "Equal"
    value: "bar"
    effect: "NoSchedule"

Adding tolerations to your pods using Devtron

With Devtron you can use to add tolerations to your pods. To add tolerations to your pods you can use the deployment template.

Go to your application > App Configuration > Deployment Template.

In the deployment template, search for tolerations and add it there.

I have one demo application running in the background and I will add tolerations to that.

devtron-tolerations

And after saving this deployment template and I will run the deployment pipeline once again for changes to take place.

After running, you will see the status of all your resources. All the resources are healthy now.

healthy-resources

Now let’s check whether the taints were applied during the new deployment or not. For that hover over the pod, and you will see that that tolerations were applied to the pods.

pod-tolerations

Difference b/w nodeAffinity & taints

In case of NodeAffinity we are telling the pods that you love some types of nodes and the scheduler will try to send the pods to the nodes which pods love. So, is it mandatory that the pods will be scheduled only on the nodes that they have affinity towards?

Clearly No, we have not mentioned anywhere that pods hate any kind of nodes. So, if the nodes which pods love are full/not available, then the scheduler will schedule it on some other nodes.

In case of taints & tolerations, it’s the node which is in focus. The node is demanding that I don’t like these sets of pods. Depending on the situation, we can have different effects of taints. Again, here, pods can be scheduled on other nodes because other nodes don’t have any criteria.

Spread the word

Keep reading