Cost saving has always been one of the important objectives for organizations but now it is more important than ever before. Because of uncertainty in the business, earlier motto of growth at all cost has been replaced with responsible growth.
You must be thinking that it is trivial as Kubernetes supports it out of the box, hold your thoughts for a while. I promise you, by the end of this post you will know the best possible solution to use spot instances in Kubernetes clusters using mechanisms provided by Kubernetes.
Leveraging AWS Spot Instances within Kubernetes clusters offers a compelling solution, providing cost reductions compared to On-Demand instances. However, the inherent variability of Spot Instances necessitates robust strategies to ensure workload resilience. Devtron platform simplifies the integration of Spot Instances, balancing cost efficiency with operational stability.
How to handle termination notifications and drain resources? is also important to maintainthe SLA of micro-services but is beyond the scope of this article. We will cover that in a separate article.
In this article, we'll explain how to leverage spot instances in Kubernetes, exploring approaches, tools, and best practices for optimal cost savings. With efficient management of such instances, organizations can maximize resource utilization and save substantial costs in cloud-native environments.
How auto-scaling works in Kubernetes?
Before we go into details, lets understand the process of auto-scaling of nodes in Kubernetes cluster.
- If the kube scheduler is unable to place the pod on any of the nodes, it marks the pod as unschedulable.
- Cluster autoscaler watches unschedulable pods.
- If the cluster autoscaler finds an unschedulable pod, it filters and prioritizes node groups to select a node group on which this pod can be scheduled.
- Cluster autoscaler increases desired instance count in the Autoscaling Group (ASG) of the selected node group.
- ASG scale nodes based on the scaling strategy.
- Kube Scheduler filters and prioritizes nodes and schedules the pod on the node with the highest priority score.
Without much further ado let’s start our journey.
Spoiler Alert: First two attempts are failures and third attempt is successful.
Attempt 1
Based on my discussion, this is the second most popular approach to use spot instances with Kubernetes. It goes like this
If nodes have the right spot-is-to-on demand ratio then pods will automatically have the right ratio.
Kops supports the mixed instance group for AWS since version 1.14. Mixed Instance groups can be used to achieve the right ratio of spot and on demand instances.
Let’s look at a relevant part of a sample instance group configuration.
spec:
mixedInstancesPolicy:
onDemandBase: 3
onDemandAboveBase: 30
spotAllocationStrategy: capacity-optimized
nodeLabels:
lifecycle: Spot
As per the above configuration minimum 3 on demand nodes will be available. For additional requirements, 30% of the nodes will be of on demand type and rest 70% will be of spot type.
For node affinity, following is the relevant portion of the pod spec
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: lifecycle
operator: In
values:
- Spot
Case 1: Scaling not Required
If the cluster has capacity to schedule the pod, kube-scheduler will use its filter and priority algorithms to schedule the pod on the best possible node.
Priority algorithm of kube scheduler doesn’t differentiate between spot and on demand nodes. Therefore, even though the nodes will be in approx 70-is-to-30 ratio for spot-is-to-on demand, distribution of pods across these nodes may not be in this ratio.
Case 2: Scaling Required
If cluster doesn’t have capacity to schedule the pod then cluster autoscaler will increase the count of desired instances in ASG.
ASG will then provision a new node such that the ratio of 70-is-to-30 ratio for spot-is-to-on demand is maintained.
After provisioning, kube scheduler will schedule the pod to the new node, assuming there were no pod evictions in between. So in case of a scaling event, the pod will be assigned to the right kind of node.
Can we do better
We can use inter pod anti affinity for better distribution of pods but it will still not guarantee distribution of 70-is-to-30 unless number of pods is equal to number of nodes.
Outcome: failure
Even though nodes will have the desired spot-is-to-on demand ratio, pods may or may not be spread in this ratio, resulting in unstable services in case of spot node outage.
Attempt 2
This is the most often cited approach to use spot instances in Kubernetes clusters
Use node affinity to control distribution of pods across spot and on demand nodes.
For this to work, at least two node groups are needed, one with spot instances only and other with on demand instances only.
Following are the relevant configurations from the two node groups, this can be done without mixed instance nodegroup also.
For spot
spec:
mixedInstancesPolicy:
onDemandBase: 0
onDemandAboveBase: 0
spotAllocationStrategy: capacity-optimized
nodeLabels:
lifecycle: Spot
Similarly, for on demand
spec:
mixedInstancesPolicy:
onDemandBase: 3
onDemandAboveBase: 100
nodeLabels:
lifecycle: OnDemand
Following is the relevant pod spec for node affinity
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 70
preference:
matchExpressions:
- key: lifecycle
operator: In
values:
- Spot
- weight: 30
preference:
matchExpressions:
- key: lifecycle
operator: In
values:
- OnDemand
What does weight mean?
Weight of 30 and 70 doesn’t mean the scheduler will distribute pods between these two node labels in the ratio of 30-is-to-70.
Scheduler combines weight mentioned in the above spec with the ones it has computed using priority functions and assigns pod to node group with the highest score.
Case 1: Scaling not required
If scaling is not required then it will prefer to schedule on spot instances as it has a weight of 70, though actual placement may depend on the score obtained through various priority functions used by the kube scheduler.
Case 2: Scaling required
In case scaling of nodes is required to schedule the pod, then the cluster autoscaler will filter all node groups and will prioritize eligible nodegroups based on its priority algorithm.
Priority algorithms used by the cluster autoscaler are not the same as the kube scheduler. By default it uses the random algorithm, so it will pick one node group randomly out of eligible node groups. In this case, it will select one node group in random order and increase desired instance count in ASG.
Once ASG has provisioned the node, kube scheduler will assign the pod to this new node.
Can we do better?
No, pod anti-affinity will not help as the ratio of spot-is-to-on demand is not equal to the desired ratio.
Outcome: failure
Neither nodes will not have the desired spot-is-to-on demand ratio nor pods will have the desired ratio. This turns out to be worse than attempt 1.
Attempt 3
This is the least mentioned approach, it uses Pod Topology Spread Contraints, which was introduced in Kubernetes 1.16 and was made beta in 1.18. We will use pod topology spread constraints to control how pods are spread across the spot and on demand instances in the cluster.
Following are the relevant configurations from the two node groups, again this can be done without mixed instance nodegroup also.
For spot
spec:
mixedInstancesPolicy:
onDemandBase: 0
onDemandAboveBase: 0
spotAllocationStrategy: capacity-optimized
nodeLabels:
lifecycle: Spot
Similary, for on demand
spec:
mixedInstancesPolicy:
onDemandBase: 3
onDemandAboveBase: 100
nodeLabels:
lifecycle: OnDemand
Following is the relevant portion of the pod spec, for pod topology constraints
metadata:
labels:
app: sample
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: lifecycle
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: sample
How topologySpreadConstraints works?
Topology Spread Constraints uses node labels to identify topology domain(s) of the node. topologyKey is the key of node labels. Kube scheduler tries to place a balanced number of pods across all unique values of this node label (topologyKey).
In our example topologyKey is lifecycle, which has 2 unique values Spot and OnDemand. Kube Scheduler will place pods across nodes with these two values such that the maximum difference between pod count across these two values cannot be more than maxSkew(1 in this case).
In case this label was missing in any node group then scheduler would have not scheduled pod to that node group.
Important point to note is that maxSkew doesn’t favor any label value against topologyKey. It will skew in either direction based on the availability and priority of nodes. Though it will ensure that unbalance between label values is not more than maxSkew.
whenUnsatisfiable is set to DoNotSchedule, kube scheduler ensures that pod is not scheduled in such a way that maxSkew cannot be maintained.
Case 1: Scaling not required
If scaling is not required, kube-scheduler will filter and prioritize node groups which honour the maxSkew and therefore pods will be scheduled in the desired ratio.
Case 2: Scaling required
When scaling is required, cluster autoscaler will filter node groups which honour the topology contraints and increment desired instance number in related ASG.
After ASG has scaled the instance, kube scheduler will assign the pod to the node and therefore pods will be scheduled in the desired ratio.
What’s the catch?
maxSkew is a number, which means that when we use it with HPA as pods will scale the ratio of spot-is-to-on demand will change.
maxSkew can be on either side, it is possible to have
- number of pods on spot = number of pods on ondemand + maxSkew
- number of pods on ondemand = number of pods on spot + maxSkew
Which means, for a replica count of 5 and maxSkew of 1, ratio can be 2-is-to-3 or 3-is-to-2 for spot-is-to-on demand nodes. This becomes more unpredictable as value of maxSkew becomes higher.
To achieve higher skew its better to create more buckets with the topologyKey.
Outcome: failure
We cannot get an exact spot-is-to-on demand ratio but we can have a predictable ratio nonetheless.
Complete configuration of samples used in this blog is available in this git repo.
Conclusion
Leveraging the power of Spot instances can be extremely beneficial for achieving on Kubernetes. However, if done incorrectly, you might be compromising on cluster stability. Spot nodes can be deleted at any time if the spot goes above your bidding price, which can lead to service disruptions.
There are many different ways in which we can properly configure pod distribution on spot instances. The two most popular methods include giving the nodes the right spot-is-to-on demand ratio and using node affinity to control pod distribution across nodes. However, both these approach have certain limitations. By leveraging the Pod Topology Spread Contraints, we can get a great balance of cost savings with great reliability for our applications.
To learn more about Kubernetes, and different ways in which you can optimize your cloud costs, join Devtron's Discord server.
FAQ
How can I use AWS Spot Instances in Kubernetes for cost savings?
Use Spot Instances by configuring node groups with lifecycle: Spot labels and managing pod scheduling using Kubernetes features like node affinity or topology spread constraints to optimize cost without sacrificing reliability.
What are Pod Topology Spread Constraints in Kubernetes?
Pod Topology Spread Constraints ensure even distribution of pods across node groups using specific node labels. It uses topologyKey, maxSkew, and whenUnsatisfiable fields to limit the difference in pod count between different node types like Spot and OnDemand.
Why do node affinity and mixed instance groups fail to ensure correct Spot instance usage?
Node affinity and mixed instance groups do not guarantee consistent pod distribution across Spot and OnDemand nodes because Kubernetes scheduler and cluster autoscaler use different scoring and prioritization logic, which leads to unpredictable outcomes during scaling.
What’s the best Kubernetes strategy for balancing Spot and OnDemand instances?
Using Pod Topology Spread Constraints with lifecycle node labels and maxSkew provides a balanced and resilient approach to schedule pods across Spot and OnDemand nodes, offering cost efficiency with predictable distribution, especially during scaling events.
