Winter Soldier: Scale down your Infrastructure in the easiest possible way

Running Kubernetes can be very expensive, especially when done inefficiently. This is often the case when companies have just started to roll out Kubernetes in their organizations for different environments such as Staging, QA, Dev, Prod. etc. Still, over time these environments end up with workloads(k8s resources) that have outlived their utility and add to Total Cost of Ownership of infrastructure. Removing every unused resource manually can be a tedious task.

Now think out a situation where you want to scale down the non-prod infra automatically during the night or maybe weekends to avoid extra resource consumption, again hibernating the resources is a task that administrators avoid doing manually

In this post, we’ll focus on how to remove or scale down the orphaned (unused resources)

This can be achieved using Winter Soldier, no it's not Bucky, Captain America’s best friend. It is an open-source command line utility tool by Devtron that is used to -

1.Delete k8s resource based on conditions.

2.Scale down the Workload to Zero at a Specific period of date & time.

Winter Soldier is an Operator which expects conditions to be defined using a CRD hibernator, it contains three config files.

1. CRD -  An extension of the Kubernetes API that contains a custom definition of a hibernator.
2. Winter-soldier deployment - It contains cluster roles, role binding, service account & deployment required by the hibernator.
3. Hibernator- It contains custom policies based on which operations are done.

Install Winter Soldier in your cluster

Note: Currently, winter soldier supports k8s version less than or equal to v1.21, in future releases, we will add support for v1.21+ also.

Step1- Execute the following command to apply the CRD

kubectl apply -f https://raw.githubusercontent.com/devtron-labs/winter-soldier/main/config/crd/bases/pincher.devtron.ai_hibernators.yaml

Step2- Create winter-soldier deployment

vim winter-soldier.yaml

Change the namespace spec according to need & apply the manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: winter-soldier
    app.kubernetes.io/part-of: winter-soldier
    app: winter-soldier
  name: winter-soldier
  namespace: demo    
spec:
  selector:
    matchLabels:
      app: winter-soldier
  replicas: 1
  template:
    metadata:
      labels:
        app: winter-soldier
    spec:
      containers:
      - command:
        - /manager
        args:
        - --enable-leader-election
        image: quay.io/devtron/winter-soldier:3f10a62b-196-5753
        name: manager
      terminationGracePeriodSeconds: 10
      serviceAccount: winter-soldier
      serviceAccountName: winter-soldier
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: winter-soldier
    app.kubernetes.io/part-of: winter-soldier
  name: winter-soldier
  namespace: demo
rules:
- apiGroups:
  - pincher.devtron.ai
  resources:
  - hibernators
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - "*"
  resources:
  - "*"
  verbs:
  - get
  - patch
  - update
  - create
  - delete
  - list
  - watch
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: winter-soldier
    app.kubernetes.io/part-of: winter-soldier
  name: winter-soldier
  namespace: demo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/name: winter-soldier
    app.kubernetes.io/part-of: winter-soldier
  name: winter-soldier
  namespace: demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: winter-soldier
subjects:
- kind: ServiceAccount
  name: winter-soldier
  namespace: demo

kubectl apply -f winter-soldier.yaml -ndemo

This will automatically create cluster-roles, role binding, Service account & deployment in demo namespace which are required by the hibernator.

Step3- Create & apply hibernator policy according to your use case, here is a sample hibernator.

apiVersion: pincher.devtron.ai/v1alpha1
kind: Hibernator
metadata:
  name: k8s-hibernator
spec:
  timeRangesWithZone:
    timeZone: "Asia/Kolkata"
    timeRanges:
    - timeFrom: 00:00
      timeTo: 23:59:59
      weekdayFrom: fri
      weekdayTo: Sun
  selectors:
  - inclusions:
    - objectSelector:
        name: ""
        type: "deployment,statefulset"
        fieldSelector:
          - AfterTime(Now(), AddTime(ParseTime({{metadata.creationTimestamp}}, '2006-01-02T15:04:05Z'), '1d'))
      namespaceSelector:
        name: "all"
    exclusions:
    - objectSelector:
        name: ""
        type: "deployment,statefulset,rollout"
      namespaceSelector:
        name: "kube-system,demo,devtroncd,kube-node-lease,kube-public,argo"
  action: sleep

The above hibernator policy will reduce the replicas to zero as action set is sleep of all Deployments in all namespace excluding (kube-system, demo, devtroncd, kube-node-lease, kube-public, argo) namespace from Friday to Sunday at 00:00 to 23:59:59

At the end of the hibernation cycle, it sets the replica count of workload to the same number as before hibernation.

Feel free to explore this sample hibernator policy for more examples

Terminology Primer

Actions: actions are the operations performed on resources, winter-soldier support two types of  action

Conditions: It is a parameter based on which resources are selected to take action

Timezone: Used to Specify the time & time-zone at which winter-soldier will be active

Finding and removing orphaned resources takes time & effort and you might not remember to make it a part of your routine. Winter Soldier takes off the burden from your shoulders so you can easily maintain the hygiene of your Kubernetes clusters and optimize the cost as much as possible.