Skip to content

EMK - High available workloads

Estimated time to read: 2 minutes

Your Kubernetes cluster control plane can be high available (HA). This is possible by setting your cluster HA setting.

K8s ha cluster

Source: Gardener documentation

In maintenance windows or during performed maintenance it can happen that a node will be drained from your Kubernetes cluster. When this happens there are systems in place within Kubernetes that take care of your workload. In case of a node drain to keep your applications to be highly available Kubernetes will look at your applications and it's PodDisruptionBudget.

Below we go a bit more in-depth about restrictions with PodDisruptionBudget, although there are more best practices. You can find some at the Gardener documentation.

PodDisruptionBudget

Kubernetes offers a feature called PodDisruptionBudget (PDB) for each application. A PDB limits the number of pods of a replicated application that are down simultaneously from voluntary disruptions.

The most common use case is when you want to protect an application specified by one of the built-in Kubernetes controllers:

  • Deployment
  • ReplicationController
  • ReplicaSet
  • StatefulSet

A PodDisruptionBudget has three fields:

  • A label selector .spec.selector to specify the set of pods to which it applies.
  • .spec.minAvailable which is a description of the number of pods from that set that must still be available after the eviction, even in the absence of the evicted pod. minAvailable can be either an absolute number or a percentage.
  • .spec.maxUnavailable which is a description of the number of pods from that set that can be unavailable after the eviction. It can be either an absolute number or a percentage.

Cluster failures with PDB

Misconfiguration of the PDB could block the cluster upgrade or node deletion processes. There are two main cases that can cause a misconfiguration. Those are:

  • The replica of Kubernetes controllers is 1
  • HPA configuration violates PDB

The replica of Kubernetes controllers is 1

  • Only 1 replica is running: there is no replicaCount setup or replicaCount for the Kubernetes controllers is set to 1
  • PDB configuration 
    spec:
  minAvailable: 1
  • To fix this PDB misconfiguration, you need to change the value of replicaCount for the Kubernetes controllers to a number greater than 1

HPA configuration violates PDB

In Kubernetes, a HorizontalPodAutoscaler automatically updates a workload resource (such as a Deployment or StatefulSet), with the aim of automatically scaling the workload to match demand. The HorizontalPodAutoscaler manages the replicas field of the Kubernetes controllers.

  • There is no replicaCount setup or replicaCount for the Kubernetes controllers is set to 1
  • PDB configuration 
    spec:
  minAvailable: 1
  • HPA configuration 
    spec:
  minReplicas: 1
  • To fix this PDB misconfiguration, you need to change the value of HPA minReplicas to be greater than 1