Checking if a Kubernetes Deployment is balanced

Every now and then I find myself needing to check if pods are split evenly across the available nodes. This can be for a number of reasons, but the most common for me is when Kubernetes, during pod scheduling, has preferred one or two nodes.

This can cause a node to use more resources than planned and could cause service disruptions, and who wants that?

If you have a small number of pods and nodes you could run the following command to get a quick overview of pod distribution.

kubectl get pods  -o wide --sort-by=.spec.nodeName | grep -i website

But when you are dealing with hundreds of pods across many nodes its not as easy to scan the results.

So I created this bash script to help me with just that! It takes a deployment name and queries the nodes to check if they are balanced.

#!/usr/bin/env bash

# Name of the target kubernetes deployments
DEPLOYMENT_NAME="example-deployment"
# Count of all running target deployments
DEPLOYMENT_COUNT=$(kubectl get pods --field-selector=status.phase=Running | grep -i ${DEPLOYMENT_NAME} | wc -l)
# List of all nodes
NODES=($(kubectl get no -o name | sed "s/node\///g"))
# Count of nodes
NODE_COUNT=${#NODES[@]}
# Are the pods balanced?
BALANCED=1

# Get the replica count from the deployments
DEPLOYMENT_EXPECTED_COUNT=$(kubectl get deployment ${DEPLOYMENT_NAME} -o go-template --template="{{.status.replicas}}")

EXPECTED_PODS_PER_NODE=$(( ${DEPLOYMENT_EXPECTED_COUNT} / ${NODE_COUNT} ))

for node in ${NODES[@]}; do
    echo -e "------\e[93m${node}\e[39m------\n"
    PODS=($(kubectl get pods --all-namespaces --field-selector spec.nodeName="${node}" -o wide -o name | sed "s/pod\///g" | grep -i ${DEPLOYMENT_NAME}))
    for pod in ${PODS[@]}; do
        echo ${pod}
    done

    echo -e "\n"

    POD_COUNT=$(kubectl get pods --all-namespaces --field-selector spec.nodeName="${node}" -o wide -o name | sed "s/pod\///g" | grep -i ${DEPLOYMENT_NAME} | wc -l)
    if (( "${POD_COUNT}" > "${EXPECTED_PODS_PER_NODE}" )); then
        EXCESS=$(( ${POD_COUNT} - ${EXPECTED_PODS_PER_NODE} ))
        echo -e "Node (${node}) has ${EXCESS} too many pods"
        BALANCED=0
    fi

    echo -e "\n"
done

if [[ ${BALANCED} == 0 ]]; then
    echo "Pods are not balanced"
else
    echo "Pods are balanced"
fi

This will output something like this

------node_1------
example-deployment-1
example-deployment-2

------node_2------
example-deployment-3
example-deployment-4

 ------node_2------
example-deployment-5


Pods are not balanced

This script has helped me a few times identify if there is a problem with pod balancing in my clusters. Feel free to take it for your self and modify it, there are plenty of improvements that could be made for it.

If you use this or make any adjustments please do let me know (Just for my own curiosity)