Eviction vs Preemption
Both kick pods. Different reasons.
Eviction
Eviction and preemption are two different mechanisms by which Kubernetes removes pods. They are sometimes confused; the distinction matters because the response to each is different. The discipline is recognizing which is which.
What eviction is:
- Resource pressure.: Eviction happens when a node is under resource pressure. Memory pressure, disk pressure, PID pressure all can trigger eviction. The node's kubelet is the actor.
- Pod removed.: The kubelet evicts pods to relieve pressure. The eviction can be ordered (lowest priority first) but the trigger is resource pressure, not other pods' needs.
- Not necessarily replaced.: The evicted pod's controller (Deployment, StatefulSet) reschedules the pod. The reschedule depends on cluster capacity; if no capacity exists, the pod stays pending.
- Memory full.: When the node is running out of memory, the kubelet evicts pods. The evicted pods' memory is freed; the node returns to a healthy state.
- Disk full.: When the node's disk is filling, the kubelet evicts pods. Pods using significant ephemeral storage are evicted; the disk space is reclaimed.
Eviction is the resource-pressure response. The kubelet is acting to preserve the node's health.
Preemption
Preemption is different. A higher-priority pod needs to schedule but no capacity exists. The scheduler removes lower-priority pods to make room; preemption is the scheduler's tool.
- Higher-priority pod needs space.: A new pod with high priority needs to schedule. The cluster has no available capacity; without preemption, the pod stays pending.
- Lower-priority preempted.: The scheduler identifies lower-priority pods that could be removed to make room. Those pods are preempted; the high-priority pod schedules.
- Graceful for the higher pod.: The high-priority pod's needs are met. The cluster's most important workloads are protected; lower-priority workloads bear the cost.
- The scheduler is the actor.: Preemption is the scheduler's mechanism, unlike eviction's kubelet. The trigger is scheduling pressure, not node-level pressure.
- PreemptionPolicy controls behavior.: Pods can specify whether they can preempt others. The PreemptionPolicy field determines this; some workloads should never preempt others.
Preemption is the priority-driven response. The scheduler is satisfying high-priority needs at the cost of lower-priority pods.
Respond
The response depends on which mechanism is firing. Eviction patterns drive resource-pressure investigation; preemption patterns drive priority-class review.
- Eviction: address resource pressure.: When pods are being evicted, the cluster has resource pressure. The fix is more capacity, better resource limits, or both. The investigation focuses on resource utilization.
- Preemption: review priority classes.: When pods are being preempted, the cluster's priority configuration is determining which pods survive. The review verifies the priorities reflect the workloads' actual importance.
- Different fixes.: The two patterns have different fixes. Conflating them produces wrong responses; recognizing the distinction produces appropriate responses.
- Track both metrics.: The team tracks eviction and preemption rates separately. Both growing indicates cluster pressure; one growing indicates one pattern.
- Postmortem repeated patterns.: When eviction or preemption patterns repeat, postmortems drive structural fixes. The cluster's capacity, the workload's resource specs, the priority hierarchy all are candidates for adjustment.
Eviction vs preemption is one of those Kubernetes distinctions that matters in practice. Nova AI Ops integrates with cluster events, surfaces both patterns separately, and supports the targeted investigation that each pattern needs.