Priority vs Preemption

High-priority pods can preempt low. Behaviour.

Idea

Priority and preemption together produce ordered workload survival under cluster pressure. Priority specifies relative importance; preemption is the mechanism that enforces it. The discipline is using both correctly to protect critical workloads.

What the idea looks like:

• When cluster is full, high-priority pods evict low-priority.: When a high-priority pod cannot schedule, preemption identifies low-priority pods that could be removed to make room. Those pods are evicted; the high-priority pod schedules.
• Preemption is automatic.: The Kubernetes scheduler handles preemption automatically. The team configures priorities; the preemption mechanism enforces them when capacity is tight.
• Per-pod priority class.: Each pod has a priority class. Higher priority class equals more important; the cluster's eviction order respects this.
• Cluster-wide capacity protection.: The mechanism protects critical workloads cluster-wide. A node-level pressure event might evict; a cluster-level capacity event triggers preemption; both protect the high-priority workloads.
• Documented hierarchy.: The team's priority hierarchy is documented. New workloads get appropriate priority; the discipline is consistent.

The idea is the safety net. Critical workloads are protected automatically when capacity is tight.

When

Priority and preemption fit mixed-priority clusters. Some workloads must always run; others tolerate preemption; the cluster supports both with the right priority configuration.

• Mixed-priority clusters.: Clusters running production and batch workloads benefit. Production gets high priority; batch gets low priority; preemption protects production when capacity is tight.
• Production must always run.: The discipline is that production never gets preempted. Priority class assignments reflect this; the eviction order respects it; production stays running.
• Batch jobs accept preemption.: Batch workloads tolerate interruption. They restart from checkpoints; the preemption is operational normal; the cluster's flexibility is preserved.
• Single-priority clusters do not need this.: If all workloads have the same priority, preemption does not help. The discipline applies when there is meaningful priority diversity.
• Calibrate the hierarchy.: Too many priority levels produces complexity; too few does not capture the workload diversity. The team picks 3-5 levels typically; the calibration matches the workload mix.

Mixed-priority is the natural fit. The discipline produces the right behavior under capacity pressure.

Policy

The preemptionPolicy field on pods controls preemption behavior. Some pods should not preempt others; the policy field handles these cases.

• preemptionPolicy: Never disables for specific pods.: A pod with preemptionPolicy: Never cannot preempt other pods. The pod waits if capacity is unavailable; it does not displace lower-priority pods.
• Use for non-critical pods that should wait, not preempt.: Some workloads should not displace others. Best-effort batch jobs, optional workloads, observability collection all might benefit from preemptionPolicy: Never.
• Default is PreemptLowerPriority.: Without the field, pods preempt lower-priority pods normally. The default is what most workloads should use.
• Document the policy choices.: Pods with preemptionPolicy: Never have documented justification. Future maintainers understand; the choice is preserved.
• Test the behavior.: The team tests preemption behavior. Workloads that should preempt do; workloads that should not, do not. The configuration matches the intent.

Priority vs preemption is one of those Kubernetes scheduling disciplines that pays off when capacity gets tight. Nova AI Ops integrates with cluster scheduling telemetry, surfaces priority and preemption patterns, and supports the team's workload survival decisions.