Graceful Shutdown for Pods

Pods can shut down gracefully. The pattern.

preStop hook

Graceful shutdown is the discipline of having pods exit cleanly when terminated. The pod's in-flight work completes; new traffic is rejected; the termination is bounded. Without graceful shutdown, restarts and rolling updates produce dropped requests.

What preStop hooks provide:

• Runs before SIGTERM.: The preStop hook executes before Kubernetes sends SIGTERM. The hook prepares the pod for shutdown; subsequent SIGTERM finds the pod ready.
• Drain connections.: Common preStop work is draining connections. The pod stops accepting new requests; existing requests complete; the connection pool empties cleanly.
• Save state.: Stateful pods save state in preStop. In-memory caches flush to persistent storage; ongoing operations checkpoint; the data is preserved.
• Up to terminationGracePeriodSeconds.: The preStop hook has bounded time. terminationGracePeriodSeconds (default 30) limits the total grace period; the preStop fits within this.
• Sleep for service propagation.: A common preStop is just sleeping for a few seconds. Kubernetes propagates the pod's removal from services; existing connections drain; the discipline is structural.

preStop is the cleanup hook. The discipline is using it for the cleanup work each application needs.

Signal handling

The application receives SIGTERM. The signal handler is what makes the application graceful; without it, the application hard-exits at SIGTERM.

• App handles SIGTERM.: The application code includes a SIGTERM handler. The handler triggers graceful shutdown; the application stops accepting new work; finishes in-flight work; exits.
• Closes listeners.: The handler closes network listeners. New connections are refused; existing connections continue serving their requests; the load balancer routes new traffic elsewhere.
• Finishes in-flight.: The handler waits for in-flight requests to complete. Each request finishes its processing; the response is sent; the connection closes naturally.
• 30s default; tune by app.: The default terminationGracePeriodSeconds is 30. Some applications need more (long-running requests); some need less (stateless workers). The team tunes per application.
• SIGKILL fallback.: If the application does not exit within the grace period, Kubernetes sends SIGKILL. The discipline is exiting before SIGKILL; SIGKILL produces dropped work.

Signal handling is the application discipline. Graceful shutdown depends on the application code.

Test

The graceful shutdown is tested under load. Kill pods; verify no requests are dropped; the discipline is verified.

• Kill pods.: The team simulates pod termination. kubectl delete pod, rolling update, node drain all produce pod termination; the test exercises the path.
• Verify no in-flight requests dropped.: The test checks for dropped requests. Load generators run; pods terminate; the dropped-request count must stay zero.
• Real test under load.: The test must be under realistic load. Idle pods may shut down cleanly while loaded pods do not; the test under load reveals the truth.
• Document the procedure.: The test procedure is documented. The team can repeat it; new applications inherit the test discipline; the consistency is preserved.
• Production validation.: Some teams verify in production. Routine deploys exercise graceful shutdown; metrics confirm no impact; the discipline is continuously validated.

Graceful shutdown is one of those Kubernetes operational disciplines that prevents customer-visible failures during routine operations. Nova AI Ops integrates with cluster pod telemetry, surfaces shutdown patterns, and supports the team's reliability discipline.