Secret Rotation in K8s

Secrets in K8s rotate. The discipline.

Source

Secret rotation discipline in Kubernetes is the practice of keeping secrets fresh and ensuring rotated secrets propagate to consumers. The discipline involves a source of truth outside Kubernetes, automated refresh in pods, and audit trails for compliance.

What the source approach looks like:

• External Secrets Operator pulling from Vault/AWS SM.: The External Secrets Operator (ESO) syncs secrets from external stores (Vault, AWS Secrets Manager, GCP Secret Manager) into Kubernetes Secrets. The external store is the source of truth.
• Source-of-truth outside K8s.: The actual secret values live outside Kubernetes. Kubernetes Secrets are projections of the external source; rotations happen in the source; ESO propagates.
• Centralized management.: The team's secret management is centralized in the external store. Rotation, audit, access controls all happen there; Kubernetes is the consumer side.
• RBAC at the source.: The external store's access controls determine who can manage secrets. Vault policies, AWS IAM all apply; the controls are richer than Kubernetes Secret RBAC.
• Multi-cluster support.: One source can serve many Kubernetes clusters. The team's secret management is consistent across clusters; the external source is the single point of management.

The source approach is the foundation. Without it, secret management is per-cluster; with it, it is centralized.

Refresh

When secrets rotate, consumers need to pick up the new values. Two approaches: pod restart on change, or graceful in-place reload.

• Pod restarts on secret change.: When the secret changes, pods restart to pick up the new value. The restart ensures the new value is in use; the operational story is simple.
• Or app reloads gracefully.: Some applications support graceful reload of secrets. The application detects the change; reloads the secret; continues serving without restart.
• Choose by app capability.: The choice depends on the application. Stateless web apps tolerate restarts well; stateful applications benefit from graceful reload; the team picks per application.
• Reloader pattern.: The Reloader operator watches for secret changes and triggers pod restarts. The team installs Reloader once; pods inherit the rotation discipline.
• Application support for hot reload.: Applications can implement secret hot-reload natively. The application's code watches for changes; reloads the secret; updates connections. The investment is real but bounded.

The refresh discipline ensures rotation actually takes effect. Without it, rotated secrets do not reach the consumers.

Audit

Rotation events are logged. The audit trail supports compliance, postmortem investigation, and operational visibility.

• Rotation log.: Each rotation is logged with timestamp, secret name, source. The log is the audit trail; investigations reference it.
• Last rotated date per secret.: The team knows when each secret was last rotated. Secrets that have not rotated in too long are surfaced; the discipline is monitored.
• Compliance trail.: Compliance frameworks expect secret rotation evidence. The rotation log is the evidence; auditor questions have answers.
• Stale secret alerts.: Alerts fire when secrets exceed rotation thresholds. The team is notified; rotation happens; the discipline is enforced.
• Per-secret policies.: Different secrets have different rotation policies. Production credentials rotate frequently; less-sensitive secrets rotate less. The audit reflects the policy.

Secret rotation discipline is one of those security practices that pays off across many years and many secrets. Nova AI Ops integrates with secret-management tools, surfaces rotation patterns and stale secrets, and produces the audit-ready visibility that the security team uses.