Secrets Management: Vault vs Cloud KMS vs Kubernetes Secrets

Three popular ways to handle secrets, three different operational models. Picking the wrong one isn't catastrophic, it just costs you a year of pain you didn't need.

Three patterns

HashiCorp Vault, cloud-native KMS (AWS Secrets Manager / GCP Secret Manager / Azure Key Vault), and Kubernetes-native secrets with sealed-secrets or external-secrets-operator. Different trade-offs.

The three patterns reflect three different operational philosophies. Vault: full control, full responsibility. Cloud KMS: convenience, vendor lock-in. Kubernetes-native: ecosystem-native, smaller surface. Each is right for some teams; the choice depends on where the team's complexity tolerance is.

HashiCorp Vault

Most powerful: dynamic secret generation, ad-hoc DB credentials, transit encryption, audit log built in. Heaviest operationally, Vault itself becomes a tier-zero service you're now responsible for.

The dynamic-secrets feature. Application requests a database credential from Vault; Vault generates one on demand, with a short TTL (e.g., 1 hour); application uses it; Vault revokes after TTL. No long-lived database credentials anywhere. Compromise becomes time-bounded.

The operational burden. Vault is a tier-zero service. Its outage means every application that needs a secret is broken. High availability requires multiple Vault nodes with consensus (Raft). Backup/restore is non-trivial because the encrypted data needs the unseal keys. Most teams underestimate the ongoing operational cost.

When Vault is right. Teams with mature platform engineering (5+ engineers dedicated to platform). Compliance environments requiring audit logs. Multi-cloud deployments where cloud-native KMS is impractical. Each is a setting where the operational cost is amortised across enough value.

Cloud KMS

Easiest start: managed by your cloud, IAM-integrated, no separate service to run. Lock-in increases as you use deeper features. Cost scales with API calls; for a high-throughput service this can sting.

The IAM integration is the core benefit. Pod's IAM role allows reading specific secrets; SDK reads them transparently; no application code knows the secret's location. Compared to Vault (which requires its own auth), the IAM-native approach is much simpler.

The lock-in cost. AWS Secrets Manager APIs are AWS-specific. Code written against them works only on AWS. Migrating to GCP requires rewriting. For multi-cloud teams, this is a meaningful constraint; for AWS-only teams, it's not.

The cost scaling. AWS Secrets Manager charges $0.40/secret/month plus $0.05 per 10k API calls. For a service reading the same secret 100 million times per month, the API costs are real. Most teams cache secrets in-memory after first read; the cache's TTL determines the API cost.

Kubernetes secrets (with help)

Native, in-cluster, but base64 (not encrypted at rest by default). Add sealed-secrets or external-secrets-operator and it becomes a real solution. Best for cluster-local secrets; weaker for cross-cluster needs.

The sealed-secrets pattern. Engineer encrypts a secret on their laptop using the cluster's public key; commits the encrypted blob to git; on cluster, the operator decrypts using the cluster's private key. Secrets are git-friendly; the blob is opaque to anyone without the cluster's private key.

The external-secrets pattern. The cluster pulls secrets from an external store (Vault, AWS Secrets Manager) and presents them as native Kubernetes Secrets. Best of both worlds: the secret lives outside the cluster; pods read it as a normal K8s secret. Most teams converge on this pattern.

The key rotation conversation

Whatever you pick, rotation needs to be automatic. Manual rotation gets postponed; postponed becomes 3 years; 3 years is how the eventual breach gets old keys. Vault dynamic secrets do this naturally; cloud KMS needs custom Lambda; Kubernetes secrets without operators rotate by hand.

The operational reality of manual rotation. Senior engineer remembers the rotation procedure. They leave; documentation is incomplete; new engineers don't know how. Quarterly rotation becomes annual; annual becomes biennial; biennial becomes "we should rotate, but it's been so long..." The drift is invisible until a breach.

The automation requirement. Whatever pattern is chosen, automation must be part of it. AWS Secrets Manager has automatic-rotation Lambda templates. Vault has dynamic secrets. K8s external-secrets-operator can refresh secrets on a schedule. Without one of these, manual rotation will fail.

The hybrid

Cloud KMS for cloud service credentials, Vault for application secrets, Kubernetes secrets (synced via external-secrets) for cluster-local config. Most enterprises end up here. Looks complex on a diagram; works cleanly when each is doing what it's good at.

The integration story. Cloud KMS holds master credentials (database admin, S3 access keys). Vault uses those master credentials to generate dynamic, short-lived application credentials. K8s external-secrets pulls the dynamic credentials and presents them to pods. Each layer does what it's good at.

The operational discipline. The hybrid works when each layer's responsibility is clear. Engineers know: "API keys live in cloud KMS." "Database creds are dynamically generated by Vault." "Config values are in K8s secrets via external-secrets." Without this clarity, secrets end up duplicated across systems and rotation becomes confused.

Common antipatterns

Secrets in environment variables in deployment YAML. Plain-text in git. Anyone with repo access has the secrets. Use any of the three patterns above; never commit plaintext secrets.

The "I'll rotate it manually" promise. Manual rotation never happens. Always automate.

Vault on the same nodes as applications. Vault outage takes down everything; cluster issue takes down Vault. Run Vault on dedicated infrastructure; treat it as tier-zero.

The "all secrets in one place" approach. Single secret store for everything; outage takes down everything. Diversify: cloud KMS for some categories, Vault for others. Failure of one doesn't cascade.

What to do this week

Three moves. (1) Audit your secrets for plaintext exposure. Search the repo for password=, key=, token=. Most teams find at least a few that escaped review. (2) For each system that holds secrets, verify automatic rotation is configured. Manual rotation will fail. (3) If you don't have a clear pattern, pick one based on your scale and complexity tolerance. Cloud KMS for most teams; Vault if you've outgrown it.