GitOps Secret Management
Secrets in git? No. The pattern.
Idea
GitOps says "the desired state of the system is what is in git." That works beautifully for declarative resources like Deployments and Services. It runs into a wall at secrets, because the most basic security rule is "do not commit secrets to git." The reconciliation between these two principles produces the GitOps secret management pattern: encrypted secrets, decrypted only at deploy.
The two main implementations:
- Encrypted secrets in git.: The secret value is encrypted using a key that lives outside git (typically a KMS, HSM, or cloud-managed key service). The encrypted blob is committed; the plaintext never is. The deploy pipeline decrypts at apply time, using the key it has access to but engineers do not.
- SOPS (Secrets OPerationS).: The dominant tool for this pattern. SOPS encrypts individual fields in YAML/JSON files using AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault, or PGP. The committed file looks structurally normal; only the values are ciphertext. Diff-friendly and reviewable.
- Sealed Secrets.: A Kubernetes-native alternative. The Bitnami Sealed Secrets controller has a public key; engineers encrypt SealedSecret resources against it; only the controller running in the target cluster can decrypt. The encryption is asymmetric, so engineers do not need cluster access to commit secrets.
- Decrypted at deploy time.: The decryption happens inside the cluster, in the deploy pipeline, or in the operator that applies the resource. The plaintext exists only in memory of the consuming pod. It is never written to disk in plaintext, never logged, never visible to engineers.
- Audit trail in git.: Every secret rotation, every value change, every secret addition or removal goes through a PR like any other config change. The history is reviewable, blameable, and rollback-able. This is the property that makes the pattern compliant with most change-management controls.
The pattern is widely adopted because it gives you the GitOps benefits (declarative state, reviewable changes, audit trail) without putting plaintext credentials in source control. The trade-off is a real but bounded operational complexity around key management.
Rotation
Encrypting at rest in git is the floor. The ceiling is automatic rotation: secrets that change on a schedule, with no human in the loop, sourced from a secret store that is the canonical owner. The pattern that does this well is External Secrets Operator (ESO).
- External Secrets Operator pulls from Vault, AWS SM, GCP SM.: ESO is a Kubernetes operator that watches ExternalSecret resources. Each ExternalSecret references a path in HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or similar. The operator pulls the current value and creates or updates a Kubernetes Secret in the target namespace.
- Auto-rotation when the upstream changes.: When the secret store rotates a credential (typically on a schedule, but can be event-driven), ESO syncs the change into the cluster automatically. Pods that use the secret pick up the new value either on restart or via mounted volume reload. No human action required.
- Source of truth is the secret store, not git.: The encrypted-in-git pattern works for rarely-changing secrets (TLS keys, API keys with long lifetimes). The ESO pattern works for frequently-rotating secrets (database passwords, short-lived service tokens). Most production environments use both: stable secrets in git, dynamic secrets via ESO.
- Per-environment binding.: ExternalSecrets in dev reference dev-tier secret store paths. Same in prod. The git resource is identical across environments; the path it resolves to differs by namespace or cluster. This is what makes the same code deployable to multiple environments without environment-specific git branches.
- Tighter audit trail.: The secret store records every read and every rotation. ESO records every sync. The audit chain from "credential created" to "credential used by pod X at time Y" is reconstructable, which is the level of forensic readiness regulated environments require.
The combination of encrypted-in-git for static secrets and ESO for dynamic ones covers most production needs. Each pattern has a place; treating them as alternatives instead of complementary tools is a common adoption mistake.
Avoid
The shortest section is also the most important one. Whatever else you do with secrets, do not do this.
- Plaintext secrets in git.: Never. Not even in a private repo. Not even in a "we'll fix it later" branch. Not even encoded in base64 (which is encoding, not encryption, and which every scanner detects). Once a plaintext secret hits git, it is in the history forever and on every developer's laptop, and you have to assume it is compromised.
- Plaintext secrets in environment-variable files committed to git.: .env files, application.yml with passwords, terraform files with hardcoded credentials. All the same category. The fact that the file does not say "secret" in its name does not change what it is.
- Plaintext secrets in chat tools.: Slack, Teams, email. These are searchable, often retained for years, and frequently breached. A password shared in DM is a password that is no longer secret.
- Plaintext secrets in CI logs.: Echo statements, debug output, error messages that include credentials. Most CI systems retain logs for weeks. A secret printed once is a secret leaked once.
- Recovery if it happens.: Rotate the leaked credential immediately. Do not delete the commit and assume that fixed it. Git history can be restored; the secret must be considered compromised the moment it left the encrypted boundary. Rotate, then audit access, then write the postmortem.
The pattern is well-established and the tools are mature. Nova AI Ops integrates with SOPS, Sealed Secrets, and External Secrets Operator deployments, watches for accidental plaintext-secret commits via integration with secret scanners, and surfaces secret rotation cadence per service so the security and reliability teams have one view of how the credential lifecycle is actually working.