Kubernetes Image Promotion Pipelines

Same image, four environments, one signed digest. Provenance-by-default, the immutable-digest rule, and the promotion pattern that drops “works in staging” bugs to zero.

Why “rebuild for prod” loses

The most expensive bug in a build pipeline is the one that says “it worked in staging.” The cause: the staging image and the production image are different artifacts. The build ran twice; the inputs drifted; one of them has a different glibc version, a different timezone db, a different anything. The bug only shows up in the binary that wasn’t tested.

The pattern that fixes it is also the simplest: build once, deploy the same artifact to every environment. The artifact is identified by its content-addressed digest (sha256:abc...), not by a tag. The whole pipeline is a sequence of promotions of the same digest through environments.

The benefit isn’t just bug reduction. Provenance, what code is running where, becomes trivially answerable. Rollback becomes deterministic; you point at the previous digest. Compliance audits move from spreadsheets to kubectl get deployment -o yaml | grep image:.

The immutable-digest rule

The core rule: production deployments reference images by digest, not by tag.

The why. Tags are mutable. app:v1.2.3 can be repushed; the digest can’t. The digest is a content hash; if the bytes change, the digest changes. Pinning to a digest means “exactly this artifact, no substitutions.”

The shape. Instead of image: myorg/app:v1.2.3, you write image: myorg/app@sha256:abc123.... Every CI build emits the digest; the deployment manifest references it. The CI pipeline updates the manifest; the GitOps controller applies it; the cluster pulls the exact bytes.

The objection. “Digests are ugly; tags are readable.” Use both: tag the image for human readability, digest for the deployment. The tag is informational; the digest is binding. The deployment YAML has the digest; the changelog has the tag-and-digest pair.

The objection 2. “Floating tags like :latest or :stable are convenient.” They’re convenient until production runs an image you didn’t mean to deploy. Floating tags belong in dev only; never in any pipeline that ends in production.

The promotion pipeline

The promotion pipeline is a sequence of stages, each of which gets the same digest applied to a different environment.

Stage 1: build and digest. CI builds the image once; pushes to the registry; captures the digest. The digest is the artifact ID; everything downstream references it.

Stage 2: deploy to dev. The dev environment gets the digest. Smoke tests run; integration tests run; if they pass, mark the digest “dev-passed.”

Stage 3: promote to staging. Same digest, staging environment. Full test suite; performance tests; canary smoke. If they pass, mark “staging-passed.”

Stage 4: promote to prod-canary. Same digest, 5% of prod traffic. Watch error rate, latency, business metrics. If clean for 30 minutes, mark “canary-passed.”

Stage 5: promote to prod-full. Same digest, 100% of prod. Done. The artifact has now passed four environment-specific verifications; the bytes haven’t changed once.

The implementation. GitOps repo with a directory per environment; each directory has a kustomization.yaml with the image digest. Promotion is a PR that copies the digest from one directory to the next. Argo CD or Flux applies; the cluster updates.

Signing and provenance

The digest gives you immutability; signing gives you authenticity. Cosign (sigstore) signs an image with a key; the cluster admission controller verifies the signature before pulling.

The shape. CI signs the digest after build (cosign sign --key cosign.key registry/app@sha256:...). The signature is stored in the registry alongside the image. At admission, Kyverno or Sigstore policy-controller verifies the signature; unsigned images are rejected.

The provenance shape. SLSA (Supply-chain Levels for Software Artifacts) provenance attaches a record to the image: which CI run built it, which commit, which builder identity. The record is signed; verifiable without trusting the CI logs. At admission, you can require “built by our CI, from our git org, on a protected branch.”

The why. Supply chain attacks are real (SolarWinds, log4shell adjacent). The chain of trust runs from source code to running pod; signing closes the gap from registry to cluster.

Admission policies

The promotion pipeline is only as strong as the cluster’s willingness to enforce it. Three admission rules.

Rule 1: digest required. Reject any pod whose image references a tag instead of a digest. Kyverno policy:

match:
  resources:
    kinds: [Pod]
validate:
  pattern:
    spec:
      containers:
        - image: "*@sha256:*"

Rule 2: signature required. Reject any pod whose image isn’t signed by a trusted key. Sigstore policy-controller does this with a few lines of YAML.

Rule 3: registry restriction. Reject any pod whose image isn’t from your trusted registry. Catches the typo, the dependency-confusion attack, the “I copied a manifest from the internet.”

The combination. Three policies; all three pass; the deployment is authentic, immutable, and traceable. Anything that gets through has been built by your pipeline, signed by your key, and pulled from your registry.

Antipatterns

Re-tagging in promotion. Build app:dev, tag-promote to app:staging, then to app:prod. The bytes are the same but the auditing is bad, you can’t tell at a glance whether prod is the same as staging. Use digests; tags are informational.

Building per environment. One CI pipeline per env, each running its own build. Inputs drift; bugs hide. Build once; promote always.

Floating tags in prod. :latest, :stable, :main in production deployments. The tag drifts the moment someone pushes; production silently runs new code. Always pin to digest in prod.

Skipping signature verification. Signing without verifying is theatre. The verification is what enforces the chain; without it, signing is decoration. Run the admission controller.

What to do this week

Three moves. (1) Audit your prod manifests for tag-based image references. Convert the top-10 deployments to digest references; the rest follow. (2) Set up Kyverno (or whatever admission controller you run) to require @sha256: in image references in prod namespaces. The lint catches future drift. (3) Add cosign signing to the CI build, it’s 5 lines. Verification can come later; the signing has to happen first to have anything to verify against.