Terraform State at Scale: Locking, Splitting, and Surviving

One Terraform state file per company is fine until five engineers want to ship at once. The patterns that scale state from a single file to a hundred without losing your weekend.

When one state stops working

One state file works for the first six months. Then: plan times exceed five minutes, two engineers can't apply at the same time, a corrupted state takes the whole infra hostage. Symptom triggers the split.

The signals that one state has stopped working. terraform plan takes more than 5 minutes. State locking blocks the team during normal work. Apply failures leave state inconsistent and hard to recover. Each is a sign that the single state has outgrown its design point.

The reason teams delay splitting. The split is one-time work; until it's done, the team feels productive enough. Splitting is "ops work that's not building features." Most teams delay until the pain is unbearable; the right move is to anticipate the pain and split before it happens.

When to split, and how

Three reasonable splits, each solving a different problem. The choice depends on which problem is biting hardest. Most growing teams end up with all three over time.

Split by environment

One state for prod, one for staging, one for dev. Easiest split. Doesn't help with the "two engineers shipping prod" problem, but it does mean dev mistakes stay in dev.

The blast-radius benefit. A bad terraform apply in dev can't accidentally affect prod because they're in separate states. The most common Terraform disaster, applying the wrong workspace, is structurally prevented.

The duplication cost. Each environment has its own copy of the resource definitions. Updates require touching multiple states. Most teams use Terraform modules to manage the duplication; one module definition, instantiated per environment.

Split by service

One state per microservice / per team. Engineers ship in parallel. Cost: cross-service references become data-source lookups; some shared resources (VPC, IAM) need to live in their own foundation state that everyone reads.

The parallelism benefit. 10 services × 10 engineers all running terraform plan simultaneously without waiting on each other. The single-state team has serialised plans; the per-service team has parallel plans. Velocity improves dramatically.

The cross-service reference cost. Service A needs Service B's database endpoint. With one state, it's a resource reference. With per-service, it's a data-source lookup that queries Service B's state. The first time a team encounters this, it's awkward; with patterns, it becomes routine.

Split by blast radius

Networking + IAM + DNS in one state (rarely changes, high blast radius). Application infra in another (changes daily, lower radius). Lets you put different review gates on each.

The review-gate pattern. The high-blast-radius state requires VP approval for any change; the low-blast-radius state allows engineer-level merging. The split lets the team move fast on application changes without exposing the infrastructure foundation to mistakes.

The pattern in practice. foundation state: VPC, subnets, IAM roles, DNS zones. Changed weekly at most. Two-engineer review minimum; manual approval to apply. application state: ECS services, Lambda functions, application IAM permissions. Changed daily. Standard PR review.

State locking

S3 + DynamoDB (AWS), GCS native locking, Terraform Cloud, pick one and never run with locking off. The "I'll just disable locking real quick" is how teams corrupt state into unrecoverable forms.

The corruption mechanism. Two engineers run apply at the same time without locking. Both modify state; the second overwrite wins. Resources that the first apply created but the second doesn't know about become orphaned. The state and reality drift; reconciliation is hours of manual work.

The "real quick" rationalisation. Engineer's apply is hung; they decide to bypass the lock. State gets overwritten while another apply was mid-flight. Two-week incident follows. NEVER bypass the lock; investigate why the lock exists instead.

Disaster recovery

Versioned state bucket. Cross-region replication. A documented procedure for "my state is corrupt, how do I rebuild from the last known good version." The procedure is short. Write it before you need it.

The versioning's value. State files in S3 with versioning enabled keep all historical versions. Corruption recovery is "restore the previous version", straightforward when versioning is on; impossible when it isn't.

The cross-region replication. State bucket in us-east-1 replicated to us-west-2. If us-east-1 has a regional outage, state is still readable. The team can continue applying from the replica. Without replication, regional outages block all infrastructure changes.

The DR runbook. "If state is corrupt: list versions, identify last good version, restore it, run terraform plan to verify, terraform apply if drift is acceptable." Five steps; one page; rehearsed once. The rehearsal is what makes the runbook real; without it, the first corruption is an improvisation.

Common antipatterns

Splitting too late. Plan times above 10 minutes; engineer waits compounding. The split was overdue 6 months ago. Anticipate; split before the pain.

Splitting too aggressively. Per-service split for a 10-service team produces 10 micro-states with massive cross-state coordination. Group services with shared dependencies into a single state.

Locking disabled "for development." Dev state has no lock; engineers learn to work without locks. The habit transfers to prod; corruption follows. Always lock; even in dev.

State stored in git. Terraform state has secrets in it (passwords, ARNs). Storing in git exposes them. Always use a remote backend (S3, GCS, Terraform Cloud); never commit state files.

What to do this week

Three moves. (1) Measure your current plan time. If over 5 minutes, splitting is overdue. (2) Verify state locking is enabled and working. Test by attempting two simultaneous applies; one should be rejected. (3) Document the DR runbook for state corruption. The exercise of writing it reveals what you don't have configured.