CPU-Bound vs IO-Bound

Different optimisations.

Overview

CPU-bound versus IO-bound is the most fundamental performance distinction. CPU-bound workloads spend their time in computation (encoding, ML inference, math); the threads block on cycles, and parallelism (more cores, more processes) helps. IO-bound workloads spend their time in syscalls (network, disk); the threads block waiting for data, and concurrency (async, event loops, more threads) helps. Picking the wrong optimization produces wasted engineering effort that does not move the needle.

• Different optimizations. CPU-bound benefits from parallelism; IO-bound benefits from concurrency; the right tool depends on the bottleneck.
• CPU-bound. Compute-heavy: encoding, ML inference, hash computation; threads block on cycles, not syscalls.
• IO-bound. Network or disk-heavy: web servers, database clients, log processors; threads block on syscalls, not cycles.
• Profiling reveals plus tooling differs. CPU profile vs syscall trace identifies the bottleneck; threads and processes for CPU, async and event loops for IO.

The approach

The practical approach is to profile first (CPU profile if you suspect compute, syscall trace if you suspect IO), match the tooling to the bottleneck (parallelism for CPU, concurrency for IO), recognise mixed workloads need hybrid approaches (async for IO with a thread pool for CPU work), and document the per-service bottleneck profile so future optimization targets the right surface.

• Profile first. CPU profile reveals computation hotspots; syscall trace reveals IO waits; the data tells you which optimization applies.
• CPU-bound: threads/processes. Parallelism uses available cores; the bottleneck moves from one core to memory bandwidth eventually.
• IO-bound: async/event loop. Concurrency uses idle time during syscall waits; one thread can handle thousands of concurrent IO operations.
• Mixed workload: hybrid plus documented profile. Async for IO, thread pool for CPU work; per-service bottleneck profile committed to the repo.

Why this compounds

The CPU-vs-IO discipline compounds across services. Each correctly-matched optimization produces ongoing performance gains where the bottleneck actually was; each profile teaches the team where the bottleneck hides; the team builds intuition for performance work that pays off on every new service.

• Performance. Right tooling for the bottleneck; the optimization moves the metric the team cares about.
• Cost efficiency. Right scale for the workload; CPU-bound scales horizontally with cores, IO-bound scales with concurrency.
• Engineering culture. Profile-first culture produces evidence-based decisions; the team optimizes what the data shows.
• Institutional knowledge. Each profile teaches workload patterns; the team learns which workloads bottleneck on which surface.

The CPU-vs-IO distinction is an engineering discipline that pays off across years. Nova AI Ops integrates with performance telemetry, surfaces bottleneck patterns, and supports the team’s optimization discipline.