Kafka vs Pulsar

Streaming.

Overview

Kafka and Pulsar are two leading streaming platforms with different architectural choices. Kafka has the ecosystem maturity (every connector imaginable, decade of production experience, KRaft mode removing ZooKeeper); Pulsar has tiered storage natively (separates compute from storage via BookKeeper, automatic offload to S3). The right answer depends on whether ecosystem breadth or storage architecture matters more.

• Kafka: ecosystem maturity. Decade of production experience, every connector and integration imaginable, KRaft mode removed the ZooKeeper dependency. Default for established streaming workloads.
• Pulsar: native tiered storage. BookKeeper separates compute from storage; automatic offload to S3 keeps long-tail retention cheap. Default when retention spans months without retention-cost pain.
• Throughput per workload. Both scale to enormous throughput. Pulsar's separated architecture shines for variable retention; Kafka's mature partitioning shines for steady high-throughput.
• Operational fit per team. Existing Kafka expertise biases toward Kafka; greenfield streaming workloads with retention requirements bias toward Pulsar.

The approach

Workload-driven choice, per-team operational fit considered, documented rationale per platform. The discipline is making the streaming choice once with a written reason rather than running both Kafka and Pulsar in the same org.

• Workload-driven. Platform per workload. Reality drives the answer.
• Kafka for established streaming workloads. Connector ecosystem, mature operations. Default for orgs already invested.
• Pulsar for retention-heavy workloads. Tiered storage handles months of retention cheaply. Default when retention drives the architecture.
• Operational fit plus documented rationale. Team expertise considered; per-platform rationale captured. Future migrations have a paper trail.

Why this compounds

The right streaming platform compounds across years. Producer and consumer patterns align with the platform; cross-stream tooling (schema registry, observability, retention policy) gets built once and reused. By year two the choice is automatic per workload.

• Better operational fit. Platform matches team. Velocity stays high.
• Workload-driven decisions. Replaces tribal preference with documented rationale. Quality of choice improves.
• Better operational reliability. Right platform means brokers behave predictably. Incident MTTR drops.
• Year-one investment, year-two habit. First platform choice is the investment; subsequent workloads inherit the patterns.