Kafka Producer/Consumer

Kafka basics: enough to understand the model, not yet enough to operate it in production.

Step 1: Run Kafka

Local Kafka is one compose file away. The bitnami image bundles Zookeeper or KRaft so you do not assemble a cluster by hand.

• Compose. docker compose up -d with the bitnami/kafka image; sample compose files ship in the official docs.
• Bootstrap. The broker advertises localhost:9092 by default; clients on the host machine connect there.
• Verify. kafka-topics.sh --bootstrap-server localhost:9092 --list returns an empty list on a fresh broker.
• Reset. docker compose down -v wipes volumes; topics and offsets are gone, useful between exercises.

Step 2: Producer

• from kafka import KafkaProducer
  producer = KafkaProducer(bootstrap_servers="localhost:9092")
  producer.send("test", b"hello")
  producer.flush()

Step 3: Consumer

from kafka import KafkaConsumer
consumer = KafkaConsumer("test", bootstrap_servers="localhost:9092")
for msg in consumer:
    print(msg.value)

Step 4: Topics and partitions

Topics and partitions are the only two concepts that matter at this stage. Get the partition model right and the rest of Kafka follows.

• Topic. A named, append-only log; producers write to it, consumers read from it.
• Partition. A shard of the topic; ordering is guaranteed within a partition, not across them.
• Parallelism. One consumer per partition is the upper bound; more consumers than partitions sit idle.
• Key-based routing. Messages with the same key land on the same partition, preserving per-key order across the topic.

Antipatterns

• Single-partition topics for high-throughput. Bottleneck.
• No consumer group config. Wrong delivery semantics.
• Local Kafka in production. Use managed.

What to do this week

Three moves. (1) Run the tutorial end-to-end on your own laptop / sandbox. (2) Apply the pattern to one production workload. (3) Document the variations you needed; share with the team.