Streaming Data with Apache Kafka: Real-Time Pipelines Guide

Streaming data is data generated continuously by many sources. Apache Kafka is the dominant platform for building real-time data pipelines. According to Confluent’s 2024 Data Streaming Report, 72% of organizations now use event streaming for at least one production workload, with Kafka powering the majority of deployments. From clickstream analytics to fraud detection and IoT data processing, Kafka has become the standard infrastructure for event-driven architectures.

What Is Apache Kafka?

Apache Kafka is a distributed event streaming platform designed for high-throughput, fault-tolerant, real-time data feeds. Originally developed at LinkedIn to handle their massive clickstream data, it was open-sourced in 2011 and has since become the standard infrastructure for event-driven architectures across thousands of organizations.

Key characteristics include: a publish-subscribe messaging model with durable event persistence on disk, horizontal scalability across many broker nodes, and the ability to handle millions of events per second with sub-millisecond latency. Unlike traditional message queues, Kafka stores events durably and allows multiple consumers to read the same events independently, making it suitable for both streaming and batch processing use cases.

Core Concepts

Topics and Partitions

A topic is a category of events — orders, user signups, page views, sensor readings. Each topic is divided into partitions, which are ordered, immutable sequences of events. Partitions are the unit of parallelism in Kafka:

Topic: "orders"
  +-- Partition 0 (on broker 1)
  +-- Partition 1 (on broker 2)
  +-- Partition 2 (on broker 3)

Each event within a partition has a sequential ID called an offset, which consumers use to track which events they have processed. Events are retained for a configurable period (default 7 days) or until a size threshold is reached, after which they are eligible for deletion. This durability enables replay: consumers can rewind to any offset and reprocess events.

Partitions are distributed across brokers to provide fault tolerance through replication. The replication factor determines how many copies of each partition are maintained across different brokers. If a broker fails, another broker with a replica of the affected partitions takes over, ensuring continuous availability.

Producers

Producers write events to topics. They can specify a partition key — all events with the same key go to the same partition, ensuring order for a specific entity like a customer ID or order ID:

producer.send('orders', key=order_id, value=order_data)

Producers can configure acknowledgment levels to balance durability with latency: acks=0 (fire-and-forget, fastest but no delivery guarantee), acks=1 (leader acknowledges write, good default), or acks=all (all in-sync replicas acknowledge, strongest durability but highest latency). For production systems handling critical data, acks=all with min.insync.replicas=2 is the recommended configuration.

Consumers and Consumer Groups

Consumers read events from topics. They belong to consumer groups, which enable load-balanced consumption across multiple consumer instances:

consumer = KafkaConsumer('orders', bootstrap_servers='localhost:9092',
                         group_id='analytics')
for message in consumer:
    process_order(message.value)

Each partition is assigned to exactly one consumer within a group. If a consumer fails, the group coordinator detects the failure through missed heartbeats and triggers a rebalance, reassigning the failed consumer’s partitions to the remaining members. This provides fault tolerance and automatic load balancing. The number of consumers in a group determines the maximum parallelism — you can have at most as many active consumers as partitions in the topic.

Consumer offset management can be automatic (committed periodically with enable.auto.commit=true) or manual (committed explicitly after processing). Manual offset management ensures exactly-once processing semantics at the cost of more complex code, as offsets are committed only after the corresponding events have been fully processed and their results persisted.

Event-Driven Architecture Patterns

Event Sourcing

Event sourcing stores the full sequence of state-changing events rather than just the current state. Each state change is appended to an event log, and the current state is derived by replaying all events:

OrderCreated → PaymentReceived → ItemShipped → ItemDelivered

Any point in time can be reconstructed by replaying events up to that point. This provides a complete audit trail of all changes, enables temporal queries, and allows building multiple read models from the same event stream. The trade-off is that querying current state requires either replaying all events (computationally expensive for long-lived entities) or maintaining a materialized view that is updated incrementally as new events arrive.

CQRS

Command Query Responsibility Segregation separates write models from read models. Commands update the primary store and emit events through Kafka. Read models are updated asynchronously by consuming those events, providing eventually consistent views optimized for different query patterns:

Command (write) → Primary Store → Kafka Event → Read Model 1 (analytics)
                                              → Read Model 2 (search)
                                              → Read Model 3 (UI)

CQRS enables independent scaling of read and write workloads, different data models for different query patterns, and the ability to add new read models without modifying the write path. It is commonly used with event sourcing to build scalable, event-driven systems.

Stream Processing

Stream processing transforms, aggregates, and analyzes events as they arrive. Kafka Streams provides a lightweight Java library for building stream processing applications directly on Kafka topics without requiring a separate processing cluster:

KStream<String, Order> orders = builder.stream("orders");
orders
    .filter((key, order) -> order.value() > 100)
    .to("high-value-orders");

Alternatives include Apache Flink for complex event processing and large-scale stateful operations, ksqlDB for SQL-based stream processing that lowers the barrier for analysts, and Spark Streaming for micro-batch processing that integrates with the broader Spark ecosystem.

Schema Management

A schema registry ensures producers and consumers agree on the format of events. The registry stores schemas, validates compatibility between schema versions, and provides schema IDs that producers and consumers reference in message headers. This prevents serialization errors and enables schema evolution without breaking downstream consumers:

• Avro: The most common choice — compact binary format with rich schema evolution capabilities including default values and field ordering
• Protobuf: Google’s serialization format, efficient and widely adopted in microservice architectures, with strong typing and code generation
• JSON Schema: For JSON-based events, easier to debug and inspect but larger message size and slower serialization

Compatibility modes control what schema changes are allowed: BACKWARD (new schema can read data written with the old schema), FORWARD (old schema can read data written with the new schema), and FULL (both directions). Most production systems use BACKWARD or FULL compatibility to ensure that schema changes do not break existing consumers.

Production Deployment

Key Configuration Settings

Common Production Patterns

• Dead letter queue: Route events that fail processing to a separate topic for investigation, ensuring the main pipeline continues processing without interruption
• Event bridge: Connect systems without direct coupling — each system produces and consumes events independently through shared topics
• Change data capture: Stream database changes to downstream systems using Debezium or Kafka Connect, providing real-time replication without modifying source applications

Managed Kafka Services

• Confluent Cloud: Fully managed Kafka with Kora engine, auto-scaling, Schema Registry, and ksqlDB for SQL-based stream processing
• Amazon MSK: AWS-managed Kafka with automatic patching, monitoring with CloudWatch, and VPC integration
• Red Hat AMQ Streams: Kafka on Kubernetes using the Strimzi operator, providing declarative cluster management via custom resources

When to Use Streaming

Use streaming when you need real-time or near-real-time processing (sub-second to sub-minute latency), data arrives continuously and unpredictably rather than in scheduled batches, you need to decouple producers from consumers for independent scaling, or you want an immutable audit log of all events for replay and debugging. Use batch processing when data arrives in scheduled batches, sub-second latency is not required, or simplicity is more important than real-time capabilities.

Frequently Asked Questions

What is the difference between Kafka and a message queue like RabbitMQ? Kafka is designed for high-throughput, persistent event streaming with replay capability — events are stored durably and multiple consumers can read them independently at their own pace. RabbitMQ is designed for message queuing with complex routing patterns and immediate delivery. Kafka stores events on disk for a configurable retention period; RabbitMQ deletes messages after they are consumed. Use Kafka for event streaming and data pipelines; use RabbitMQ for task queues and request-response patterns.

How do I ensure exactly-once processing in Kafka? Exactly-once semantics require three conditions working together: idempotent producers (enable with enable.idempotence=true), transactional writes (using beginTransaction and commitTransaction APIs), and consumer offset management that commits offsets only after processing is complete, within the same transaction as the output. Kafka 0.11+ supports this through the idempotent producer and transactions API. For many use cases, at-least-once processing with idempotent downstream consumers is a simpler and equally effective alternative.

What happens when a consumer crashes? The consumer group coordinator detects the failure through missed heartbeat signals (configured via session.timeout.ms and heartbeat.interval.ms). It triggers a rebalance, reassigning the crashed consumer’s partitions to the remaining group members. The new consumers start processing from the last committed offset, which may result in some duplicate processing if the crashed consumer processed events but did not commit offsets before failing. Enable automatic offset commits with careful tuning to minimize the duplicate processing window.

How do I choose the number of partitions for a topic? Partitions determine maximum parallelism — you can have at most as many active consumers as partitions. Consider your throughput requirements (each partition can handle tens of MB/s), the ordering guarantees you need (events within a partition are ordered, across partitions they are not), and the overhead of partitions (each partition adds metadata overhead and file handles). A general rule is to start with 3x the number of expected consumers and monitor performance, adjusting as needed.

Can Kafka lose data? With proper configuration, Kafka provides strong durability guarantees. Set replication.factor to 3, min.insync.replicas to 2, and use acks=all on producers. With these settings, a produced event is guaranteed to be persisted on at least two brokers before the producer receives an acknowledgment. Kafka can survive individual broker failures without data loss as long as one in-sync replica remains available for each partition.

Data Warehousing Guide — ETL Pipelines Guide — Data Orchestration Guide