Event-Driven Architecture: A Complete Guide

Software Design Software Design 8 min read 1524 words Beginner ExcellentWiki Editorial Team

Event-driven architecture (EDA) is a software architecture pattern where components communicate by producing and consuming events. An event is a significant change in state — “order placed,” “payment received,” “user registered.” Components react to events asynchronously, enabling loose coupling, scalability, and real-time processing.

This guide covers the fundamentals of EDA, including event sourcing, CQRS, message brokers, and practical implementation patterns.

Core Concepts

Events

An event represents something that happened in the past. Events are immutable facts — they cannot be changed or deleted. Each event has a type, a timestamp, and payload data.

{
  "eventType": "OrderPlaced",
  "eventId": "evt_abc123",
  "timestamp": "2024-01-15T10:30:00Z",
  "data": {
    "orderId": "ord_456",
    "customerId": "cus_789",
    "total": 49.99
  }
---

Producers and Consumers

Producers emit events without knowing who consumes them. Consumers subscribe to events without knowing who produced them. This decoupling is the fundamental strength of EDA.

Event Channels

Events travel through channels — message brokers like Apache Kafka, RabbitMQ, Amazon SQS/SNS, or Redis Streams. The broker stores, routes, and delivers events to subscribers.

Event Broker Patterns

Pub/Sub

In publish-subscribe, producers publish events to topics. Consumers subscribe to topics. Each event is delivered to all subscribers of the topic.

kafka_producer.produce("order_events", event)

@kafka_consumer.subscribe("order_events")
def send_confirmation(event):
    email_service.send_receipt(event.data)

@kafka_consumer.subscribe("order_events")
def update_inventory(event):
    inventory_service.reduce_stock(event.data.orderId)

Competing Consumers

In competing consumers, multiple consumer instances compete for messages from a queue. Each message is processed by exactly one consumer. This enables parallel processing and horizontal scaling.

Apache Kafka

Kafka is the most popular event streaming platform. It is designed for high throughput, durability, and replayability.

Key Concepts

Topic — A category of events (e.g., “order_events”)
Partition — A topic is split into partitions for parallelism
Offset — Each message has a sequential ID within its partition
Broker — A Kafka server
Consumer Group — A group of consumers that coordinate to process partitions

Producing Events

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

producer.send('order_events', {
    'orderId': 'ord_456',
    'status': 'placed'
---)
producer.flush()

Consuming Events

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'order_events',
    bootstrap_servers=['localhost:9092'],
    group_id='notification-service',
    auto_offset_reset='earliest'
)

for message in consumer:
    process(message.value)

Event Sourcing

Event sourcing stores state as a sequence of events. Instead of storing the current state (e.g., “current balance: $100”), you store every event that led to that state (e.g., “deposited $50,” “withdrew $20,” “deposited $70”).

Benefits

Complete audit trail — Every change is recorded
Time travel — Reconstruct past state by replaying events up to a point
Debugging — Reproduce bugs by replaying events in a development environment

Snapshotting

Replaying all events from the beginning becomes slow over time. Periodically create snapshots — save the current state and the event offset. Replay from the latest snapshot, not from the beginning.

CQRS (Command Query Responsibility Segregation)

CQRS separates read and write models. Commands (writes) use one model; queries (reads) use a different, optimized model.

When to Use CQRS

Read and write workloads have different shapes
The read model needs to aggregate data from multiple sources
You are already using event sourcing

Saga Pattern

A saga manages a distributed transaction across services. Instead of ACID transactions, a saga uses a sequence of local transactions with compensating actions for rollback.

Choreography-Based Saga

Each service produces events that trigger the next step (or compensating step):

# Order service emits OrderPlaced
# Payment service listens, processes payment, emits PaymentProcessed
# Inventory service listens, reserves stock, emits StockReserved
# If stock cannot be reserved, emit StockReservationFailed
# Payment service listens, issues refund (compensating action)

Orchestration-Based Saga

A central orchestrator tells services what to do:

class OrderSagaOrchestrator:
    def execute(self, command):
        payment_result = payment_service.process_payment(command)
        if payment_result.is_failure:
            return SagaResult.failure()
        
        inventory_result = inventory_service.reserve_stock(command)
        if inventory_result.is_failure:
            payment_service.refund(command)
            return SagaResult.failure()
        
        notification_service.notify(command)
        return SagaResult.success()

Event Sourcing and CQRS Integration

Command Query Responsibility Segregation pairs naturally with event sourcing. Commands validate and append events to the write store. Queries read from optimized read models (projections) that are rebuilt from the event stream. The two sides can use different data stores optimized for their access patterns. CQRS without event sourcing is also valid — simply separate read and write interfaces without maintaining an event log.

Schema Evolution

Events live for years. Manage schemas carefully with Avro, Protobuf, or JSON Schema. Use a schema registry to enforce compatibility rules — backward compatibility allows consumers to read old events, forward compatibility allows producers to write new event versions without breaking consumers. Never delete or rename fields; mark them as deprecated instead.

Challenges

Eventual consistency — The read model may lag behind writes
Duplicate events — Consumers must be idempotent (processing the same event twice has the same effect)
Schema evolution — Events live long; manage schemas carefully (Avro, Protobuf with schema registry)
Debugging complexity — Tracing an asynchronous flow is harder than a synchronous one

Conclusion

Event-driven architecture enables loosely coupled, scalable, and real-time systems. Start simple — use a message queue for one integration point, like sending notifications when orders are placed. Add event sourcing and CQRS only when you need audit trails or read/write separation. Kafka is the standard for high-throughput event streaming, but RabbitMQ or Redis Streams work well for lower-scale needs.

FAQ

When should I use event sourcing? Use event sourcing when auditability is critical (financial systems, compliance tracking) or when you need to reconstruct past state (temporal queries). Avoid it for simple CRUD applications where current state is all you need.

What is the difference between event sourcing and CQRS? Event sourcing stores state changes as an event log. CQRS separates read and write models. They are independent patterns that work well together but can be used separately.

How do I handle duplicate events? Make consumers idempotent. Track processed event IDs and skip duplicates. Use exactly-once processing where available (Kafka transactions, idempotent producers).

What are the alternatives to Kafka? RabbitMQ for lower-throughput, routing-heavy workloads. Redis Streams for in-memory performance and simple deployments. Amazon SQS/SNS for fully managed cloud infrastructure. NATS for ultra-low latency messaging.

Eventual Consistency Patterns

Event-driven architectures inherently embrace eventual consistency — after an event is published, downstream consumers may not process it immediately. This is a deliberate trade-off for scalability and availability (following the CAP theorem). Key patterns for managing eventual consistency:

Outbox Pattern — When a service needs to update a database and publish an event atomically, it writes both the entity change and the event to an outbox table in the same database transaction. A separate process reads the outbox and publishes events to the message broker. This ensures at-least-once delivery without distributed transactions.

Idempotent Consumers — Since events may be delivered more than once, consumers must handle duplicates. Assign each event a unique ID, and track processed event IDs in the consumer. Skip events that have already been processed. This pattern allows at-least-once delivery semantics without data corruption.

Saga Pattern — A saga is a sequence of local transactions where each step publishes an event that triggers the next step. If a step fails, compensating transactions undo the previous steps. Sagas provide data consistency across services without distributed transactions. Choreographed sagas use event publication to coordinate; orchestrated sagas use a central coordinator.

Message Ordering and Partitioning

Message brokers like Kafka guarantee ordering within a partition but not across partitions. Choose your partition key carefully — all messages with the same key go to the same partition, preserving order. Common partition keys include entity ID (all events for an order go to the same partition) or customer ID (all events for a customer are ordered). For most business workflows, per-entity ordering is sufficient.

Event Format Versioning

Events are contracts between producers and consumers. Schema evolution requires care. Use a schema registry (Confluent Schema Registry, Apicurio) to manage event schemas and validate compatibility. Follow the rule of additive-only changes: add new fields with defaults, never remove or rename existing fields. When breaking changes are unavoidable, create a new event type and maintain both versions during a migration period.

Event Sourcing vs Event-Driven

Event-driven architecture publishes events as notifications — services react to events but the current state is still stored in databases. Event sourcing stores all state changes as a sequence of events, and the current state is derived by replaying those events. Event sourcing provides a complete audit trail, enables temporal queries (state at any point in time), and eliminates object-relational impedance mismatch. The trade-off is operational complexity — event store management, snapshot handling, and schema evolution for historical events.

Dead Letter Queues

Messages that cannot be processed successfully (after retries) are moved to a dead letter queue (DLQ). This prevents poison messages from blocking the main queue. Monitor DLQ depth as a health metric — a growing DLQ indicates systemic issues that need investigation. Set up alerts for DLQ growth and automate DLQ reprocessing after the underlying issue is resolved.

CQRS (Command Query Responsibility Segregation)

CQRS separates read and write models — commands (writes) use one model, queries (reads) use another. This allows optimizing each model independently: writes use a normalized, transactional model; reads use denormalized views optimized for specific queries. CQRS is often paired with event sourcing, where event handlers update read models. Apply CQRS selectively to parts of the system with different read/write characteristics, not as an architecture-wide mandate.

Share this article

X LinkedIn Facebook Email