Distributed Tracing: Jaeger, Zipkin, and OpenTelemetry

Distributed tracing tracks a single request as it flows through multiple services in a distributed system. While metrics tell you something is slow and logs tell you something failed, traces tell you exactly which service, database call, or queue operation caused the problem.

The Problem Distributed Tracing Solves

In a monolith, a single function call stack traces a request from entry to exit. In a microservice architecture, a single request may touch 10, 20, or 50 services, each with its own logs and metrics. When a request is slow or fails, correlating the data across services is nearly impossible with logs alone.

Distributed tracing solves this by assigning every request a unique trace ID and recording spans — named, timed operations — as the request passes through each service. A trace is the complete tree of spans for one request.

Core Concepts

Trace: /checkout
├── Span: API Gateway (2ms)
│   ├── Span: Auth Service (1ms)
│   │   └── Span: Redis session lookup (0.5ms)
│   ├── Span: Cart Service (3ms)
│   │   ├── Span: DB query: get_cart (1ms)
│   │   └── Span: RPC: inventory check (1.5ms)
│   └── Span: Order Service (4ms)
│       ├── Span: DB insert: create_order (1ms)
│       └── Span: Queue: send_confirmation (0.5ms)

OpenTelemetry

OpenTelemetry (OTel) is the industry standard for distributed tracing. It provides a unified API and SDK for generating, collecting, and exporting telemetry data (traces, metrics, logs).

Instrumentation

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Configure the tracer
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

# Create a span
with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("order.total", total)

    # Nested span — automatic parent relationship
    with tracer.start_as_current_span("validate_inventory") as child_span:
        child_span.set_attribute("product_count", len(items))
        validate_inventory(items)

Auto-Instrumentation

For common frameworks, OTel provides auto-instrumentation packages:

# No code changes needed
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

Auto-instrumentation wraps library calls — HTTP requests, database queries, gRPC calls — and creates spans with appropriate attributes, context propagation, and timing.

Context Propagation

The trace context must be propagated across service boundaries. OpenTelemetry uses the W3C Trace Context standard:

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: vendor=some-data

• trace-id (32 hex chars) — globally unique trace identifier
• span-id (16 hex chars) — current span identifier
• trace-flags — sampling decision (01 = sampled, 00 = not sampled)

Jaeger

Jaeger is a distributed tracing backend originally built by Uber. It ingests spans, stores them, and provides a UI for querying and visualizing traces.

Architecture

Application → Jaeger Agent → Jaeger Collector → Storage (Elasticsearch/Cassandra) → Jaeger Query → UI

Running Jaeger

# docker-compose.yaml for Jaeger all-in-one
version: "3"
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
      - "14250:14250"   # Jaeger gRPC
    environment:
      - COLLECTOR_OTLP_ENABLED=true

Querying Traces

The Jaeger UI (http://localhost:16686) supports:

• Search by service, operation, tags, time range
• Trace detail view — waterfall diagram with span timing, tags, and logs
• System architecture — service dependency graph generated from trace data
• Compare traces — side-by-side comparison of two traces (useful for finding what differs between fast and slow requests)

Zipkin

Zipkin is an earlier distributed tracing system, originally by Twitter. It uses similar concepts but with its own data model and APIs.

Zipkin Integration

from py_zipkin import zipkin

@zipkin(service_name="order_service", span_name="create_order")
def create_order(order_data):
    with zipkin.new_span("validate_payment") as span:
        span.add_binary_annotation("payment.method", order_data["method"])
        validate_payment(order_data)

Zipkin vs Jaeger

| Feature | Jaeger | Zipkin | |

Sampling Strategies

Tracing every request is expensive and often unnecessary. Sampling reduces storage and compute costs:

# Probabilistic sampler — 10% of traces
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

provider = TracerProvider(
    sampler=TraceIdRatioBased(0.1)
)

Tail-Based Sampling

Tail-based sampling keeps all traces temporarily and samples based on the completed trace. This ensures important traces — errors, high latency, specific user cohorts — are always captured:

// Jaeger tail-based sampling policy
{
  "sampling": {
    "strategies": [
      {
        "type": "probabilistic",
        "param": 0.1
      },
      {
        "type": "rate_limiting",
        "param": 100
      }
    ],
    "tail_sampling": {
      "policies": [
        {"name": "errors", "type": "status_code", "status": "ERROR"},
        {"name": "slow",   "type": "latency",   "threshold": "2s"}
      ]
    }
  }
---

Observability Triad

Distributed tracing completes the observability triad:

Connecting the three is essential. A Grafana dashboard shows latency spikes (metrics). Drill down to the affected trace (traces). Open the span logs to see the exact error message (logs). OpenTelemetry bridges all three with a single data model and correlation mechanisms:

{
  "traceId": "0af7651916cd43dd8448eb211c80319c",
  "spanId": "b7ad6b7169203331",
  "body": "Connection timeout connecting to database",
  "severity": "ERROR",
  "attributes": {
    "db.system": "postgresql",
    "db.connection_id": "conn-42"
  }
---

The trace ID in log entries links directly to the trace in Jaeger, making root-cause analysis a single click from any log view.

Distributed Tracing Fundamentals

Trace Context Propagation

Distributed tracing requires propagating trace context across service boundaries. Each request is assigned a trace ID at the entry point (API gateway, load balancer). As the request flows through services, each service creates spans — named operations with start and end timestamps. The trace context (trace ID, parent span ID, and propagation metadata) is passed via HTTP headers (W3C Trace-Context format, Zipkin B3 headers) or message queue metadata. Standardized propagation ensures interoperability between services using different instrumentation libraries. OpenTelemetry provides vendor-agnostic APIs and SDKs for instrumentation in multiple languages, with automatic context propagation through HTTP, gRPC, and messaging libraries. Without proper propagation, traces are fragmented into disconnected spans, making it impossible to reconstruct the full request path.

Sampling Strategies

Collecting every trace in a high-throughput system generates enormous volumes of data. Head-based sampling makes the sampling decision at the request entry point — typically keeping 1-10% of all traces. This approach is simple but may miss important low-traffic traces. Tail-based sampling makes decisions after spans complete, using processors that analyze complete traces before deciding whether to store them. Tail sampling can keep all error traces, all traces exceeding latency thresholds, and a representative sample of successful traces. Probabilistic sampling with consistent hashing ensures that all spans for the same trace are sampled or not sampled together — critical for trace completeness. Adaptive sampling adjusts sampling rates based on system load and error rates, reducing sampling during normal operation and increasing during incidents for diagnostic data.

Identifying Performance Bottlenecks

Distributed traces enable precise identification of performance bottlenecks through span analysis. The Gantt-chart visualization shows each span’s duration relative to the total trace duration, immediately revealing which service or operation is the bottleneck. Hot spots appear as spans consuming a disproportionate share of trace duration. Repeated patterns across traces — a particular database query that is consistently slow, a downstream API with high latency variance, or a serialization step that grows with payload size — become statistically apparent when aggregating span data. Critical path analysis identifies the single chain of spans that determines the overall trace duration — optimizing spans outside the critical path does not improve response time. Service dependency graphs built from span data reveal architectural anti-patterns like synchronous deep call chains and unnecessary intermediate services.

Frequently Asked Questions

What is the difference between tracing and logging? Tracing tracks the end-to-end path of a single request through distributed services, showing timing and relationships between operations. Logging records discrete events with timestamps but does not inherently relate events from different services. Tracing is essential for understanding latency in distributed systems; logging is essential for understanding detailed state within a single service.

Do I need distributed tracing for a monolith? For a monolithic application running on a single process, traditional profiling and logging are usually sufficient. Consider distributed tracing when you have multiple services, microservices, or serverless functions that handle a single request. Even in monoliths, tracing across database calls and external API calls can provide valuable insights.

What is OpenTelemetry and why is it important? OpenTelemetry (OTel) is an open standard for observability that provides vendor-agnostic APIs, SDKs, and instrumentation libraries. It unifies metrics, logs, and traces under a single specification. OTel is important because it eliminates vendor lock-in — you can instrument once and export to any OTel-compatible backend (Jaeger, Grafana Tempo, Datadog, Honeycomb) without changing instrumentation code.

For a comprehensive overview, read our article on Blue Green Deployments.

For a comprehensive overview, read our article on Ci Cd Pipeline Guide.