Monitoring and Observability: Prometheus, Grafana, and ELK Stack

DevOps DevOps 8 min read 1577 words Beginner ExcellentWiki Editorial Team

Modern distributed systems generate vast amounts of telemetry data. Monitoring and observability help you understand what your systems are doing, why they fail, and how to improve them. While monitoring tells you something is broken, observability helps you figure out why.

This guide covers the three pillars of observability — metrics, logs, and traces — and the tools that implement them: Prometheus, Grafana, and the ELK Stack.

The Three Pillars of Observability

      ┌──────────────────┐
      │   Observability   │
      ├────────┬────────┬─┤
      │Metrics │  Logs  │ │
      │(What)  │(Why)   │ │
      │        │        │ │
      │ Prom-  │ ELK    │T│
      │etheus  │ Stack  │r│
      │Grafana │        │a│
      │        │        │c│
      │        │        │e│
      └────────┴────────┴─┘

Metrics — numerical measurements over time (CPU, requests, latency)
Logs — timestamped events with structured or unstructured messages
Traces — end-to-end request paths through distributed services

Prometheus: Metrics Collection

Prometheus is a pull-based monitoring system that scrapes metrics from targets at configured intervals.

Architecture

Service A ──┐
Service B ──┼──► Prometheus ──► Grafana
Service C ──┘       │
                    ▼
               Alertmanager ──► Email, Slack, PagerDuty

Exporting Metrics

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']  # Node exporter

  - job_name: 'app'
    static_configs:
      - targets: ['app:8080']

  - job_name: 'kubernetes'
    kubernetes_sd_configs:
      - role: pod

Instrumenting Applications

# Python with prometheus_client
from prometheus_client import Counter, Histogram, start_http_server
import time

REQUESTS = Counter('http_requests_total', 'Total HTTP requests',
                   ['method', 'endpoint'])
LATENCY = Histogram('http_request_duration_seconds',
                    'HTTP request latency in seconds',
                    ['method'])

@REQUESTS.labels(method='GET', endpoint='/api/users').inc()
def handle_request():
    with LATENCY.labels(method='GET').time():
        # Process request
        time.sleep(0.1)

// Go with Prometheus client
package main

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "net/http"
)

var (
    httpRequests = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "endpoint"},
    )
    httpDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name: "http_request_duration_seconds",
            Help: "HTTP request duration",
        },
        []string{"method"},
    )
)

func init() {
    prometheus.MustRegister(httpRequests)
    prometheus.MustRegister(httpDuration)
---

func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8080", nil)
---

PromQL (Prometheus Query Language)

# Rate of requests per second over 5 minutes
rate(http_requests_total[5m])

# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# CPU usage by instance
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Alert if error rate > 5%
rate(http_requests_total{status=~"5.."}[5m])
/
rate(http_requests_total[5m]) > 0.05

Grafana: Visualization

Grafana connects to data sources (Prometheus, Elasticsearch, InfluxDB, etc.) and creates dashboards.

Dashboard Configuration

{
  "title": "Application Overview",
  "panels": [
    {
      "title": "Request Rate",
      "type": "graph",
      "targets": [{
        "expr": "rate(http_requests_total[5m])",
        "legendFormat": "{{method}} {{endpoint}}"
      }]
    },
    {
      "title": "Error Rate",
      "type": "stat",
      "targets": [{
        "expr": "rate(http_requests_total{status=~\"5..\"}[5m]) / rate(http_requests_total[5m]) > 0.01"
      }]
    }
  ]
---

Common Dashboard Widgets

Singlestat — current value (e.g., active users)
Time series — metrics over time (requests, latency)
Heatmap — distribution over time (latency heatmap)
Table — detailed data with sorting
Alert list — active alerts and their status

Alertmanager

# alertmanager.yml
global:
  slack_api_url: 'https://hooks.slack.com/services/T...'

route:
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
      repeat_interval: 5m

receivers:
  - name: 'default'
    slack_configs:
      - channel: '#alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ .CommonAnnotations.description }}'

  - name: 'pagerduty'
    pagerduty_configs:
      - routing_key: 'your_pagerduty_key'

# alert.rules.yml
groups:
  - name: application
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5%"
          description: "Error rate is {{ $value | humanizePercentage }} on {{ $labels.instance }}"

ELK Stack: Logging

ELK = Elasticsearch (storage/search) + Logstash (processing) + Kibana (visualization).

Logstash Configuration

# logstash.conf
input {
  beats {
    port => 5044
  }
  file {
    path => "/var/log/nginx/access.log"
    start_position => "beginning"
  }
---

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
  }
  geoip {
    source => "clientip"
  }
---

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-access-%{+YYYY.MM.dd}"
  }
---

Filebeat (Lightweight Shipper)

# filebeat.yml
filebeat.inputs:
  - type: log
    paths:
      - /var/log/*.log
      - /var/log/nginx/*.log

output.logstash:
  hosts: ["logstash:5044"]

# Or directly to Elasticsearch:
output.elasticsearch:
  hosts: ["localhost:9200"]

Tracing with Jaeger

Tracing follows a request through multiple services, showing latency at each hop.

// OpenTelemetry tracing
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

func handleRequest(w http.ResponseWriter, r *http.Request) {
    tracer := otel.Tracer("web-service")
    ctx, span := tracer.Start(r.Context(), "handleRequest")
    defer span.End()

    // Call downstream service
    span.AddEvent("Calling database")
    result, err := queryDatabase(ctx)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
    }
---

Setting Up a Monitoring Stack

# docker-compose.monitoring.yml
version: "3.9"

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"

  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200

volumes:
  grafana-data:

Best Practices

Define SLOs — Service Level Objectives (e.g., 99.9% uptime)
Alert on symptoms, not causes — alert when users are affected, not when CPU is high
Use dashboards for diagnosis — not for alerting
Set up log retention — hot (7 days), warm (30 days), cold (1 year)
Use structured logging — JSON format for easy parsing
Correlate metrics, logs, and traces — use common request IDs
Test your alerts — regularly verify that alerts fire correctly

Conclusion

Observability requires metrics, logs, and traces working together. Prometheus and Grafana handle metrics and visualization, the ELK stack manages logs, and distributed tracing tools like Jaeger connect it all. Start with metrics and logging, add tracing for critical services, and build dashboards that answer real operational questions.

Related: Check our Infrastructure as Code guide.

Monitoring Strategies

USE Method and RED Method

Brendan Gregg’s USE (Utilization, Saturation, Errors) method provides a systematic approach for resource monitoring. For every resource (CPU, memory, disk, network), check utilization (percentage busy), saturation (queue size or waiting time), and errors (failure counts). This method is comprehensive and catches most infrastructure issues. Tom Wilkie’s RED (Rate, Errors, Duration) method focuses on service-level monitoring — measure the rate of requests, the rate of errors (HTTP 5xx, exceptions), and the distribution of request duration. RED is simpler and more aligned with user experience than resource-focused monitoring. Most mature observability strategies use both: USE for infrastructure health and RED for application health, with the Four Golden Signals (latency, traffic, errors, saturation) providing a unified framework.

Service Level Objectives

SLOs (Service Level Objectives) translate business expectations into measurable targets. An SLO might specify API availability of 99.9% over a 30-day rolling window. SLOs are based on SLIs (Service Level Indicators) — measurable metrics like request latency at p99, error rate, or throughput. Error budgets are derived from SLOs: if the SLO is 99.9%, the error budget is 0.1% of total requests over the window. When error budget is exhausted, development velocity is deliberately slowed to focus on reliability improvements. This framework aligns engineering decisions with business priorities. Google’s SRE book popularized this approach, and tools like Google Cloud Monitoring, Datadog SLO tracking, and Grafana SLO dashboards support implementation. The hardest part of SLO adoption is defining meaningful SLIs that correlate with actual user experience.

Anomaly Detection

Static thresholds for alerting generate excessive noise and miss slow-moving degradation. Statistical anomaly detection applies techniques like moving averages with standard deviation bands, seasonal decomposition to detect deviations from expected patterns, and forecasting-based detection where actual values are compared to predicted ranges. Machine learning approaches use models trained on historical metric patterns — PCA for dimensionality reduction of multi-dimensional metrics, isolation forests for outlier detection, and LSTMs for time series forecasting. The challenge is distinguishing genuine anomalies from routine variability: an e-commerce site’s traffic dip during lunch hours is normal, but the same dip at midnight could indicate an outage. Context-aware anomaly detection incorporates business logic and known patterns to reduce false positives.

On-Call and Incident Response

Monitoring is only as valuable as the response it enables. An effective on-call rotation ensures that alerts are handled promptly without burning out individual engineers. Primary and secondary on-call roles provide escalation paths — the secondary handles alerts when the primary is on another incident or unavailable. Follow-the-sun rotations distribute on-call across global time zones so engineers handle incidents during their working hours. Incident response follows a structured process: detection (alert fires), triage (assess severity and impact), mitigation (stop the bleeding — rollback, redirect traffic, scale up), diagnosis (determine root cause), resolution (apply permanent fix), and follow-up (postmortem, action items). Runbooks document incident response procedures for common scenarios — database connection exhaustion, degraded API endpoints, certificate expiration — reducing mean time to mitigation by eliminating investigation steps for known failure modes. Regular incident drills — Game Days — simulate failures to test response effectiveness and identify gaps in monitoring coverage.

Frequently Asked Questions

What is the difference between monitoring and observability? Monitoring collects predefined metrics, logs, and traces to detect known failure modes and track system health. Observability is a property of the system — how well you can understand its internal state from external outputs, especially during novel failure modes. Observability requires high-cardinality data, structured logging, and distributed tracing to enable ad-hoc debugging without pre-defining every query.

What are the three pillars of observability? Metrics (numeric measurements collected at intervals), logs (timestamped records of discrete events), and traces (representations of request paths through distributed systems). Modern observability platforms unify these pillars with correlated data — clicking on a high-latency trace shows related logs and metrics for that specific request.

How do I reduce alert fatigue? Use SLO-based alerting instead of static thresholds — alert only when the error burn rate exceeds the budget. Tune alert sensitivity to avoid firing for transient issues. Group related alerts into incidents. Implement dependency-aware alerting to suppress downstream alerts when a root cause is identified. Regularly review and retire alerts that never trigger actionable responses.

For a comprehensive overview, read our article on Blue Green Deployments.

Share this article

X LinkedIn Facebook Email