Cloud Monitoring and Observability: Metrics, Logs, Traces

Cloud Computing Cloud Computing 8 min read 1626 words Beginner ExcellentWiki Editorial Team

Monitoring tells you what is happening. Observability lets you understand why. In cloud environments with dozens of microservices, ephemeral containers, and auto-scaling, traditional monitoring approaches (SSH into a server, check logs) do not scale. Observability is the practice of instrumenting systems so you can understand internal states from external outputs — metrics, logs, and traces. According to the 2024 CNCF Observability Survey, 78% of organizations now have dedicated observability teams, and 65% report that observability directly reduced mean time to resolution (MTTR) by over 50%.

The Three Pillars of Observability

Metrics

Metrics are numerical measurements collected over time — request counts, error rates, CPU utilization, memory consumption, queue depths. They are optimized for storage and querying (time-series databases) and provide the fastest path to understanding system health.

Metric types:

Counters — cumulative values that only increase (total requests, total errors)
Gauges — point-in-time values that go up and down (CPU usage, active connections)
Histograms — sampled observations with configurable buckets (request latency distribution)

RED method for services (Rate, Errors, Duration):

from prometheus_client import Counter, Histogram

REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

REQUEST_DURATION = Histogram(
    'http_request_duration_seconds',
    'Request latency in seconds',
    ['method', 'endpoint'],
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
)

# Instrument
REQUEST_COUNT.labels(method='GET', endpoint='/api/articles', status='200').inc()
with REQUEST_DURATION.labels(method='GET', endpoint='/api/articles').time():
    process_request()

Logs

Logs are discrete event records with timestamps. Structured logging (JSON format) is essential for automated processing and querying. Unstructured log lines (“User logged in”) require manual grepping and break in distributed systems.

Structured logging with Python structlog:

import structlog
logger = structlog.get_logger()

# Structured event
logger.info("article_created",
    article_id=a123,
    category="cloud-computing",
    author_id=456,
    processing_time_ms=42
)

Log levels:

ERROR — system is degraded, immediate investigation required
WARN — potential issue, not yet impacting users
INFO — normal operations, useful for tracing request flow
DEBUG — diagnostic detail, disabled in production

Log aggregation tools (Loki, ELK Stack, CloudWatch Logs Insights) index structured fields for querying: {app="api"} |= "ERROR" | json | processing_time_ms > 100

Traces

Distributed traces follow a single request across multiple services, showing the latency contribution of each service call. OpenTelemetry is the industry standard for distributed tracing, supported by AWS X-Ray, GCP Cloud Trace, Azure Application Insights, Jaeger, and Grafana Tempo.

Trace structure: A trace is a tree of spans. Each span represents a unit of work (database query, HTTP call, function execution) with a start time, duration, and optional metadata.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("order.value", 99.99)
    
    try:
        with tracer.start_as_current_span("validate_payment"):
            validate_payment(order_id)
    except Exception as e:
        span.set_status(Status(StatusCode.ERROR, str(e)))
        span.record_exception(e)
        raise
    
    with tracer.start_as_current_span("update_inventory"):
        update_inventory(order_id)
    
    with tracer.start_as_current_span("send_notification"):
        send_confirmation_email(order_id)

Cloud Monitoring Services

Each provider offers integrated monitoring, but their capabilities differ:

Feature	AWS	GCP	Azure
Metrics	CloudWatch	Cloud Monitoring	Azure Monitor
Logs	CloudWatch Logs	Cloud Logging	Log Analytics
Traces	X-Ray	Cloud Trace	Application Insights
Dashboards	CloudWatch Dashboards	Monitoring Dashboards	Azure Dashboards
Alerting	CloudWatch Alarms	Alerting Policies	Alert Rules
SIEM	Security Hub	Security Command Center	Microsoft Sentinel

CloudWatch Logs Insights supports SQL-like querying for log data:

fields @timestamp, @message, statusCode
| filter statusCode >= 400
| stats count() by statusCode
| sort @timestamp desc
| limit 100

Prometheus and Grafana

The Prometheus + Grafana stack is the most popular open-source monitoring solution, adopted by over 60% of organizations surveyed in the CNCF 2024 Annual Report.

Prometheus scrapes metrics from instrumented targets at configurable intervals (default 15s). It stores data in a time-series database with a powerful query language (PromQL):

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'api'
    metrics_path: '/metrics'
    static_configs:
      - targets: ['api:8000', 'web:3000']

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'postgres'
    static_configs:
      - targets: ['postgres-exporter:9187']

alerting:
  alertmanagers:
    - static_configs:
      - targets: ['alertmanager:9093']

PromQL example queries:

rate(http_requests_total[5m]) — request rate per second (5-min average)
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) — p95 latency
avg(node_cpu_seconds_total{mode="idle"}[5m]) < 0.2 — nodes with less than 20% idle CPU

Grafana visualizes metrics from Prometheus, Loki, CloudWatch, and dozens of other data sources. Dashboards should follow these conventions:

Left-side panels: request rate, error rate, latency (RED metrics)
Right-side panels: CPU, memory, disk, network (USE metrics)
Bottom panels: logs correlated with the time range selected
Templating for environment, service, and region selection

Alerting

Effective alerting requires careful design — too many alerts cause alert fatigue, too few allow incidents to go unnoticed.

Alert severity levels:

Critical — user-facing outage, immediate response required (pages on-call)
Warning — potential issue, investigate during business hours (Slack notification)
Info — informational, no action needed (dashboard annotation)

groups:
  - name: excellentwiki-alerts
    interval: 30s

    rules:
    - alert: HighErrorRate
      expr: |
        sum(rate(http_requests_total{status=~"5.."}[5m]))
        /
        sum(rate(http_requests_total[5m])) > 0.01
      for: 5m
      labels:
        severity: critical
        team: backend
      annotations:
        summary: "Error rate above 1% for {{ $value | humanizePercentage }}"
        runbook: "https://runbook.excellentwiki.com/high-error-rate"

    - alert: HighLatency
      expr: |
        histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "p95 latency above 2 seconds"

Alertmanager handles deduplication, grouping, and routing. Common receivers: Slack (team channel), PagerDuty (on-call), OpsGenie, email, and webhook.

Dashboards and SRE

USE method for resources (Utilization, Saturation, Errors):

Utilization: percentage of resource being used (CPU, memory, disk)
Saturation: queue depth or pending requests
Errors: failed operations, dropped packets

Dashboard design principles — every dashboard should answer specific operational questions. The “Four Golden Signals” approach (Google SRE Book) tracks latency, traffic, errors, and saturation. For a web application, this translates to:

Latency: time to first byte, API response time p50/p95/p99
Traffic: requests per second, active users, bandwidth consumed
Errors: HTTP 5xx rate, exception count, failed database queries
Saturation: CPU utilization, memory pressure, connection pool usage, disk queue depth

Service Level Objectives (SLOs) define target performance:

99.9% availability = 8.77 hours of downtime per year
99.99% availability = 52.56 minutes per year
99.999% availability = 5.26 minutes per year

Track error budgets (1 - SLO) and alert when error budget is being consumed faster than expected. For example, if your monthly SLO is 99.9%, your error budget is 43 minutes of downtime per month. If you exhaust it by day 15, you should halt all risky deployments for the remainder of the month.

Grafana best practices: use templated variables for environment, service, and region selection; annotate dashboards with deployment events to correlate code changes with metric anomalies; share dashboards via Grafana Cloud or provisioning-as-code (JSON or YAML definitions committed to version control).

Incident Response

Runbooks — document step-by-step procedures for common incident types (high error rate, database failure, deployment failure). Store runbooks alongside code in version control.

On-call rotations — use PagerDuty, OpsGenie, or Grafana OnCall for scheduling and escalation.

Postmortems — blameless incident reviews focused on system improvements, not individual mistakes. Answer: what happened, why did it happen, what prevented detection, how do we prevent recurrence.

Mean Time to Resolve (MTTR) — track recovery time. Elite SRE teams maintain MTTR under 1 hour.

FAQ

What is the difference between monitoring and observability? Monitoring tells you when something is wrong using predefined dashboards and alerts (known unknowns). Observability lets you explore and understand unknown failure modes using high-cardinality data, traces, and ad-hoc querying (unknown unknowns). You need both.

How much does monitoring cost at scale? CloudWatch: $0.30 per metric per month (first 10K metrics). Prometheus on Kubernetes with Thanos: infrastructure cost (~$100/month for moderate scale). Log retention: CloudWatch Logs $0.03/GB ingested, $0.03/GB stored. Typical bill for 100-microservice deployment: $500-2000/month.

How do I choose between Prometheus and Datadog? Prometheus is open-source, self-hosted (or managed via Grafana Cloud, Amazon Managed Service for Prometheus), and requires operational investment. Datadog is SaaS, easier to set up, with built-in integrations for 700+ services, but costs $15-23 per host per month plus data volume charges. Choose Prometheus for cost control and data sovereignty; choose Datadog for speed to value and breadth of integrations.

What is sampling in distributed tracing? Head-based sampling decides whether to trace a request at the entry point (e.g., keep 1% of all requests). Tail-based sampling buffers traces and decides based on properties (errors, slow requests) — more useful but more expensive. Most cost-conscious teams use head-based sampling at 1-5%.

How do I set up monitoring for serverless functions? CloudWatch Logs (Lambda), Cloud Logging (Cloud Functions), and Application Insights (Azure Functions) automatically capture invocation metrics. For custom metrics, use CloudWatch Embedded Metric Format (AWS), OpenTelemetry SDK (GCP), or Application Insights SDK (Azure). Cold starts and duration metrics are critical for serverless performance monitoring.

Cloud Networking Guide — Cloud Cost Optimization — CI/CD Pipeline Guide