Site Reliability Engineering: SRE Principles and Practices

Site Reliability Engineering (SRE) is a discipline that applies software engineering to operations problems. Originated at Google, SRE bridges the gap between development and operations by treating infrastructure as a software problem. Instead of hiring more operators to manage growing systems, SRE builds automation and tooling that lets a small team manage large-scale systems.

This guide covers the core SRE principles: service level indicators (SLIs), service level objectives (SLOs), error budgets, toil reduction, incident management, and building reliable systems.

Core SRE Principles

Service Level Indicators (SLIs)

An SLI is a quantitative measure of a specific aspect of service performance. Common SLIs include:

• Latency — Time to respond to a request (e.g., p99 < 500 ms)
• Availability — Fraction of requests that succeed (e.g., 99.9%)
• Throughput — Requests per second
• Error rate — Fraction of requests that fail
• Durability — Probability data is not lost
• Freshness — How current the data is (for caches, replicas)

Define SLIs at the user-facing level. Internal metrics matter only if they correlate with user experience:

# Example SLI definitions
slis:
  api_latency:
    type: histogram
    unit: milliseconds
    good_event: duration < 500
    valid_events: all HTTP requests
  api_availability:
    type: boolean
    good_event: status_code < 500
    valid_events: all HTTP requests

Service Level Objectives (SLOs)

An SLO is a target value or range for an SLI over a measurement window:

SLO: 99.9% of API requests complete in < 500 ms (measured over 30 days)

SLOs should be ambitious but achievable. A 100% SLO is impossible — failures happen. Choose the right target:

• 99.9% (“three nines”) — ~8.7 hours downtime/year, suitable for internal tools
• 99.99% (“four nines”) — ~52 minutes downtime/year, common for customer-facing APIs
• 99.999% (“five nines”) — ~5 minutes downtime/year, required for critical infrastructure

Error Budgets

An error budget is the acceptable amount of unreliability: 100% - SLO. If your SLO is 99.9%, your error budget is 0.1% — roughly 8.7 hours of unreliability per year.

Error budgets resolve the tension between feature velocity and reliability:

• The development team can deploy risky features as long as error budget remains
• When error budget is exhausted, deployments stop until reliability improves
• This creates a data-driven conversation instead of emotional arguments

# Error budget tracking
class ErrorBudget:
    def __init__(self, slo_percent: float, window_seconds: int = 30 * 86400):
        self.budget = 1.0 - (slo_percent / 100.0)
        self.window = window_seconds

    def remaining(self, failures: int, total: int) -> float:
        error_rate = failures / total if total > 0 else 0
        return max(0.0, self.budget - error_rate) / self.budget * 100

    def is_exhausted(self, failures: int, total: int) -> bool:
        return self.remaining(failures, total) <= 0

Toil Reduction

Toil is operational work that is manual, repetitive, automatable, and devoid of enduring value. Examples: restarting services by hand, rotating credentials manually, responding to non-urgent pages with a known runbook.

SRE aims to keep toil under 50% of team time. Strategies include:

• Automation — Script common tasks, then build self-service tooling
• Elimination — Remove processes that no longer add value
• Delegation — Give developers self-service access to deploy and monitor their services

Example: Automating Certificate Rotation

#!/bin/bash
# Manual approach (toil)
# scp cert.pem user@server:/etc/ssl/
# ssh user@server "systemctl reload nginx"

# Automated approach
certbot renew --deploy-hook "systemctl reload nginx"

Automate certificate renewal once and never think about expiring certs again.

Incident Management

SRE defines a structured incident management process:

Severity Levels

Incident Response Flow

1. Detection — Monitoring alert or user report
2. Triage — Determine severity and assign responders
3. Mitigation — Stop the bleeding (rollback, feature flag, scale up)
4. Resolution — Apply permanent fix
5. Postmortem — Blameless analysis and action items

Blameless Postmortems

A blameless postmortem asks what the system allowed to go wrong, not who caused it. The template:

## Summary
- Duration: 2025-03-15 14:32 — 15:47 UTC
- Impact: 15% of users experienced 5xx errors
- Root cause: Database connection pool exhaustion after deploy

## Timeline
- 14:32 — PagerDuty alert: error rate spike
- 14:35 — Responder joins incident channel
- 14:50 — Identified recent deploy as trigger
- 15:00 — Rolled back to previous version
- 15:47 — Error rate returns to baseline

## Action Items
- [ ] Add connection pool metrics dashboard
- [ ] Configure pool size limits per service
- [ ] Add integration test for connection churn

The goal is systemic improvement, not blame assignment. Postmortems should be stored and searchable so teams learn from each incident.

Monitoring Strategy

The Four Golden Signals

Google identifies four key metrics for user-facing systems:

1. Latency — Time to service a request (distinguish success vs error latency)
2. Traffic — Demand on the system (requests per second, active users)
3. Errors — Rate of failed requests (explicit 5xx, implicit success with wrong data)
4. Saturation — How “full” the service is (CPU, memory, queue depth)

Alerting Wisdom

Not every anomaly needs a page. Define alerts by urgency:

• Page — Imminent user impact, requires immediate action
• Ticket — Needs investigation but not urgent
• Log — Informational, visible in dashboard

# Good alert: user-impacting
alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01
for: 5m
labels:
  severity: page

# Bad alert: likely noise
alert: CPUHigh
expr: node_cpu_seconds_total > 80
# What action should the responder take?

Every page should have a runbook. If the responder needs to think about what to do, the alert needs improvement.

Capacity Planning

SRE uses data to predict when systems will run out of capacity:

1. Measure — Track resource usage (CPU, memory, disk, network)
2. Trend — Fit a trendline to usage over time
3. Forecast — Project when usage hits limits
4. Provision — Add capacity before the forecast date

import numpy as np

def forecast_exhaustion(dates, usage_percent, limit=80):
    """Simple linear forecast for capacity planning."""
    x = np.arange(len(usage_percent))
    slope, intercept = np.polyfit(x, usage_percent, 1)
    days_to_limit = (limit - intercept) / slope - len(usage_percent)
    return days_to_limit

Build a buffer of 50% headroom for normal operations. Add more for predictable spikes like Black Friday or product launches.

Release Engineering

SRE treats releases as a process to de-risk, not a ceremony to endure:

Progressive Rollouts

stages:
  - canary: 1% of traffic, 10 minutes
  - regional: 25% of traffic, 1 hour
  - global: 100% of traffic, 24 hour watch

Each stage monitors error rates, latency, and business metrics. Roll back immediately if SLOs are breached.

Feature Flags

Separate deployment from release. Deploy code with feature flags disabled, then enable for a subset of users:

if feature_flags.is_enabled("new_checkout", user_id):
    return new_checkout_flow(request)
else:
    return legacy_checkout_flow(request)

Feature flags allow instant rollback without redeployment.

Best Practices

• Start with one SLO — Pick the most user-visible metric and define an SLO. Iterate from there.
• Automate the runbook — If a human has done something twice, script it the third time.
• Share on-call rotation — Rotate every developer through on-call to build operational empathy.
• Test your backups — A backup that has never been restored is wishful thinking.
• Chaos engineering — Deliberately inject failures to validate resilience (e.g., Chaos Monkey).

Summary

SRE applies software engineering to operations. SLIs and SLOs define reliability in measurable terms. Error budgets balance feature development against system stability. Toil reduction through automation lets teams scale without proportional headcount growth. Structured incident management with blameless postmortems drives continuous improvement. Together, these practices build systems that are both reliable and evolvable.

FAQ

What is DevOps? DevOps is a cultural and technical movement that combines software development (Dev) and IT operations (Ops) to shorten the development lifecycle and deliver high-quality software continuously. It emphasizes automation, collaboration, and monitoring.

What is Infrastructure as Code? IaC manages infrastructure (servers, networks, databases) through code and version control instead of manual processes. Tools like Terraform, CloudFormation, and Ansible enable reproducible, auditable, and automated infrastructure provisioning.

What is the difference between Docker and a virtual machine? Docker containers share the host OS kernel and run as isolated processes, starting in seconds and using fewer resources. VMs virtualize the entire hardware stack with a full OS per guest, providing stronger isolation at higher resource cost.

What is Kubernetes? Kubernetes is a container orchestration platform that automates deployment, scaling, and management of containerized applications. It handles service discovery, load balancing, rolling updates, and self-healing across clusters of machines.

How do monitoring and observability differ? Monitoring collects predefined metrics and alerts on known failure modes. Observability enables understanding system state from external outputs (logs, metrics, traces) without knowing what might fail in advance. Observability is essential for complex distributed systems.

For a comprehensive overview, read our article on Blue Green Deployments.

For a comprehensive overview, read our article on Ci Cd Pipeline Guide.