SRE: SLIs, SLOs, Error Budgets & Reliability Engineering

Site Reliability Engineering (SRE) is a discipline that applies software engineering to operations problems. Originated at Google in 2003 when they hired software engineers to run production systems, SRE bridges the gap between development and operations by treating infrastructure as a software problem. Instead of hiring more operators to manage growing systems, SRE builds automation and tooling that lets a small team manage large-scale systems reliably. Google’s approach, documented in the Site Reliability Engineering book series, has been adopted by organizations worldwide including LinkedIn, Netflix, and Dropbox. The 2024 SRE Report found that organizations implementing SRE practices see a 50% reduction in incident frequency and a 40% improvement in mean time to recovery.

Core SRE Principles

SRE is built on a foundation of measurement, automation, and engineering approaches to operations. The core principles provide a framework for making data-driven decisions about reliability.

Service Level Indicators (SLIs)

An SLI is a quantitative measure of a specific aspect of service performance. Common SLIs include:

• Latency — time to respond to a request (measured at p50, p95, p99 percentiles)
• Availability — fraction of requests that succeed (HTTP 200-299)
• Throughput — requests per second the system handles
• Error rate — fraction of requests that fail (HTTP 5xx)
• Durability — probability that stored data is not lost

Define SLIs at the user-facing level. The user does not care about CPU utilization or database query time unless those correlate with their experience. A good SLI starts with: “How does the user experience reliability?” Google’s SRE team recommends choosing SLIs that represent the user’s critical path — the sequence of operations that must succeed for the user to consider the service working.

Service Level Objectives (SLOs)

An SLO is a target value or range for an SLI over a measurement window:

SLO: 99.9% of API requests complete in < 500 ms (30-day rolling window)

SLOs should be ambitious but achievable. A 100% SLO is impossible — failures happen. Target the number that keeps users happy while allowing for maintenance, upgrades, and the occasional infrastructure issue.

Error Budgets

An error budget is the acceptable amount of unreliability: 100% - SLO. If your SLO is 99.9%, your error budget is 0.1% per measurement window — roughly 8.7 hours of unreliability per year. Error budgets resolve the fundamental tension between feature velocity and reliability. When error budget is available, developers can deploy risky features. When error budget is exhausted, deployments stop until reliability improves. This creates a data-driven mechanism for balancing innovation and stability that replaces subjective debates between development and operations teams.

Toil Reduction

Toil is operational work that is manual, repetitive, automatable, and devoid of enduring value. Examples include restarting services by hand, rotating credentials manually, responding to non-urgent pages with a known runbook, and manually approving routine deployments. SRE aims to keep toil under 50% of team time. Automation strategies include:

• Auto-remediation — scripts that automatically handle common failure scenarios
• Self-service tooling — internal platforms that let developers deploy and manage their own services
• Runbook automation — converting manual runbook steps into automated workflows
• Capacity automation — auto-scaling that eliminates manual instance provisioning

Incident Management

Severity Levels

Incident Response Flow

1. Detection — monitoring alert or user report. Aim for 95% of incidents to be detected by monitoring rather than user reports
2. Triage — determine severity and assign responders within the defined response time
3. Mitigation — stop the bleeding (rollback, feature flag, scale up). Mitigation comes before root cause analysis
4. Resolution — apply permanent fix after mitigation
5. Postmortem — blameless analysis with action items tracked to completion

Monitoring Strategy

The Four Golden Signals

Google’s SRE team identified four metrics that capture the most important aspects of service health:

1. Latency — time to service a request. Track p50, p95, and p99. High latency is often an early indicator of impending failure
2. Traffic — demand on the system (requests per second, active users). Traffic anomalies can indicate routing issues or attacks
3. Errors — rate of failed requests (explicit 5xx, implicit success with wrong data)
4. Saturation — how full the service is (CPU, memory, disk, connection pools). Saturation often predicts future latency increases

Alerting Wisdom

Not every anomaly needs a page. Define alert severity by impact:

• Page — user-facing impact, immediate action required. Every page should trigger a runbook
• Ticket — needs investigation within business hours. For issues that degrade but do not break the service
• Log — informational, no immediate action. For tracking trends and capacity planning

Every page should have a runbook. If the responder needs to think about what to do, the alert needs improvement. Google’s internal target is that 95% of pages should be handled without escalation. Runbooks should be version-controlled and tested during game days.

SLO-Based Alerting

Rather than alerting on static thresholds, SRE teams alert on error budget burn rate:

# Prometheus alert for fast error budget burn
groups:
  - name: slo-alerts
    rules:
      - alert: ErrorBudgetBurn
        expr: |
          (
            sli:error_rate:5m > (14.4 * (1 - slo:target))
          )
        for: 5m
        labels:
          severity: page

This approach alerts when the error rate is high enough to exhaust the error budget within a short time, rather than alerting on every single error spike.

Release Engineering

Progressive rollouts reduce deployment risk through staged exposure:

• Canary — 1% of traffic, validate for 10-30 minutes with automated metric comparison
• Regional — deploy to one region or availability zone (25% of traffic)
• Global — full rollout after validation at each stage

Each stage monitors error rates and latency against SLO targets. Roll back immediately if SLOs are breached. Feature flags separate deployment from release, allowing instant rollback without code deployment. According to Google’s research, progressive rollouts catch 70% of deployment-related issues in the canary stage before they reach broad production.

Implementing SRE in Practice

Adopting SRE practices requires organizational changes as well as technical ones. Here is how teams typically transition from reactive operations to proactive reliability engineering.

Starting with SLIs

Begin by instrumenting a single user-facing service with latency and error rate SLIs. Use your existing monitoring infrastructure — most tools (Prometheus, Datadog, New Relic) support SLI measurement natively:

# Prometheus recording rules for SLIs
groups:
  - name: sli
    rules:
      - record: sli:request_latency_p99:5m
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
      - record: sli:error_rate:5m
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

Setting Your First SLO

Choose a realistic target. For a service that currently has 99.5% availability, setting a 99.99% SLO is unrealistic. Start at current performance plus a small improvement:

• Measure current SLI value over 30 days
• Set SLO at current value + 0.1% (achievable stretch)
• Calculate error budget: (100% - SLO) × measurement window
• Monitor error budget burn rate and alert when consumption accelerates

Building an Error Budget Policy

Define clear rules for what happens when error budget is exhausted:

error_budget_policy:
  consumption_thresholds:
    - percentage: 50%
      action: "Review deployment pace, add monitoring"
    - percentage: 75%
      action: "Pause non-critical deployments, focus on reliability"
    - percentage: 100%
      action: "Stop all deployments until budget recovers"
  measurement_window: 28d
  review_frequency: weekly

The error budget policy gives teams clear guidance on when to prioritize reliability over features. It removes the subjective debate and replaces it with data-driven decision-making.

Automating Toil

Runbook automation is the highest-impact toil reduction strategy. Use tools like Rundeck, StackStorm, or custom scripts to automate:

• Restarting failed services
• Rotating credentials
• Scaling services based on time-of-day patterns
• Running database maintenance operations
• Generating compliance reports

Each automated task should produce a record of execution and notify the team on failure. The goal is to handle 80% of operational tasks without human intervention. Track toil percentage weekly and celebrate automation wins that reduce it.

FAQ

How do I get started with SRE?

Start with one user-facing service. Define one SLI (latency or availability), set one SLO, and calculate the error budget. Automate the runbook for the most common failure scenario. Measure toil and set a reduction target. Expand to additional services and metrics as the team matures. The key is to start small and prove the value before scaling.

What is the difference between SRE and DevOps?

DevOps is a cultural movement focused on breaking down silos between development and operations. SRE is a specific engineering practice that implements DevOps principles through quantifiable reliability targets, automation, and software engineering approaches to operations. SRE can be seen as a concrete implementation of the DevOps philosophy. While DevOps describes what to do, SRE provides the how.

How do I calculate an SLO for a new service?

Start with user expectations: what level of reliability does the user need? For an internal API used by other teams, 99.9% is typical. For a customer-facing payment API, 99.99% may be required. Use a 30-day rolling window. Set the SLO, measure for a month, then adjust. The initial SLO is a starting point, not a permanent commitment. You can tighten SLOs as you improve reliability.

Can SRE work for small teams?

Yes, but adapt the practices to your scale. A two-person team cannot have a dedicated on-call rotation, but they can define SLIs and SLOs, automate repetitive tasks, and conduct blameless postmortems after incidents. Start with the principles that provide the most value for your context. The core ideas — measurement, automation, blameless culture — scale down as well as they scale up.

How do I convince my organization to adopt SLOs?

Start with a pilot service. Show the data: when the service was below the SLO, users were unhappy. When it met the SLO, users were satisfied. Demonstrate that error budgets enable faster deployment when reliability is good and prevent risky deployments when it is not. Show the business value of predictable reliability. Case studies from Google, Etsy, and LinkedIn that document their SRE adoption journeys provide compelling evidence for leadership.

Conclusion

SRE applies software engineering to operations problems. SLIs and SLOs define reliability in measurable terms. Error budgets balance feature development against system stability. Toil reduction frees engineers for high-value work. Together, these practices build systems that are both reliable and evolvable. Start with one SLO, automate one runbook, and measure the improvement.

For advanced practices, see our Advanced SRE guide covering chaos engineering and incident command. For deployment strategies, see blue-green deployments.