Applying SLA and Error Budget

SLI, SLO, SLA #

These three metrics are strictly distinguished based on their measurement scope and purpose.

SLI Service Level Indicator

• It is a quantitative data value that measures the current state of a service.
• The calculation formula is successful requests / total valid requests * 100.
• Measurement items include availability, latency, throughput, and error rate.

SLO Service Level Objective

• It is an SLI target that internal engineering teams aim to achieve.
• The setting criteria are established by considering system architecture limitations and business requirements, and are usually set more broadly than SLAs.
• For example, for the past 30 days, the 99th percentile latency for the payment API is less than 200ms, with a target achievement rate of 99.9%.

SLA Service Level Agreement

• It is a formal contract between a service provider and a customer.
• It specifies financial penalties, refund credits, and other consequences that occur when the SLO is not met.
• If monthly availability falls below 99.9, 10% of that month's billing amount will be compensated as credit.

Error Budget Calculation and Control Policy #

Error budget refers to the maximum allowable failure occurrence limit, time, or number of requests that a system can tolerate.

Since perfect 100% availability is technically impossible and cost-ineffective, it is used to accept the risk of failure within the allowed budget and accelerate the pace of new deployments.

Error Budget Calculation Method (Time-based)

• Formula: 100% - SLO
• Calculation example based on 30 days
  • SLO 99.0%: 30 days x 24 hours x 60 minutes x 0.01 = allowed downtime 432 minutes, approx. 7.2 hours
  • SLO 99.9%: 30 days * 24 hours * 60 minutes * 0.001 = allowed downtime 43.2 minutes
  • SLO 99.99%: 30 days * 24 hours * 60 minutes * 0.0001 = allowed downtime 4.32 minutes

There is also a Deployment Control Policy based on the remaining Error Budget.

Error Budget becomes a systemic standard that enforces guidelines for action between development teams (feature deployment) and operations teams (stability).

• If the error budget is sufficient (over 50%), new feature development and regular deployments proceed normally, and infrastructure changes and migrations are allowed.
• If the error budget is at a cautionary state with less than 20% remaining, conservative approaches are implemented, such as strengthening pre-deployment QA procedures and reducing canary deployment ratios.
• If the budget is depleted to 0% or less, it's a Feature Freeze, meaning all new feature deployments are halted. All resources are dedicated solely to reliability improvement tasks such as bug fixes, enhanced monitoring, and architectural improvements.

Applying Alarms Based on Burn Rate (Error Depletion Rate) #

Static thresholds, such as a simple 'error rate exceeds 5%' alarm, are vulnerable to traffic fluctuations.

In practice, alarms are triggered by measuring the burn rate to see how quickly the error budget is being depleted.

• Burn Rate 1: The rate at which the error budget is completely depleted in exactly 30 days.
• Burn Rate 10: The rate at which the budget is depleted in 3 days (72 hours). (Requires quick response)
• Burn Rate 14.4: The rate at which 5% of the budget is depleted in just 1 hour. (Requires immediate SEV-1 response)

Example #

This is the prometheus/alertmanager configuration code that triggers an immediate alarm when a specific API's error depletion rate exceeds a threshold.

Detecting a burn rate of 14.4, which depletes 5% of the budget in 1 hour, based on an SLO of 99.9%.

groups:
- name: SLO_Burn_Rate_Alerts
  rules:
  - alert: HighErrorBudgetBurnRate_1Hour
    # Evaluate if the 5xx error rate over 1 hour is occurring at 14.4 times the allowed budget (0.1%).
    expr: |
      (
        sum(rate(http_requests_total{status=~"5.."}[1h]))
        /
        sum(rate(http_requests_total[1h]))
      ) > (0.001 * 14.4)
    for: 5m
    labels:
      severity: critical
      team: backend-core
    annotations:
      summary: "Critical: Error Budget Burn Rate > 14.4 (1h window)"
      description: |
        The error occurrence rate over the last hour is very high.
        If this rate persists, this month's 99.9% SLO Error Budget will be completely depleted within 3 days.
        Proceed with immediate Triage and Mitigation.