Comparing SLO and SLI

When dealing with data from monitoring systems, there inevitably comes a moment when we need to measure, "Are we doing well?"

At this point, let's explore the comparison and usage of SLO and SLI, which are often confused.

Conceptual Definitions #

SLI and SLO are key metrics for measuring and managing service reliability.

While these two concepts always work as a set, their roles are entirely different.

SLI (Service Level Indicator) is the actual measured value (reality) that indicates how well a service is currently performing.

It is the quantified fact of the service's state as perceived by users, and its calculation is typically defined as the ratio of successful events to total events.

SLI = \frac{\text{Successful Events}}{\text{Total Events}} \times 100

(Example: If the ratio of 200 OK responses among all API requests over the last 30 days is 99.95%, then 99.95% is the SLI.)

SLO (Service Level Objective): It is an internal target for what level the SLI should maintain. It's the bottom line agreed upon by engineering and business teams.

For example, if we aim to maintain our service's successful API response rate SLI at 99.9% or higher, then 99.9 is the SLO.

Consequently, continuously tracking whether the current SLI meets the set SLO is at the core of reliability concerns.

For reference, an SLA (Service Level Agreement) is an external business contract, such as paying penalties to customers if the SLO is not met.

Problem Definition #

If SLIs and SLOs are set incorrectly or their concepts are confused, engineers may find themselves drowning in a flood of meaningless alerts or experience side effects such as a slowdown in service innovation.

Incorrect SLI selection based on Cause: It's common to use infrastructure metrics like CPU usage exceeding 80% or memory usage exceeding 90% as SLIs. However, even if the CPU hits 99%, if users are receiving API responses normally within 0.1 seconds, the service is in a healthy state. If SLIs are chosen based on metrics unrelated to user experience, engineers will only be doing unnecessary work.
The curse of unrealistic 100% SLO setting: Setting an SLO of 100% ("Our service must never go down!") immediately creates difficulties. 100% availability is not only technically impossible but also implies zero tolerance for errors, meaning all new feature deployments and infrastructure updates would have to be halted.
Absence of an error budget: Without an SLO, we cannot calculate how much "headroom" for failures we can tolerate in a month. Without a standard, decisions about whether to proceed with a deployment or halt it to focus on stabilization are made purely by gut feeling.

Example #

Let's illustrate incorrect metrics and the absence of goals with a worst-case scenario.

The operations team has an implicit goal of keeping database CPU utilization below 85%. However, every evening at 8 PM, when traffic peaks, the CPU hits 90%, triggering an alert, and the responsible engineer has to open their laptop every night.

Yet, customers don't experience any slowdown, and payments are flowing smoothly. The engineer experiences burnout and might start ignoring alerts when a real outage occurs.

Correct Scenario: Utilizing Error Budgets based on SLI/SLO

The team agreed to set an SLO of 99.9% for the SLI, which is the percentage of user payment API responses that fall within 500ms, on a monthly basis. (Customer-centric)

This means an error budget has been created, allowing for payments to be slow or fail for 0.1% of approximately 43,200 minutes in a month, which is 43 minutes.

When a large-scale new feature deployment is scheduled for this week, measuring the SLI shows 99.98, indicating a generous remaining error budget, so the deployment can proceed confidently. This is about risk-taking, and having a clearly defined error budget makes it easier to approach with a mindset of allowing a certain degree of tolerance.

Solution #

Let's directly perform the process of extracting user-perceived symptom-based metrics from the Prometheus monitoring system, calculating the actual SLI, and comparing it with the SLO, all from the terminal.

Calculating 30-day Availability SLI in the Terminal using PromQL #

We will calculate the ratio of normal responses (2xx, 3xx, 4xx) versus HTTP 5xx errors to check our service's factual SLI.

Let's query the Prometheus HTTP API using curl to verify.

# 지난 30일(30d) 동안 전체 요청 중 서버 에러(5..)가 아닌 정상 요청의 비율을 계산 (PromQL)
$ curl -s "http://prometheus:9090/api/v1/query" --data-urlencode 'query=sum(rate(http_requests_total{status!~"5.."}[30d])) / sum(rate(http_requests_total[30d])) * 100' | jq '.data.result[0].value[1]' | tr -d '"'

99.9523104

Analyzing the results, our service's actual availability and measured SLI for the last 30 days are calculated as 99.95%.

If the target SLO was set at 99.9, then the goal is being successfully met.

Let's continue by looking at an example of calculating the remaining error budget and checking the burn rate.

Beyond simply checking if the SLI is higher than the SLO, we'll determine how much of the allowed 0.1% error budget we are currently consuming.

# 전체 허용된 에러량(1 - SLO) 대비 현재 발생한 에러 비율을 계산하여 에러 버짓 소진율을 확인 (예: SLO가 99.9%인 경우)
$ curl -s "http://prometheus:9090/api/v1/query" --data-urlencode 'query=(sum(rate(http_requests_total{status=~"5.."}[30d])) / sum(rate(http_requests_total[30d]))) / (1 - 0.999) * 100' | jq '{ "error_budget_consumed_percent": .data.result[0].value[1] }'

{
  "error_budget_consumed_percent": "47.6895"
}

The terminal check shows that 47.6% of the allowed error budget for the month has been consumed, and more than half of the buffer remains. This data proves that it is safe to approve planned new feature deployments or migration tasks.

As such, SLI and SLO are not just pretty numbers displayed on a dashboard; they become powerful benchmarks for the development team to decide whether to proceed with a deployment or halt it.