Incident Severity Calculation Method

Last time, we looked at the criteria for determining priority during incident response (which incidents are most severe and how to respond), and today we'll explore the calculation method.

In other words, it's the Incident Severity calculation method for objectively quantifying and grading incidents.

The Need for Severity Calculation and its Two Core Axes #

If incident severity (SEV) is not clearly calculated, company-wide engineers might be woken up in the middle of the night for minor bugs, or resources could be easily wasted.

Conversely, a catastrophic business failure could occur if, for example, payments are down and it's only fixed the next day.

According to SRE standards at global IT companies like Google and Netflix, severity is typically calculated by multiplying two axes: impact and urgency.

Business Impact #

This assesses how much an incident affects the business (revenue, user experience, data integrity).

• High (Critical): Core payment failure, large-scale data loss, complete access outage (company money burning in real-time)
• Medium (Moderate): Major feature delays, errors affecting only specific regions/some users (e.g., less than 10%), or paralysis of features not directly tied to revenue.
• Low (Minor): Internal admin tool bugs, minor UI/UX glitches, simple errors with workarounds.

Urgency #

This assesses whether it needs to be fixed immediately or if it can be addressed over time. The existence of a workaround is crucial.

• High (Immediate Response): No workaround exists, and damage will increase exponentially if left unaddressed (e.g., security breach, database nearing 100% capacity).
• Medium (Prompt Response): Customers are inconvenienced, but there's a workaround (e.g., paying via web instead of app), or the damage is not spreading.
• Low (Normal Response): Customers barely notice, or it can be fixed with the next regular deployment schedule.

Matrix #

Combining the two axes above completes a matrix that allows anyone to mechanically determine the severity level based on criteria, not emotions.

Some organizations use a SEV-1 system where lower numbers indicate higher severity, while others use a Priority 1 (P1) system. Here, SEV-1 represents the highest severity.

Automated Alert Rules and Command Examples #

In modern infrastructure environments, while people sometimes manually determine SEV, it's also common to set thresholds in APM or alerting tools

and automate the system to primarily calculate severity and send alerts.

Datadog Alert Monitor Configuration Code: Terraform Example for Automated Severity Calculation #

This is infrastructure code that automatically assigns SEV levels based on the 5xx error rate of a payment API and specifies the targets to be notified.

resource "datadog_monitor" "payment_api_error_rate" {
  name               = "[Payment API] 5xx Error Rate is too high"
  type               = "query alert"
  query              = "avg(last_5m):avg:trace.http.request.errors{service:payment-api} / avg:trace.http.request.hits{service:payment-api} > 0.1"
  
  # Mechanically separate Severity levels based on error rate thresholds
  monitor_thresholds {
    critical = 0.10  # 5-minute error rate exceeds 10% -> SEV-1
    warning  = 0.05  # 5-minute error rate exceeds 5% -> SEV-2
  }

  # Branch processing for Slack channels and PagerDuty (phone) targets based on the triggered threshold
  message = <<EOF
  {{#is_alert}}
    🚨 **[SEV-1] Payment API error rate exceeded 10%!** Business Impact: High, Urgency: High
    Immediate response required! @pagerduty-payment-team-critical
  {{/is_alert}}
  
  {{#is_warning}}
    ⚠️ **[SEV-2] Payment API error rate exceeded 5%**
    Business Impact: Medium, Urgency: High
    On-call personnel, please investigate the situation. @slack-payment-backend-team
  {{/is_warning}}
  EOF
}

Let's also look at manual incident declaration commands via Slack, by integrating PagerDuty or Jibot.

If a logical bug that the system cannot catch is discovered by a CS team or engineer (e.g., payment succeeds but product is not delivered), this method allows them to immediately declare an incident according to the matrix criteria and open a war room via a Slack bot.

# Enter command in Slack channel
/incident declare

# Enter into the popup displayed by the bot (example)
- Title: Bug where payment amount becomes 0 when applying promotion coupon
- Impact: High (Direct loss of company revenue currently occurring)
- Urgency: High (Malicious users increasing at this very moment, no workaround)
- Severity: SEV-1 (Automatically mapped according to the above matrix)
- Component: Coupon Service, Payment Gateway

1. Automatic creation of a dedicated war room (Slack channel) named #inc-20260316-coupon-bug.
2. Forced phone call (Call) to backend leads and payment/coupon owners' smartphones via PagerDuty.
3. Incident dashboard status immediately changes from 'Green' to 'Red'.