Incident Flow: Detection → Triage → Mitigation → RCA

Incident Flow is the incident response lifecycle.

Today, we'll explore the flow in the order of detection, triage, mitigation, and RCA.

Detection #

The start of incident response is detection. Ideally, detection means the system screams before a customer's CS inquiry comes in.

The metric used to measure this is MTTD (Mean Time To Detect).

The core goal is to minimize the time to detect an incident.

Key actions involve detecting error rate and latency spikes based on SLO (Service Level Objective) after APM (Datadog, Scouter) and metric (Prometheus) monitoring.

To avoid alert fatigue, alarms should be set up based on business impact, such as "payment API success rate below 95%," rather than symptom-based alerts like "CPU reaching 80%."

Triage (Classification and Escalation) #

This is the stage where the engineer who checked the alarm assesses the severity of the situation and gathers the necessary people.

The metric used to measure this is MTTA (Mean Time to Acknowledge).

The core goal is to determine the incident's Severity level and set up a war room.

Key actions include assigning SEV-1 to SEV-4 based on severity assessment metrics.

Relevant domain owners, infrastructure engineers, and leadership groups are forcibly convened through integrations like PagerDuty and Slack.

As a tip: if you can't grasp the situation within 5 minutes, declare an Incident immediately and escalate it, rather than saying "Let me take a look" and digging into it alone for 30 minutes.

Mitigation (Alleviation and Containment) #

This is the core of the cycle and the stage where an engineer's capabilities are most clearly revealed. It's measured by MTTR (Mean Time To Resolve/Recover).

The core goal is to minimize business impact (customer damage).

• Rollback: Revert code to the previous deployed version.
• Scale-out: Immediate server expansion during traffic spikes.
• Circuit Breaking: Forcibly block traffic to failing downstream systems.
• Feature Toggle: Switch off problematic new features.

At this stage, don't open source code or debug DB queries; instead, focus first on isolation and bypass, as it needs to be handled quickly.

Of course, if you can quickly verify with a query, that's fine... but response is the priority.

RCA (Root cause Analysis) #

Once the fire is out and the server is stable, let's calmly analyze logs and dumps to find the root cause and establish preventive measures.

The purpose is to prevent recurrence of the same incident, and a post-mortem analysis report is written. The 5 Whys method is used to identify the root cause, and action items including systemic defense logic are established and tickets are issued.

A blameless culture is essential; rather than saying "xxx wrote the code incorrectly," it's better to identify blind spots in the CI/CD where this code was deployed or foster a culture of shared responsibility.

Example #

[Detection] Example Prometheus Alertmanager Rule Configuration

To detect a surging payment failure rate before customers complain

groups:
- name: payment_alerts
  rules:
  - alert: HighPaymentFailureRate
    expr: rate(http_requests_total{job="payment_api", status=~"5.."}[5m]) / rate(http_requests_total{job="payment_api"}[5m]) > 0.05
    for: 5m
    labels:
      severity: critical # SEV level to proceed to Triage stage
    annotations:
      summary: "Payment API Failure Rate is too high"
      description: "The 5xx error rate for the payment API currently exceeds 5%. Immediate investigation is required."

[Mitigation] Emergency Circuit Breaker Application using Istio

To prevent our server from crashing due to a failing external API integration when there's no time for RCA. Fault injection.

# When an external API (external-point-service) continuously times out, causing our threads to backlog,
# immediately deploy this configuration to block traffic to that service (Circuit Open).
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: external-point-service-cb
spec:
  host: external-point-service.internal
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 3 # If 3 consecutive 5xx errors occur
      interval: 10s           # Observe for 10 seconds
      baseEjectionTime: 3m    # Do not send requests to that server for 3 minutes (protects our threads with fast failure handling)
      maxEjectionPercent: 100

[RCA] 5 Whys Example (post-mortem)

• Problem: Payment API completely paralyzed for 20 minutes
• Why 1: The payment DB's CPU hit 100%, preventing new connections from being established.
• Why 2: A heavy, slow query for user statistics was run on the payment DB.
• Why 3: The statistics API was deployed to incorrectly point to the write DB instead of the read-only DB.
• Why 4: The separation of environment variables managing DB endpoints was not clear.
• Action Item: Systematically enforce dynamic datasource routing at the JPA/Mybatis level to always route to a read-replica DB when @Transactional(readOnly = true) is used.