Priority Judgment Criteria During Outage Response

When an outage occurs, a common fatal mistake is wasting time trying to figure out why it happened.

One might wonder why this isn't important, but the point is that when customers can't make payments, it's better to ensure their success rather than debugging by looking at thread dumps or code.

Root cause analysis is done after the service has been restored or during a post-mortem retrospective. In an outage situation, the top priority is to restore the service to its previous state by any means necessary.

3 Key Criteria for Determining Priority #

Not all APIs and services have equal value. Determine what needs to be restored first based on the service's tier.

Tier 1: Authentication, login, payment, and core domains are always the top priority.
Tier 2: Search, recommendations, profile editing – inconvenient if down, but not directly losing money.
Tier 3: Internal admin, statistics batches, log collection – it's acceptable to boldly take down servers during an outage.

While this isn't universally true, tiers can generally be defined in this manner.

Risk of Cascading Failure #

Assess whether a single failure risks bringing down other healthy systems.

An isolated failure, where only a specific recommendation API returns a 500 error and nothing else, has a lower priority.

A propagating failure, for example, if a recommendation API slows down and monopolizes the shared DB connection pool, causing even healthy payment APIs to time out (highest priority). The top priority action is to block the recommendation API or circuit break it to save the database.

Speed of Action #

Can it be immediately turned off with a switch, like a feature toggle? Or does it require a hotfix deployment?

It's also important to prioritize actions that can be applied within a minute, such as a rollback to a previous version or a scale-out, over time-consuming tasks.

Practical Outage Classification #

In global IT companies and real-world environments, the SEV classification system is commonly used. When an alarm goes off, this severity level is declared first, and then action is taken.

SEV-1: Critical outage, causing full service downtime, payment failure, or large-scale data loss. Responses include immediate rollback, traffic redirection (DR), and summoning all engineers.
SEV-2: Major outage, such as core functionalities (e.g., search) becoming unresponsive. Initial responses include turning off the feature via a feature toggle or scaling out servers.
SEV-3: Minor outage, such as intermittent delays for some users or errors in non-critical auxiliary functions. After identifying the cause, it can be addressed with a hotfix in the next deployment.
SEV-4: Internal office network instability or typos. This is typically handled by creating a ticket during business hours.

Example #

Here are typical examples of bypass and recovery actions that should be performed before debugging when an outage occurs.

For instance, if an outage occurs immediately after a deployment, and monitoring graphs are plummeting after deploying a new version, don't look at logs; proceed with an immediate rollback.

# 1. Check the deployment history of a specific deployment
kubectl rollout history deployment/my-backend-app

# 2. Immediately roll back to the previous stable state (revision) (the most commonly used magic command)
kubectl rollout undo deployment/my-backend-app

# 3. Check the real-time status to see if the rollback is progressing well
kubectl rollout status deployment/my-backend-app

If a specific API is killing the database, emergency blocking can be done with something like Nginx.

If a heavy API query like /api/v1/heavy-stats is experiencing a traffic surge, exhausting DB connections, and causing a cascading failure, there's no time to modify backend code. Immediately block that specific API at the Nginx frontend, returning a 503 response to protect the system.

# nginx.conf (or relevant server configuration)
server {
    listen 80;
    server_name api.mywebsite.com;

    # 🚨 Emergency Block: Temporarily return 503 error for heavy stats API (service protection)
    location /api/v1/heavy-stats {
        return 503 "Service Temporarily Unavailable due to high load";
    }

    # Normal traffic passes to the backend
    location / {
        proxy_pass http://backend_servers;
    }
}

This feels like a form of fault injection to block it and prevent cascading failures. Apply it using nginx -t && nginx -s reload (applies settings immediately without restarting Nginx) as in the example above.

Users calling that API will receive an error, but since the database will recover, the most critical APIs can be immediately restored to normal, making it a worthwhile sacrifice.