How to Write Post-Mortem Action Items (Including Runbooks)

Let's say you've successfully extinguished the forest fire that was an incident. Now, from the remaining ashes, you need to work to ensure the mountain doesn't catch fire again.

Have you heard the saying that suffering the same pain more than once is meaningless? While the first time we experience pain, it helps us realize our vulnerabilities, experiencing it more than once teaches us nothing.

Therefore, it's good to try various approaches to foster a culture of recurrence prevention. Let's explore the principles for establishing post-mortem action items, which are the pinnacle of SRE and transform incidents into engineering assets, along with practical runbook writing methods.

The First Principle of Post-Mortem Action Items: "Fix the System, Not the People" #

As engineers, this might sound obvious, but the worst kind of incident post-mortem report is one where the recurrence prevention measures include commitments like "we will test better before deployment" or "the person in charge will double-check cron job settings and get approval." This is the worst because humans inevitably make mistakes, and humans change.

True technical recurrence prevention measures must enforce system-level constraints.

• Bad: Be careful to ensure indexes are used when writing DB queries.
• Good: Integrate slow query detection static analysis tools, SQL Linters, etc., into the CI pipeline to block merges if a PR contains queries that don't use indexes. Deadline: ~~~

The method for deriving concrete Action Items is the 5 Whys. Explore the root cause by asking "why" five times, then establish defensive logic at each step.

• Symptom: The payment server crashed.
• Why?: DB connection pool exhaustion -> AI 1: Add Datadog alert for connection pool exhaustion.
• Why?: An external point integration API was unresponsive for 10 seconds, causing threads to hang -> AI (Action Item) 2: Reduce external API call timeout from 10s to 2s.
• Why?: The timeout setting code was missing, or there were certain constraints here -> AI 3: Add logic to enforce global timeouts when creating Http Client Beans. And so on.

Runbook? #

No matter how well you establish recurrence prevention measures, perfect prevention is impossible. Therefore, runbooks are written to prepare for when the same or similar alerts ring again.

A runbook is a manual written so that even a junior engineer, one month into the job, woken up by a phone call at 3 AM, can bypass an incident within 5 minutes just by copying and pasting commands from the document.

Three Key Requirements for Writing a Runbook #

1. There must be clear entry conditions. There needs to be a unique key, such as "when this specific alert rings, always use this method."
2. Copy-pasteable commands are required. Instead of abstract instructions like "restart the DB," a manual like "type sudo systemctl restart postgresql in the terminal" is needed. Beware of outdated information.
3. An escalation policy is needed, clearly stating who to call if recovery isn't achieved by following the runbook.

Practical Example (Incident Post-Mortem Report and Runbook Template) #

Here's a standard practical template and example that can be copied and pasted directly into your internal wiki (Confluence, Notion).

Post-Mortem Action Items Management Table

This must be included at the bottom of the incident report and linked to Jira tickets for tracking.

Jira Ticket SRE-101, Classification: Monitoring, Action Item: Redis memory usage exceeds 80%, PagerDuty alert integration, Owner: [Name], etc.

# 📙 [Runbook] Redis Cluster Memory Usage > 85% Alert Response

## 1. Overview
- **Trigger Alert Name:** `[SEV-2] Redis Memory Usage is critically high (> 85%)`
- **Impact:** If the cache server experiences OOM, there's a very high risk of traffic flooding the DB (MySQL), leading to a cascading full outage (SEV-1).
- **Primary Goal:** First, free up memory by deleting unnecessary keys; if that fails, immediately perform a scale-out.

## 2. Investigate
When the alert rings, immediately use the following commands to check the current memory status and identify the culprit (the largest key).

```bash
# 1. Connect to Redis server and check memory status
redis-cli -h cache.mywebsite.internal info memory | grep -E "used_memory_human|maxmemory_human"

# 2. Scan for the top 10 largest keys (Big Keys) consuming memory in real-time (Caution: use --bigkeys to minimize impact on the production environment)
redis-cli -h cache.mywebsite.internal --bigkeys

You should also include remediation steps. Following the example above, a migration script is needed to clean up unnecessary or expired keys; let's write it below.

If we add an example scenario where a specific event batch is tangled, causing keys like event:ranking:* to accumulate indefinitely, a migration script like the following can be provided.

# Asynchronously (UNLINK) safely delete keys matching a specific pattern to reclaim memory
redis-cli -h cache.mywebsite.internal --scan --pattern "event:ranking:*" | xargs redis-cli -h cache.mywebsite.internal UNLINK

It's good to have alternative solutions, such as "if step 1 doesn't work, scale out the Redis node" as a step 2. If memory usage exceeds 90%, don't hesitate to add replica nodes via AWS CLI to distribute the load!

aws elasticache increase-replica-count \
    --replication-group-id my-redis-cluster \
    --new-replica-count 3 \
    --apply-immediately

Finally, also include escalation information, such as 010-xxxx-xxxx or @tagging someone.

A well-crafted runbook like this can be the best weapon to distribute incident response knowledge, often concentrated among a few senior engineers, across the entire team.