Alertmanager's Role

Alertmanager is one of the core components of the Prometheus monitoring ecosystem. It receives alert notifications generated by the Prometheus server based on collected metrics and delivers them to the appropriate recipients.

Simply put, it acts like a central post office, controlling who gets notified, how, and how often when a system issue arises.

Key Flows #

1. deduplication When operating multiple Prometheus instances for high availability, several servers might simultaneously send the same alert for an identical issue, leading to excessive noise. Alertmanager identifies such duplicate alerts and consolidates them into a single notification.
2. grouping It groups alerts of a similar nature and sends them as a single notification message. This prevents an "alert storm" of hundreds of individual notifications during a large-scale outage, helping responders quickly identify core issues.
3. routing It determines the recipients for alerts based on conditions such as labels or severity assigned to the alerts. slack, email, pagerduty, opsgenie, webhook etc...
4. inhibition Also known as suppression, this blocks notifications for related subordinate/less critical alerts when a specific alert occurs. For example, suppressing individual server unreachable alerts when the entire network is down.
5. silencing This temporarily mutes alerts for a specific duration and under certain conditions (label matching) during scheduled maintenance or for known ongoing incidents.

ex #

Grouping Example

• Scenario: One database server in a cluster goes down, causing 50 web server instances that rely on it to simultaneously generate error alerts.
• Instead of sending 50 individual alerts, Alertmanager groups them based on labels like cluster=db-tier or alertname=DatabaseConeectionFailed and sends a single summarized alert to Slack, stating "DB connection error occurred on 50 web servers."

Routing by Label (Recipient Separation) Example

• Scenario: Various severity alerts have occurred in the system.
• Alertmanager's behavior is as follows:
  • severity=warning: Quietly sends a message only to the development team's Slack channel, for issues like disk usage reaching 80%.
  • severity=critical: For situations like a main server going down, it triggers the on-call person's PagerDuty and sends text messages, waking them up in the middle of the night to work. It also sends detailed information via email.
  • team=frontend: Forwards to the frontend team channel.
  • team=database: Forwards to the DBA channel.

Inhibition Example

• Scenario: Power supply to a specific rack in the IDC is cut, causing all 20 servers in that rack to go down.
• In this case, Alertmanager generates a RackDown alert indicating the entire rack is down, and it does not generate 20 subordinate InstanceDown alerts for individual servers within that rack. This allows focus to remain on the fundamental rack down issue.

global:
  # Time until an alert is considered resolved
  resolve_timeout: 5m

# 1. Routing & Grouping
route:
  # Default recipient for alerts that don't match any other conditions
  receiver: 'default-slack'
  
  # Grouping criteria: Group alerts with the same alertname and cluster (to prevent alert storms)
  group_by: ['alertname', 'cluster']
  
  # Grouping timer settings
  group_wait: 30s      # Time to wait for more alerts of the same group after the first one
  group_interval: 5m   # Time to wait before sending an updated group alert when a new alert is added
  repeat_interval: 4h  # Interval to resend the same alert if the issue is not resolved

  # Sub-routing rules (deliver to different recipients based on conditions)
  routes:
    # Condition A: For frontend team-related issues
    - matchers:
        - team="frontend"
      receiver: 'frontend-slack'

    # Condition B: For critical severity issues (requires waking up in the middle of the night)
    - matchers:
        - severity="critical"
      receiver: 'pagerduty-critical'

# 2. Inhibition Rules
inhibit_rules:
  # If a RackDown alert occurs, ignore InstanceDown alerts for servers within that rack
  - source_matchers:
      - alertname="RackDown"
    target_matchers:
      - alertname="InstanceDown"
    # Inhibition only applies when the 'rack' label value is the same for both source (RackDown) and target (InstanceDown)
    equal: ['rack']

# 3. Receiver Configuration
receivers:
  # Default Slack channel
  - name: 'default-slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        channel: '#general-alerts'

  # Frontend team Slack channel
  - name: 'frontend-slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        channel: '#frontend-alerts'
        title: '{{ .GroupLabels.alertname }} Alert Occurred!'
        text: 'Frontend issue occurred in cluster: {{ .GroupLabels.cluster }}.'

  # PagerDuty (sends calls/texts to on-call personnel)
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_INTEGRATION_KEY'
        description: 'URGENT! {{ .GroupLabels.alertname }} occurred'