The Dangers of High Cardinality

High Cardinality refers to a state in a database or system where a specific index column, metric, or label combination can have a very large number of unique values.

It refers to the number of unique elements in a mathematical set. From a data perspective, gender (male/female) and HTTP methods (GET/POST/PUT) have low cardinality, while email addresses or UUIDs have high cardinality.

Within monitoring systems, the definition refers to the phenomenon where unique time-series data is explosively generated as the combination of metric names and label key-value pairs in time-series databases (TSDBs) like Prometheus and InfluxDB, or in large-scale log systems, grows almost infinitely.

Problem #

High Cardinality is more than just an increase in data; it's a failure point that shakes the foundation of monitoring systems and causes a loss of overall service visibility.

Memory Exhaustion (OOM): Monitoring systems load indexed time-series metadata into memory for fast data processing. If unique combinations increase to millions, system memory usage will surge, eventually leading to process termination by the OOM killer.
Query Performance Degradation and Dashboard Paralysis: When querying data under specific conditions, the engine has to scan an enormous number of indexes and time-series data blocks. This can lead to extremely slow dashboard loading (e.g., Grafana) or query timeouts.
Disk I/O Bottleneck and Storage Waste: Time-series databases use delta compression for data with similar patterns to improve storage efficiency. However, high cardinality data, where values change every time, drastically reduces compression efficiency, causing disk write overhead and rapidly consuming storage space.

Example #

Let's assume we collect a metric called http_requests_total to monitor the response status of a web service API.

In a normal situation, for http_requests_total{method="GET", "status_code"="200"}, method would have 4-5 unique values and status_code around 10. Even combining all of these, only tens to hundreds of time-series data points would be generated, allowing the system to operate very stably.

Let's consider an example of a situation that causes failures. If you deploy a metric with userid and clientid added as labels, with the goal of tracking specific user errors in detail, then

it would look like http_requests_total{method="GET", status_code="200", user_id="1093A4", client_ip="192.168.1.55"}.

In this scenario, if the service has 1 million daily active users (DAU), millions of strange unique time-series data points would explosively appear under a single metric name. Within minutes of deployment, the metric collector would exceed its memory limit and crash, leading to a blind state where the monitoring screen is completely blank precisely when you need to observe a failure.

Solution #

This problem cannot be solved simply by increasing infrastructure specifications. Fundamentally, the dimensionality of the data itself must be controlled.

You can start by simply removing problematic labels during the collection phase. It's also necessary to anticipate the cardinality of a given label beforehand and avoid adding it if it's too high.

# prometheus.yml
scrape_configs:
  - job_name: 'api_server'
    static_configs:
      - targets: ['localhost:8080']
    metric_relabel_configs:
      # If user_id, client_ip, session_id labels are present, ignore them and do not collect.
      - action: labeldrop
        regex: "(user_id|client_ip|session_id)"

It's also possible to identify the culprits through TSDB status analysis. If the system has already reached its memory threshold,

let's identify which metrics and labels are the culprits. You can use TSDB analysis tools.

# Analyze tsdb blocks in the Prometheus data directory
$ promtool tsdb analyze /var/lib/prometheus/data

Block ID: 01H8X7A...
Duration: 2h0m0s
Series: 2504192
Label names: 15
Postings (unique label pairs): 3150201

# 1. TOP 10 metric names with the highest cardinality
High Cardinality Metric Names:
1054320 http_requests_total
 500210 db_query_duration_seconds_bucket
  12000 node_cpu_seconds_total

# 2. TOP 10 label names with the highest cardinality
High Cardinality Label Names:
1540000 user_id
 900500 client_ip
   5040 instance

# 3. Label combinations occupying the most memory
Highest Cardinality Labels:
2014020 {job="api_server", __name__="http_requests_total"}

Based on these results, it can be numerically proven that the user_id and client_id labels are creating countless combinations within the http_requests_total metric.

Based on this, request code modifications.

Finally, separating the roles of metrics and logs is the most fundamental architectural improvement.

Data whose values grow infinitely, such as user IDs, transaction IDs, and IP addresses, should essentially not be stored in a metric system.

If debugging and tracking for such unique values are necessary, instead of adding labels to metrics, you should send application logs containing that information to systems like ELK Stack, Loki, or Datadog Logs, or strictly separate roles to utilize distributed tracing systems.