estudy 한국어/English

Performance Issues Arising from NUMA Architecture

1,449 단어·7 분·원문(.md)

NUMA (Non-Uniform Memory Access) architecture was introduced to maximize performance in modern multi-core systems.

However, if applications or systems are configured without a proper understanding of its characteristics, severe performance degradation can occur.

NUMA #

In the past, multi-processor systems used a UMA (Uniform Memory Access) architecture where CPUs shared a single memory bus.

While this wasn't an issue with a small number of cores, as the core count increased, the memory bus became a bottleneck.

NUMA emerged to solve this problem.

  • NUMA Node: A bundle of a CPU and the physical RAM directly connected to that CPU.
  • Local Memory Access: When a CPU accesses memory belonging to its own node. It's very fast because the CPU monopolizes the bus.
  • Remote Memory Access: When a CPU accesses memory connected to another node. It's relatively slow and incurs latency because it must pass through inter-node connections (such as Intel's QPI/UPI, AMD's Infinity Fabric).
  • First-Touch Policy: Most operating systems, including Linux, allocate memory to the CPU node where the thread that first touches that memory is running.

Performance Issues #

Remote memory access latency can occur. If a CPU frequently references memory on another node, memory access speed can significantly drop, leading to longer CPU stall times while waiting for data.

Inter-node interconnect bottlenecks also exist. The communication bandwidth between nodes is limited. If large amounts of data frequently traverse these inter-node paths, bandwidth saturation can occur, degrading system performance.

Memory Imbalance (NUMA Imbalance) and Swapping can also happen. Let's assume Node 0's memory is 100% utilized, while Node 1's memory is almost empty. If an additional memory allocation request comes for Node 0, the OS might swap out Node 0's data to disk rather than using Node 1's available memory.

Real-world Problem Examples #

Databases like MySQL and PostgreSQL can experience "swap insanity." This is when the total available system memory exceeds 30GB, but a DB process consumes all memory on a specific node, leading to disk swapping and a sudden tenfold slowdown in query processing speed.

In-memory caches (Redis, Memcached) can also be affected, showing inconsistent and spiky response times. If a Redis thread running on Node 0 is context-migrated to Node 1 by the OS scheduler, it will have to continuously read its cached data from Node 0 remotely, which can drastically reduce performance.

In Java applications, if JVM GC threads are scattered across multiple nodes, the garbage collection process might involve a large amount of remote access during memory scanning, leading to abnormally long stop-the-world (STW) times.

How to Identify the Issue #

First, let's check the system's NUMA topology. Use numactl --hardware to verify the node configuration and the distances between nodes. Greater distances imply slower access.

$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 32768 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 32768 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10

The access distance from Node 0 to Node 0 is 10 (local), while remote access from Node 0 to Node 1 is 21 (remote), indicating it can be more than twice as slow.

Local memory access differs in nanoseconds; 70-100ns is ideal for local memory access. Remote memory access with about 1 hop can be 130-150ns, roughly 1.5-2 times slower. With 2 or more hops, it can be 200-300ns, about 3 times slower. This can occur on large servers with 4 or more sockets, and latency increases sharply as data must traverse multiple paths.

To measure actual latency on your server in nanoseconds, you can use Intel Memory Latency Checker (MLC).

# Download and run Intel MLC to see a matrix like this:
$ ./mlc --idle_latency

Numa node
Numa node            0       1
       0          78.5   135.2   <-- 135.2ns taken to go from Node 0 to Node 1
       1         136.1    79.2

You can also check NUMA memory allocation and miss rates using numastat.

$ numastat
                           node0           node1
numa_hit                85032910        12049582
numa_miss                5902104         8920194  <-- Caution!
numa_foreign             8920194         5902104
interleave_hit             32014           31988
local_node              84901020        11902030
other_node               6033994         9067746  <-- Caution!

If numa_miss and other_node values are high and continuously increasing, it's strong evidence that threads are frequently accessing remote memory, leading to performance degradation.

To check for node-specific memory imbalance, use numastat -m or cat /proc/zoneinfo to see if a specific node's memory is depleted, risking swapping.

$ numastat -m
                 Node 0          Node 1           Total
                -------         -------         -------
MemTotal       32768.00        32768.00        65536.00
MemFree          250.00        28000.00        28250.00 <-- Imbalance!
MemUsed        32518.00         4768.00        37286.00

While total memory is ample, Node 0's memory is almost full.

In this state, if Node 0's CPU requests additional memory, swapping may occur.

Solutions and Optimization Strategies #

Solutions to performance issues are broadly categorized into NUMA-friendly configurations and simply disabling NUMA, depending on the application's characteristics.

Memory Interleaving - Suitable for DBs, etc. #

Used when a single large process, like MySQL, needs to use memory evenly.

It ignores the first-touch policy and distributes memory allocations across all nodes in a round-robin fashion.

While remote access still occurs, it prevents swapping caused by memory depletion on a single node.

$ numactl --interleave=all /usr/local/mysql/bin/mysqld &

Alternatively, set the kernel parameter vm.zone_reclaim_mode = 0 (which is often the default) to allow aggressive use of remote node memory.

CPU/Memory Pinning/Binding - Suitable for Redis Cache, HPC #

This forces a specific process to use only the CPU and memory of a particular NUMA node, ensuring it never accesses remote memory.

Use numactl to specify the node.

# Restrict Redis server to use only CPU and memory of NUMA Node 0
$ numactl --cpunodebind=0 --membind=0 redis-server /etc/redis/redis.conf

Automated Tools at OS Level - numad #

numad is a Linux daemon that monitors a process's memory access patterns and dynamically migrates threads and memory pages to the same node.

It's useful when manual configuration is difficult.

BIOS Node Interleaving Setting (Hardware Level) #

If NUMA tuning is particularly challenging and all processes need to run uniformly, you can change the Node Interleaving setting to "Enable" in the server BIOS. This makes the hardware behave like a UMA system.

However, this slightly increases overall memory latency, so it should be used as a last resort.

Conclusion #

We've explored the performance issues arising from NUMA and their solutions.

One might think that since these are nanosecond-level differences, they might not be worth worrying about, but that's half true and half false.

This phenomenon reacts much more immediately to system memory usage and physical hardware configuration than to RPS (requests per second).

In other words, even with extremely high traffic, a well-configured NUMA system will be fine, but even with moderate traffic, a system with incorrect memory settings can crash.

This is because it's not a phenomenon that adds latency to every single request. If swapping occurs, for example, in a traffic environment of 1000-2000 QPS, at some point, if Node 0's 64GB of memory becomes full,

and the OS swaps out Node 0's data to disk, leaving Node 1's remaining 64GB untouched, then disk I/O will spike, and query response times, normally under 10ms, can freeze for several seconds.

Alternatively, in an in-memory cache, if the NUMA node where the network interface card (NIC) is physically connected differs from the NUMA node where the Redis process is running, the NIC must pull the received data to Redis on a different node. This means interrupts are handled on Node 0, but Redis is running on Node 1.

Because Redis is so fast, 10,000-20,000 requests per second might not show noticeable inter-node latency. However, in systems handling 50,000-100,000 requests, the bandwidth of inter-node connections (QPI/UPI) can become saturated, causing Redis, which normally has sub-1ms response times, to spike to tens of milliseconds. This is a critical issue in high-performance environments.

In other words, this absolutely won't happen on small servers or in typical cloud environments, but it might occasionally occur if data management is poor.

However, this is likely only an issue on enterprise-grade bare-metal physical servers with 64-218GB or more memory. Still, NUMA tuning is a top priority.

If a single process is heavily wasting memory, just enabling tuning options can feel like a 20-30% performance (response time) improvement. It's good if you're using an RDBMS, especially if you're running a single DB instance with a buffer pool consuming tens of GB.

For large-scale in-memory systems, it depends more on traffic, but it's in an ambiguous position where you might not need to worry about it at all. However, infrastructure administrators who split a single server into multiple heavy VMs and directly serve them might need to pay attention.

SRE/question/q_24.md