Interpreting CPU Load Average

CPU Load Average is a key metric indicating system load, but it differs from simple CPU utilization.

Load Average #

In Linux systems, load average refers to the average number of processes (tasks) that are either running or waiting to run.

Specifically, it retrieves data from the /proc/loadavg file and measures the sum of processes in the following two states:

R (Running / Runnable): Processes currently using the CPU or waiting in the run queue to use it.
D (Uninterruptible Sleep): Processes waiting for disk I/O or network responses. This state is uninterruptible by signals and is included in Linux's unique load calculation method.

When you run top, uptime, or htop commands, three numbers are displayed. These represent the Exponential Moving Average values over the last 1, 5, and 15 minutes, respectively.

uptime: Shows system uptime and load average concisely.
top: Displays overall system status and a list of processes; load average appears on the first line at the top.
htop: Provides an intuitive view of per-core load status and load average.

$ uptime
 14:20:05 up 35 days,  2:45,  3 users,  load average: 4.15, 2.30, 1.05

$ top - 14:22:10 up 35 days,  2:47,  3 users,  load average: 8.50, 5.20, 3.10
Tasks: 250 total,   2 running, 248 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.0 us,  5.0 sy,  0.0 ni, 10.2 id, 69.8 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  16000 total,   2000 free,   8000 used,   6000 buff/cache
MiB Swap:   4096 total,   4000 free,     96 used.   7500 avail Mem

$ htop
1  [||||||||||           25.0%]   Tasks: 145, 325 thr; 3 running
2  [||||||               15.0%]   Load average: 0.65 0.88 1.12 
3  [||||||||||||||||     45.0%]   Uptime: 10 days, 05:12:30
4  [||                   05.0%]
Mem[|||||||||||||    4.50G/16G]
Swp[|                120M/4.0G]

If a server has 4 cores, based on the uptime output, the current (1-minute) load is 103% (4.15/4). The load has sharply increased compared to 15 minutes ago, indicating the system is slightly exceeding its processing capacity.

The reason for using exponential moving average is that a simple arithmetic average wouldn't accurately reflect the current load state due to past data.

It applies a higher weight to recent data, allowing older data to decay.

This is for understanding the trend:

1 min > 5 min > 15 min: Load is rapidly increasing.
1 min < 5 min < 15 min: Load has peaked and is decreasing.
If the three values are similar, it indicates stable maintenance.

Threshold Judgment Criteria #

The absolute value of the Load Average alone cannot determine if a system is overloaded; it must always be compared with the number of logical CPU cores.

If Load Average / Number of CPU Cores = 1.0, it means that core is running continuously with no queue.
If it's less than 1.0, CPU resources are available.
If it's greater than 1.0, CPU resources are insufficient, causing processes to wait in the run queue or experience delays due to I/O bottlenecks.

In a 4-core system, a Load of 2.0 means 50% of resources are being used. In a 1-core system, a Load of 2.0 means one process is running, and another is waiting.

Load Average vs CPU Utilization #

Many administrators confuse CPU utilization % with Load Average. Let's clarify the critical differences.

CPU Utilization

The measured value is the proportion of time the CPU spends on computation during a specific period.
I/O wait is not reflected; the CPU is considered idle during I/O wait.
100% is the maximum per core.

Load Average

The measured value is the number of running and waiting processes (demand).
Strongly reflects I/O wait.
There is no maximum value; it can increase indefinitely as the queue lengthens.

If CPU utilization is low but load average is high, it might not be a lack of computational power but rather a disk I/O bottleneck. This can be inferred.

Or it could be a response delay from a Network File System (NFS). In any case, it's not a lack of computational ability.

Mathematical Calculation Principle #

The Linux kernel counts the number of currently running/waiting tasks every 5 seconds. The exponential moving average formula is approximately as follows:

Load_{new} = Load_{old} \times e^{-5/60} + n \times (1 - e^{-5/60})

n = number of currently measured running/waiting tasks, e is Euler's number.

This formula prevents short-term load spikes from distorting the overall average while still reflecting continuous load trends.

You can check the number of cores using something like grep -c ^processor /proc/cpuinfo.

If the current load average seems unusual, first check the 15-minute average to determine if it's a temporary phenomenon or a structural issue.

Also, track D-state processes: if the load is high but CPU is low, check ps -aux to see if many processes are in D (waiting) state, and then examine storage or network I/O.

When Both Load Average and CPU Utilization are High #

This means the system is CPU-bound, where processes are waiting to run due to insufficient CPU computational resources.

In such cases, first identify the culprit process. Use top or htop, press 'P' to sort by CPU usage, and then check the PID, USER, %CPU, and COMMAND columns.

First, determine if a specific process is monopolizing resources or if multiple processes are evenly consuming them.

ps -eo pid,ppid,cmd,%cpu --sort=-%cpu | head -n 10

After that, analyze the nature of the load (CPU State Classification).

Analyze the CPU% line at the top of top to narrow down the cause of the problem.

%us is user-mode application computation. If this is high, it indicates an application logic issue, such as an infinite loop or excessive computation, requiring code optimization.
%sy is kernel-mode operation, such as software interrupts. If this is high, it could be due to excessive system calls, context switching, or network stack load.
%ni indicates low-priority tasks like background jobs, backups, or indexing consuming resources, and priority adjustment may be needed.
%si is software interrupt processing. This could mean too much network packet processing or a driver issue.

In-depth Diagnosis #

Once you've identified the process, you need to investigate what's happening inside it.

To check multi-threading, use top -H -p PID to see which threads within a specific process are using the CPU.

Trace system calls with strace -p PID -c to see which system calls a process is excessively invoking, and monitor per-process CPU usage trends at specific intervals with pidstat -u 1.

Check vmstat 1 to see if context switching (cs) is high (tens of thousands or more per second), and determine if the CPU is wasting too much time on process switching rather than actual computation.

You can take emergency action by lowering the priority of process 1234 to 19 (lowest) using renice -n 19 -p 1234. Killing it... you can do that if possible.

Alternatively, you can enforce usage limits with cpulimit -p [PID] -l 50.

Fundamentally, you can improve cache efficiency and reduce context switching by setting CPU affinity, using taskset to bind specific processes to specific cores.
Configure Cgroups to set resource limits for each service at the system level (this would be easier in a Kubernetes environment).
Scaling up or out is often the easiest solution, haha. Also, try to optimize application code as much as possible.

Find the culprit PID with top.
If %us is high, suspect application code/logic.
If %sy is high, suspect system calls or context switching.
If urgent, lower priority with renice or terminate with kill.