How OOM Killer Works

The OOM Killer is a mechanism that forcibly terminates processes as a last resort when the Linux kernel can no longer allocate memory.

OOM Killer Trigger Conditions #

1. Physical memory shortage
2. Swap space is also insufficient or its use is restricted
3. Memory reclaim failure
   1. page cache drop
   2. slab shrink
   3. memory compaction
4. Situation where the kernel can no longer return allocation failure to the caller

At this point, the kernel determines that someone must die to save the system.

Overall Flow #

Step 1: Allocation failure. A process calls malloc() -> __alloc_pages() fails internally within the kernel.

Step 2: Attempt memory reclaim by sequentially trying the following:

1. Reclaim page cache
2. Shrink slab cache
3. Memory compaction
4. Swap out (if possible)

If all these fail, the system enters an OOM state.

Then, the situation is assessed in Step 3.

The kernel function out_of_memory() determines whether the OOM killer is active and checks for memory cgroup involvement.

After that, it decides which process to kill.

There's a concept called OOM Score. The kernel calculates an OOM score for all processes:

oom_score = f(메모리 사용량, 중요도, 보호 여부)

The higher the score, the higher the probability of being killed.

Memory usage (RSS: actual physical memory usage), and whether page cache is included, are among the main evaluation factors. Processes that use a lot of memory are prioritized for termination.

Processes with a high nice value are also more likely to be killed (a number that indirectly adjusts CPU priority).

nice value: Range from -20 to +19, where a smaller value means higher priority. A high nice value means it's a 'nice' process that readily yields CPU to other processes... haha.

oom_score_adj range: -1000 to +1000

• -1000: Kernel thread level, never killed
• 0: Default value
• +1000: Killed first

This value is also checked.

There are also special protected targets: root processes, systemd, init, kernel threads, and those with oom_score_adj=-1000 are excluded from OOM killer targets as they should not be killed.

Actual Kill Mechanism #

Once a target is determined, the kernel performs the following:

1. Sends SIGKILL
2. Immediately terminates the process
3. Frees memory
4. Attempts to restore the system to a normal state

Out of memory: Kill process 12345 (java) score 987 or sacrifice child
Killed process 12345 (java) total-vm:4096000kB, anon-rss:2048000kB

cgroup-based OOM (Container Environment) #

The difference in k8s/Docker is that OOM occurs at the cgroup level for containers.

• Node-wide OOM -> system killer
• Container memory exceed -> cgroup OOM killer

The difference is that only processes within the container are terminated, and the node remains alive.

OOMKilled: true
ExitCode: 137

The OOM killer is not a bug; while it can be a result of a memory leak, it can be considered normal operation.

The OOM killer doesn't negotiate. This means there's no SIGTERM; it immediately sends SIGKILL, and finally blocks or shutdown hooks are not executed (because it's a desperate attempt to save the kernel).

In summary, the OOM Killer is a last resort that immediately terminates the process consuming the most memory or the least critical process to save the system when memory allocation becomes completely impossible.