Node Pressure (OOM, DiskPressure)

When Node Pressure (OOM, DiskPressure) occurs in K8s

Actions on OOM (Memory Pressure) Occurrence #

Kernel Level (Linux OOM Killer)

• When a node's physical memory is exhausted, the Linux kernel's OOM Killer activates.
• It forcibly terminates processes starting with those that have a high OOM Score.
• K8s containers are also regular Linux processes, so they are subject to this.

The priority is roughly as follows:

1. Containers with no requests and large or no limits
2. BestEffort Pods that consume a lot of memory
3. Pods with lower QoS, in order

K8s QoS-based Operation #

Pods are categorized by QoS class.

• Guaranteed: requests == limits, dies last
• Burstable: request < limits, medium priority
• BestEffort: no requests or limits, dies first

When OOM occurs, they die in the order of BestEffort -> Burstable -> Guaranteed.

Kubelet Operation #

When a container terminates due to OOM, its status becomes OOMKilled.

Pods can enter CrashLoopBackOff. The important point is that OOM is not eviction.

It's not the node policy-fully removing the pod, but the kernel picking and killing it. So it's a bit different.

Actions on DiskPressure Occurrence #

DiskPressure occurs when the following resources fall below a threshold:

• node filesystem /
• image filesystem (container image storage space)
• emptyDir or localstorage

Kubelet has internal eviction thresholds.

• nodefs.available < 10%
• imagefs.available < 15%

Node Condition also changes when DiskPressure occurs:

NodeCondition:
- DiskPressure = True

When this state is reached, the following rules apply.

Pod Eviction Operation (Policy-driven Removal) #

DiskPressure operates based on eviction.

1. Blocks new pod scheduling, making it impossible to schedule new pods on that node
2. Evicts lower-priority pods first: BestEffort, low QoS pods, pods heavily using emptyDir
3. Pods are gracefully terminated (SIGTERM) -> grace period -> SIGKILL

In other words, unlike OOM, it cleans up in an orderly manner.

The order in which Kubelet attempts to clean up resources is:

1. Delete logs of dead containers
2. GC unused images
3. If that's not enough, pod eviction

OOM vs DiskPressure #

OOM is a signal of memory design failure; you should carefully check request/limit values, look for memory leaks, or review GC tuning. This is a state that must be prevented.

DiskPressure indicates a failure in cleanup strategy; it can be prevented by managing image GC cycles, preventing infinite log growth due to management failures, or regularly cleaning up excessive emptyDir usage.