CPU and Memory Control Methods in cgroup

In the Linux kernel, cgroup (control groups) is not merely a basket for grouping processes.

It is a core subsystem that abstracts, distributes, and isolates system resources.

Within the kernel, it has a hierarchical structure, where each process belongs to a specific cgroup, and groups are linked in a tree structure.

• cgroup v1 vs v2: In the past, hierarchical structures were separated by resource, leading to lower management efficiency. However, with the current standard v2 Unified Hierarchy, all resource controllers are managed under a single tree.
• Controller: A kernel module that manages specific resources such as cpu, memory, and io.

CFS, Scheduling #

Linux primarily uses CFS (Completely Fair Scheduler) to distribute CPU time.

cgroup intervenes in the operation of CFS to allocate resources.

Weight-based Allocation (Shares) #

CPU is allocated based on relative proportions rather than absolute values.

• Mechanism: Each cgroup has a cpu.weight value. It occupies CPU time proportional to its share of the total weight.
• Characteristic: When the system is idle, it uses the CPU without restriction, but when contention occurs, it limits usage to the configured proportion. This is known as a work-conserving approach.

Quota/Period-based Allocation #

It enforces an absolute time that can be used during a specific period.

If the quota is fully used during the period, all processes in that cgroup enter a throttling state until the next period.

This is a point to be cautious about in services where real-time performance is critical.

Memory Control Mechanism: RSS, Page Cache #

Unlike CPU, memory is a resource that occupies space rather than being time-shared, so its control method is more complex and risky.

Memory Accounting #

The kernel increments the counter for the corresponding cgroup whenever a process allocates memory.

• This includes anonymous memory RSS, page cache, kernel memory slab, and more.
• It is divided into memory.max (hard limit) and memory.low/high (soft limit, protection).

Memory Reclamation and OOM Killer #

• Reclaim: When a cgroup reaches its limit, the kernel immediately attempts page reclaim, primarily by clearing the page cache.
• OOM: If there is no memory to reclaim but more is requested, the kernel invokes the oom killer.
• Selection: It terminates the process with the highest oom_score within the cgroup. In v2, the memory.oom.group setting can also terminate the entire group at once.

Kernel Internal Working Principles #

cgroup provides an interface through a VFS (Virtual File System) called /sys/fs/cgroup.

When we modify files in this directory, the value of the cgroup_subsys_state structure inside the kernel changes, and the scheduler and memory manager (MM) refer to this value to determine their actions.

You can remember it as a Resource Counter and a virtual file system.

Additionally, if the CPU quota is set too tightly, processes may be interrupted before completing their tasks,

leading to a sharp increase in Tail Latency. To monitor this, you must check the nr_throttled value in the cpu.stat file.

The isolation reality of modern Kubernetes or Docker containers is precisely this combination of cgroup and namespace isolation.