Control Groups (cgroups)

Why cgroups matter

• Containers. Docker, podman, containerd all use cgroups to enforce per-container CPU and memory caps. docker run --memory=512m --cpus=1.5 writes memory.max=536870912 and cpu.max="150000 100000".
• Multi-tenant isolation. A buggy tenant cannot starve neighbours. Each tenant lives in a sibling cgroup with hard caps; the kernel enforces the partition.
• Fair-share scheduling. cpu.weight (default 100, range 1–10000) gives proportional CPU under contention without hard caps — critical for batch workloads on shared clusters.
• Kubernetes accounting. kubelet polls cgroup stats every 10 seconds for cAdvisor; HPA scales based on cgroup memory/CPU usage, not host stats.
• systemd slices. Every systemd unit lives in a cgroup. systemctl set-property nginx.service MemoryMax=2G writes memory.max on the unit's cgroup at runtime.
• OOM containment. A container exceeding its memory limit dies via per-cgroup OOM killer; the host stays up. Without cgroups, the global OOM killer might pick any process.
• Pressure Stall Information (PSI). v2 exposes cpu.pressure, memory.pressure, io.pressure — fraction of time tasks were stalled. Drives modern autoscalers and oomd.

Anatomy of a cgroup

Every cgroup is a directory in the cgroup virtual filesystem (mounted at /sys/fs/cgroup). Creating a cgroup is mkdir /sys/fs/cgroup/myapp; assigning a process is echo $PID > /sys/fs/cgroup/myapp/cgroup.procs. Limits are written as plaintext files. A pod cgroup tree looks like:

• /sys/fs/cgroup/kubepods.slice/ — root pod cgroup
• kubepods-burstable.slice/ — Burstable QoS class
• kubepods-burstable-pod<UUID>.slice/ — one pod
• cri-containerd-<CONTAINER_ID>.scope/ — one container
• cgroup.procs — list of PIDs in this cgroup
• cpu.max, cpu.weight, cpu.stat
• memory.max, memory.current, memory.events, memory.pressure
• io.max, io.stat, io.pressure
• pids.max, pids.current

Reads are O(1) (kernel maintains atomic counters); writes are validated synchronously. A typical Kubernetes node has 1,000–5,000 cgroup directories, each backed by a few hundred bytes of in-kernel state plus VFS inodes — overhead is negligible compared to the workloads.

The CPU controller in detail

Two parameters shape CPU behaviour. cpu.max is a hard cap (CFS bandwidth controller): "quota period" in microseconds. "50000 100000" means 50 ms of CPU time per 100 ms wall-clock window — equivalent to 0.5 CPUs averaged. The CFS scheduler tracks per-task runtime; when the cgroup's quota is exhausted mid-period, every task in the cgroup is throttled until the period boundary. cpu.stat exposes nr_throttled and throttled_usec — large values mean your latency-sensitive workload is hitting its cap and stalling. Common in Kubernetes; the fix is either raising the limit, lowering the period (controversial), or removing the limit entirely for non-bursty workloads.

cpu.weight is soft: a relative share enforced only under contention. Two cgroups with weight 100 and 200 split CPU 1:2 when both are runnable; if one is idle, the other gets all available CPU. Maps to the CFS task weight via a logarithmic curve.

The memory controller in detail

Memory accounting tracks anonymous pages, page cache, kernel slab, sockets, and swap. memory.max is the hard limit; the kernel will reclaim pages (push to swap, drop clean cache) when usage approaches the limit. If reclaim fails to free enough, the per-cgroup OOM killer fires. memory.high is a soft cap that triggers reclaim before the hard limit — useful for preventing latency spikes from sudden OOM events. memory.swap.max bounds swap usage independently. memory.events counts low, high, max, oom, and oom_kill events; alerts on these are the standard SLO signal for "container is being squeezed."

Common misconceptions

• "cgroups isolate processes." They limit and account; isolation (separate PID space, network stack, mount tree) is the job of namespaces. A container needs both.
• "v2 is incompatible with v1." The kernel can mount both simultaneously (hybrid mode); a process can be in v1 controllers and v2 controllers at the same time. Most distros now ship v2-only because the cleanup is worth it.
• "cgroups have no overhead." The bookkeeping costs O(controllers × tasks) on every fork and a few percent of cache traffic for atomic counters. Real, measurable, but small (1–3% on most workloads).
• "cpu.max on Kubernetes is always good." CFS throttling on a multi-CPU machine can cause p99 latency spikes even when average usage is low — a single-threaded burst exhausts the quota in one core's worth of microseconds. Many shops disable CPU limits and rely only on requests.
• "OOM kill means the cgroup is at fault." Sometimes the kernel under-counts cache or socket buffers; raising the limit or tracking memory.stat is needed before blaming the workload.
• "systemd slices are different from cgroups." A systemd slice is a cgroup. systemctl status shows the cgroup path; cat /sys/fs/cgroup/<slice>/cpu.max shows the live limit.

A concrete example

Run a 256 MB-capped, 0.5-CPU-capped shell under cgroups v2:

• sudo mkdir /sys/fs/cgroup/demo
• echo "+cpu +memory" | sudo tee /sys/fs/cgroup/cgroup.subtree_control
• echo "50000 100000" | sudo tee /sys/fs/cgroup/demo/cpu.max
• echo "256M" | sudo tee /sys/fs/cgroup/demo/memory.max
• echo $$ | sudo tee /sys/fs/cgroup/demo/cgroup.procs
• Then stress-ng --vm 1 --vm-bytes 300M will be OOM-killed; stress-ng --cpu 4 will be throttled to ~50% of one core total.