cgroups v2 Explained for Sandboxing Untrusted Code

Every few months someone ships a feature that runs model-generated code, wraps it in a container with a memory limit, and calls it sandboxed. The limit is real. The sandbox is mostly vibes. cgroups v2 does exactly one job — deciding how much of a machine a group of processes may consume — and it is routinely asked to do a different job it was never designed for: deciding what those processes may reach.

This is the precise version: what the unified hierarchy changed, what each controller really enforces, the knobs nobody sets until the outage, and then the honest section — where cgroups stop, what namespaces, seccomp and LSMs cover instead, and what changes when the boundary is a hypervisor rather than a directory in /sys/fs/cgroup. I built PandaStack on Firecracker microVMs so I have an opinion about the last part, but most of this is just Linux.

The unified hierarchy: what v2 actually changed

In cgroups v1 every controller got its own independent hierarchy: memory in one tree, cpu in another, blkio in a third, and a process could sit somewhere different in each. Flexible in the way a box of loose wires is flexible. It also made joint accounting nearly impossible — the memory controller had no coherent way to talk to the io controller's writeback path, because they disagreed about which group a page belonged to. Buffered-write throttling in v1 was, charitably, aspirational.

cgroups v2 collapses this into a single tree. One hierarchy, one place a process lives, and controllers switched on per-subtree by writing to cgroup.subtree_control in the parent. That last detail is the one people trip over: a controller must be handed down from above before a child cgroup gets the corresponding files. You don't enable memory on a cgroup; you enable it on that cgroup's parent, for its children. cgroup.controllers lists what's available, cgroup.subtree_control what's been passed down.

The second structural rule is "no internal processes": a non-root cgroup cannot both contain processes and distribute resources to children through enabled controllers. Processes live in the leaves. That feels annoying until you realise it's what makes the accounting well-defined — a parent that both competed with its children and arbitrated between them was where half of v1's weirdness lived. The root cgroup is exempt, which is why examples start by writing to root's subtree_control.

The controllers, and what each one really enforces

memory: a hard wall, a soft wall, and fuzzy accounting

memory.max is the hard limit. Cross it and the kernel reclaims aggressively; if reclaim can't free enough, the cgroup OOM killer fires and kills something inside that cgroup. Note the scoping — not the global OOM killer picking a victim across the host, but a contained execution. The runaway dies, the host survives, and memory.events records oom and oom_kill counters so you can tell an OOM from an ordinary crash after the fact.

memory.high is the more interesting and more under-used knob. It isn't a wall, it's backpressure: past that threshold the kernel puts allocating tasks under heavy reclaim and throttles them with a proportional delay. Nothing is killed — the workload just gets slow, deliberately, which is often what you want for a batch job. Set memory.high below memory.max for a degradation band before the guillotine, and set memory.swap.max to 0 unless you want a memory-limited process quietly relocating its problem into swap.

Now the surprising part: "how much memory did it use" is fuzzier than memory.current suggests. v2 charges page cache to the cgroup that first faulted a page in, and that charge outlives the process. It also charges kernel memory: slab, socket buffers, kernel stacks, page tables. A process that reads a large file and exits leaves its cgroup looking fat; one doing heavy network I/O is charged for skb memory that never appears in its RSS. Read memory.stat, not memory.current — the anon/file/sock/slab breakdown is the difference between "my app leaks" and "my app read a big file once."

cpu: quota is predictable, weight is fair, and you rarely want quota

Two fundamentally different models here, answering different questions. cpu.max is a bandwidth quota written as "$MAX $PERIOD" in microseconds — "20000 100000" means 20ms per 100ms period, i.e. 0.2 of a core, a hard cap whether or not the machine is busy. cpu.weight (1–10000, default 100) is a proportional share that only matters under contention: with nobody else competing, a weight-50 cgroup can use the whole box.

The footgun is quota, and it's a leading cause of unexplained tail latency in container fleets. Once the budget is spent, quota freezes the cgroup's runnable tasks for the rest of the period: a multi-threaded runtime can burn a 100ms budget in 10ms of wall time across eight threads, then sit dead still for 90ms. Average utilisation looks comfortable; p99 looks like a crime scene. cpu.stat exposes nr_throttled and throttled_usec — alert on those, not utilisation. Weight is usually what you want; quota is for selling a predictable slice that behaves identically on idle and busy hosts.

io: real, but conditional on your storage stack

io.max sets hard bps/iops caps per block device (keyed by major:minor); io.latency expresses a target latency so the kernel throttles competing groups to protect one; io.cost/io.weight give proportional sharing via blk-iocost. All good, all with one asterisk: this works on block devices with cgroup writeback support, and buffered-write attribution needs filesystem cooperation. Push I/O through a device-mapper stack, a loop device or a network filesystem and the charge can land on a kernel worker thread that belongs to nobody. Direct I/O to a plain NVMe device behaves as documented; further from that, measure rather than assume.

pids: the one nobody sets until they get forked to death

pids.max caps processes and threads in a cgroup, and it is the only real defence against a fork bomb. Not memory.max — a fork bomb needs PID space and task structs, not memory, and by the time memory pressure registers you can't fork a shell to fix it. Not cpu.max either; the damage is structural, not compute. One integer, costs nothing, turns the classic :(){ :|:& };: from a host-wide denial of service into a process failing clone() with EAGAIN. If you run other people's code and haven't set pids.max, set it now.

cgroup.kill, cgroup.freeze, and delegation

Three v2 features worth knowing. cgroup.kill: write 1 and every process in the cgroup and its descendants is SIGKILLed atomically — no PID-list race, no forking faster than you can kill, which is exactly the problem with something hostile. cgroup.freeze: write 1 and the subtree stops in place, without the raciness of SIGSTOP-per-PID. Delegation: chown a subtree's cgroup.procs and cgroup.subtree_control to an unprivileged user and they manage their own limits below your ceiling — systemd's Delegate=yes does this.

Don't hand-write cgroupfs; systemd already speaks it

On any modern distro systemd is the single writer of the cgroup tree, and poking /sys/fs/cgroup directly means your tuned values get reset the next time a unit reloads. MemoryMax=, MemoryHigh=, MemorySwapMax=, CPUQuota=, CPUWeight=, TasksMax= and IOReadBandwidthMax= are the same knobs with persistence, and systemd-run --scope gets you an ad-hoc cgroup for a one-off command. Below: the raw version, then the version you should ship, then two containment demos.

#!/usr/bin/env bash
# cgroup v2 tour. Assumes a unified hierarchy (kernel >= 5.8, systemd >= 244).
set -euo pipefail

# 0. Confirm you are on v2 and not the v1/hybrid museum piece.
mount | grep -q 'cgroup2 on /sys/fs/cgroup' && echo "unified hierarchy: yes"
cat /sys/fs/cgroup/cgroup.controllers   # cpuset cpu io memory hugetlb pids rdma misc

# ---- 1. Raw cgroupfs, so you know what the abstractions are hiding ----
CG=/sys/fs/cgroup/untrusted
sudo mkdir -p "$CG"
# Controllers are handed DOWN: enable them in the PARENT, for its children.
echo "+memory +cpu +pids" | sudo tee /sys/fs/cgroup/cgroup.subtree_control >/dev/null

echo 256M           | sudo tee "$CG/memory.max"      >/dev/null  # hard wall -> cgroup OOM kill
echo 200M           | sudo tee "$CG/memory.high"     >/dev/null  # soft wall -> reclaim + throttle
echo 0              | sudo tee "$CG/memory.swap.max" >/dev/null  # no swap escape hatch
echo "20000 100000" | sudo tee "$CG/cpu.max"         >/dev/null  # 20ms per 100ms = 0.2 CPU
echo 64             | sudo tee "$CG/pids.max"        >/dev/null  # the anti-fork-bomb line
# Join it with:  echo $$ | sudo tee "$CG/cgroup.procs"

# ---- 2. The same thing via systemd, which is what you should actually ship ----
sudo systemd-run --scope --unit=untrusted-demo \
  -p MemoryMax=256M -p MemoryHigh=200M -p MemorySwapMax=0 \
  -p CPUQuota=20% -p CPUWeight=50 -p TasksMax=64 \
  -- bash -c 'echo "running under a real limit"; sleep 2'

# ---- 3. Fork bomb, contained. Do NOT run this outside a cgroup. ----
sudo systemd-run --scope --unit=forkbomb-demo -p TasksMax=64 -p MemoryMax=128M \
  -- bash -c ':(){ :|:& };:' || true
CGPATH=$(systemctl show forkbomb-demo.scope -p ControlGroup --value)
cat "/sys/fs/cgroup${CGPATH}/pids.events"   # "max <n>" -> the clamp fired, host is fine
# Atomic, recursive, race-free cleanup -- no PID list to lose to a faster forker:
echo 1 | sudo tee "/sys/fs/cgroup${CGPATH}/cgroup.kill" >/dev/null

# ---- 4. OOM containment: the cgroup dies, the machine does not ----
sudo systemd-run --scope -p MemoryMax=128M -p MemorySwapMax=0 \
  -- python3 -c 'x = bytearray(512 * 1024 * 1024)'    # -> Killed
sudo journalctl -k -n 20 | grep -i 'memory cgroup out of memory' || true

# ---- 5. The throttling footgun nobody checks until p99 explodes ----
grep -E 'nr_throttled|throttled_usec' "$CG/cpu.stat"
# Climbing nr_throttled = your threads are frozen to the end of every 100ms
# period. Mean utilisation looks healthy. Tail latency does not.

# Cleanup
sudo rmdir "$CG" 2>/dev/null || true

The honest part: cgroups are accounting, not a security boundary

Everything above constrains consumption. None of it constrains reach. A process with memory.max=64M, cpu.max=0.1 and pids.max=8 can still read every file its uid can open, hit your metadata service, scan your VPC, and — given a kernel bug — become root on the host. You have made it a very frugal attacker. You have not made it a contained one.

cgroups answer "how much?". Namespaces answer "what can I see?". Capabilities and LSMs answer "what am I allowed to do?". seccomp answers "which syscalls even exist for me?". A container is all of them in a trench coat — and every single one still shares your kernel.

The kernel's documentation frames cgroups as a resource-management mechanism; the historical record is blunter. CVE-2022-0492 was a container escape that abused a cgroup v1 feature (release_agent) as the vector — cgroups have shown up in this genre as the hole more readily than the patch. What they genuinely buy you is the resource-exhaustion half of the threat model, and that half matters: a fork bomb or memory hog is the most likely thing an AI agent does to you by accident.

The other half is untouched. A privilege-escalation bug in the shared kernel doesn't care about your quotas, and with untrusted code that kernel is the whole exposure: every container on the box talks to the same few hundred syscalls, the same filesystem drivers, the same eBPF verifier. seccomp shrinks that surface, LSMs constrain what's reachable through it, capabilities strip the powerful operations — all worth doing — but it's still one kernel.

The isolation stack: what each layer actually stops

These are complementary layers, not alternatives — production systems use several at once. Behaviour varies by kernel version and distro config, so verify the specifics against the kernel's cgroup-v2 and seccomp documentation and your distro's LSM policy rather than trusting a table on the internet, including this one.

• Resource exhaustion (fork bomb, memory hog, CPU spin) — cgroups v2: precisely its job (memory.max, cpu.max, pids.max, io.max). Namespaces: nothing; a PID namespace hides processes, it doesn't cap them. seccomp: no. LSM: no. microVM: yes — vCPU/RAM are fixed at the VM boundary.
• Seeing other processes, mounts, users, networks — cgroups v2: nothing. Namespaces: their entire job (pid, mnt, net, user, uts, ipc). seccomp: only indirectly. LSM: yes, via policy. microVM: yes — a separate guest kernel means there's nothing else in the process table to see.
• Dangerous syscall surface — cgroups v2: no. Namespaces: no (user namespaces have historically widened it). seccomp: yes, this is the tool — a BPF filter over syscall numbers and arguments. LSM: partially. microVM: yes — guest syscalls hit the guest kernel; the host sees only virtio I/O.
• Fine-grained policy on files, sockets, capabilities — cgroups v2: no. Namespaces: coarse only. seccomp: no, it doesn't understand paths. LSM (SELinux/AppArmor): this is the tool. microVM: not directly — you still want an LSM inside the guest.
• Kernel exploit / privilege escalation to host — cgroups v2: no, and cgroup v1 has itself been an escape vector. Namespaces: no. seccomp and LSM: shrink the reachable code, don't eliminate the risk. microVM: this is the point — an escape must break the VMM and KVM, a far smaller and better-audited surface than the syscall interface.
• What it costs you — cgroups v2: free, already in your kernel. Namespaces: free. seccomp: cheap, but a bad filter breaks your runtime confusingly. LSM: policy authoring is real work. microVM: a few MB of guest memory and a boot — snapshot-restore puts PandaStack create at p50 179ms (p99 ~203ms) instead of a ~3s cold boot.

What changes when the boundary is a microVM

Move the same workload into a Firecracker microVM and the questions change shape rather than getting better answers. The guest gets a fixed vCPU and RAM allocation enforced by the hypervisor — not by a limit it shares a kernel with. There's no "the cgroup limit was misconfigured and it ate the host" failure mode, because the guest cannot address memory the VMM never gave it. On PandaStack that allocation is baked into the template snapshot: a property of the machine, not a tunable the tenant can argue with.

The guest's OOM killer becomes the guest's own problem. If code inside the VM allocates until the guest kernel starts killing things, that drama happens inside a machine that's about to be deleted. Same for the fork bomb: it exhausts a PID space and task-struct allocator belonging to one guest kernel nobody else shares. You might still set pids.max inside the guest — a clean non-zero exit is nicer to debug than a guest-wide OOM — but that's ergonomics now, not safety.

Crucially, cgroups don't disappear here — they move down a layer and keep doing their real job. Firecracker's jailer wraps each VMM process in exactly the defences above: a cgroup bounding what that process consumes on the host, plus a chroot, namespaces, dropped capabilities and a seccomp filter over the VMM's own syscalls. It's cgroups AND a hardware boundary, layered — the cgroup keeps one VM's device emulation from starving its neighbours, the VM keeps one tenant's kernel bug from becoming everyone's. The mistake was never using cgroups; it was expecting them to be the outermost wall.

In practice that looks like this — the untrusted code goes in a VM, and cgroups inside the guest are defence in depth rather than the plan:

from pandastack import Sandbox

# cgroups bound how much of the host a process can EAT.
# The VM boundary bounds what it can REACH. Model-written code wants both,
# and only one of them survives a kernel bug.


def run_untrusted(code: str) -> str:
    """Execute LLM-generated code with a hypervisor between it and prod."""
    # vCPU/RAM are fixed at the VM boundary -- the guest cannot renegotiate
    # them, and nothing inside can reclaim a page from a neighbour.
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=300,
        metadata={"origin": "llm", "trust": "none"},
    ) as sbx:
        sbx.filesystem.write("/work/main.py", code.encode())

        # Defence in depth, not the boundary: cgroups INSIDE the guest turn a
        # runaway into a clean non-zero exit instead of a guest-wide OOM.
        res = sbx.exec(
            "systemd-run --scope -q -p MemoryMax=512M -p TasksMax=128 "
            "-p CPUQuota=100% -- python3 /work/main.py",
            timeout_seconds=60,
        )

        if res.exit_code != 0:
            # Fork bomb, OOM, or an enthusiastic 'rm -rf /' -- all three end
            # here, inside a VM that was going to be deleted in 300s anyway.
            return f"failed ({res.exit_code}): {res.stderr[-400:]}"
        return res.stdout
    # VM destroyed on block exit. The kernel it corrupted was its own.


def explore_variants(setup: str, candidates: list[str]) -> list[str]:
    """One prepared state, N risky variants, each in its own kernel."""
    results: list[str] = []
    with Sandbox.create(template="code-interpreter", ttl_seconds=900) as base:
        base.filesystem.write("/work/setup.py", setup.encode())
        base.exec("python3 /work/setup.py", timeout_seconds=180)
        base.snapshot()   # freeze the prepared state once

        for cand in candidates:
            # Copy-on-write clone: a same-host fork lands in 400-750ms, so a
            # variant that trashes its machine costs you a machine you were
            # throwing away. No cleanup, no shared state to corrupt.
            with base.fork() as child:
                child.filesystem.write("/work/try.py", cand.encode())
                r = child.exec("python3 /work/try.py", timeout_seconds=60)
                results.append(r.stdout if r.exit_code == 0 else r.stderr)
    return results

Networking follows the same logic: each sandbox gets its own netns and tap device rather than a shared bridge, with 16,384 /30 subnets pre-allocated per agent. Namespaces doing the namespace job, underneath a VM boundary that doesn't depend on them being configured correctly.

When cgroups really are enough

Don't reach for a microVM out of reflex. If the code is yours — your services, CI steps, batch jobs, internal tooling — the threat model is "a bug eats the box," not "an adversary owns the box," and cgroups v2 plus a container is exactly the right amount of machinery. It's free, already installed, one systemd directive per unit; a hypervisor buys you a second kernel to patch in exchange for defending against a threat you don't have. Set MemoryMax, set TasksMax, alert on nr_throttled, go home.

The line to watch is whether the boundary is load-bearing for security or just for capacity. The moment the code is untrusted, adversarial or model-written at runtime — and a compromise would cross tenants rather than annoy one — you're asking cgroups to enforce something they never claimed to, and the shared kernel underneath is the real answer. The old objection to "just use a VM" was cost and boot time; snapshot-restore retired that, so the decision is now a threat-model call, not a latency one. Use cgroups for how much. Use a kernel boundary for what.