Chaos Engineering Inside microVMs: Fault Injection Without the Blast Radius

You want to know what your service does when the database is 200ms further away than usual, when 5% of packets vanish, when the disk fills up mid-write, when the clock jumps ninety minutes forward and every JWT in flight becomes invalid, or when PID 1 gets a SIGKILL with no shutdown hook. These are not exotic scenarios. They are Tuesday. The reason most teams don't test them is not that the faults are hard to produce — `tc`, cgroups, `iptables`, and `kill -9` are right there — it's that producing them for real requires kernel-level knobs, and the machine whose kernel you'd be turning those knobs on is a machine other people are using.

So the experiment gets watered down. You add a fault-injection library that makes your HTTP client pretend to time out. That's a fine unit test and a terrible chaos experiment, because it only exercises the failure paths you already remembered to write. The kernel-level version finds the ones you didn't: the connection pool that leaks sockets under RST storms, the retry loop with no jitter that turns one blip into a thundering herd, the log writer that swallows ENOSPC and then silently drops your audit trail.

I'm Ajay — I build PandaStack, which runs Firecracker microVMs as a service, so I have an obvious bias. The argument here is about the isolation boundary rather than about my product, and it applies equally if you run Firecracker or Cloud Hypervisor yourself. I'll flag the places where a microVM genuinely doesn't help at the end, because a chaos post that claims to solve everything is itself a reliability hazard.

What a real chaos experiment actually needs from the kernel

Write down the faults you actually care about and then notice, for each one, which subsystem produces it. Almost none of them live in your process.

• Latency, jitter, packet loss, reordering, duplication, bandwidth caps — the `netem` and `tbf` queueing disciplines in the kernel's traffic control layer, attached to a network interface.
• A dependency going dark versus a dependency refusing loudly — netfilter rules that DROP (your timeout fires) or REJECT with a TCP reset (your retry fires). These are very different bugs and most services handle exactly one of them well.
• Memory pressure and OOM kills — cgroup v2 `memory.max` and `memory.high`, plus the kernel OOM killer and PSI stall counters. A mocked allocator failure is not the same thing as the kernel picking a victim.
• CPU starvation and IO throttling — `cpu.max` quota and `io.max` on a real block device, so you see actual scheduler throttling in `cpu.stat`, not a sleep().
• Disk full and inode exhaustion — a filesystem that returns a real ENOSPC to a real `write(2)`, including the fun case where you're out of inodes but `df -h` says you have gigabytes free.
• Clock skew — the wall clock jumping forward or backward while `CLOCK_MONOTONIC` calmly does not, which is precisely the discrepancy that breaks token expiry, cache TTLs, and leader leases.
• Ungraceful termination — `kill -9` on PID 1, a sysrq-triggered reboot with no sync, a deliberate kernel panic. No shutdown hook, no flush, no goodbye.

Every single one of those is a property of a kernel, a network namespace, a block device, or a clock. Which is the whole problem with running them anywhere shared.

The shared-kernel problem, stated plainly

A container is a set of namespaces and cgroups over the host's one kernel. That's a great deployment boundary and a mediocre chaos boundary, because the interesting faults are exactly the ones that namespaces don't virtualize. There is one clock. There is one `dmesg`. There is one block device under your overlay. Container-scoped chaos tooling works around this with real ingenuity — entering the target's network namespace to apply `tc` there, using per-container cgroups for pressure — and for network and CPU/memory faults that genuinely does contain the blast radius. But the containment is a function of how carefully the tool scopes things, not a function of the architecture, and the escape hatches are wide.

Set the clock in a container and you have set it for the host and every other container on it, unless the tooling goes out of its way to intercept time calls per-process. Fill the disk and you've filled the shared filesystem. Trigger the OOM killer badly and the kernel may pick a victim you did not nominate. Load a kernel module or poke a debugfs fault injector and it is global by definition. And the classic: run `tc qdisc add` on the wrong interface of a shared CI host, and you will meet all of your colleagues at once, in a single incident channel, within about ninety seconds.

What a microVM changes

A microVM is not a heavier container. It's a different boundary: hardware virtualization, with its own guest kernel booted inside it. On PandaStack each sandbox is a Firecracker microVM with its own kernel, its own network namespace and tap device on the host side, its own root block device (copy-on-write, reflinked from a template), and its own wall clock. That means the list above stops being a list of hazards and starts being a list of features:

• `tc qdisc add dev eth0 root netem ...` affects one guest's virtio-net interface. There is no shared bridge to accidentally shape.
• `iptables -A OUTPUT ... -j DROP` is written into the guest's own netfilter tables. Nobody else's connections traverse them.
• cgroup v2 pressure is applied inside a guest hierarchy whose parent is a kernel only that guest runs. If you make the OOM killer angry, it has exactly one machine's worth of processes to choose from.
• Filling a filesystem fills a copy-on-write disk image that is destroyed with the VM. `df` telling you 100% is the truth, and the truth costs you nothing.
• `date -s "+90 minutes"` moves that guest's wall clock. The host's clock, and every other tenant's, is unaffected — the boundary is the hypervisor, not a `LD_PRELOAD` shim you hope nothing bypasses.
• `kill -9 1` panics that guest's kernel and the VM exits. On a Firecracker guest booted with `panic=1 reboot=k` this is a fast, clean, total death — which is the point of the experiment.
• Kernel-level fault injectors (failslab, `fail_make_request`, and friends under debugfs) are per-guest rather than global — assuming your guest kernel is built with `CONFIG_FAULT_INJECTION`, which stock minimal kernels usually are not. Check the config you're actually booting before planning an experiment around it.

The test for whether your chaos boundary is real: could you run the experiment, unsupervised, on the same host as a colleague's experiment, and be confident neither of you notices the other? Containers make that an operational promise. A hypervisor makes it a structural one.

Network faults: netem, tbf, and the DROP-versus-REJECT distinction

Here's the network half of a real experiment. Every command runs inside the guest. Nothing in this file touches a host, which is the entire argument of this post compressed into a shebang. You'll need `iproute2` and the `sch_netem` module available in the guest, which most general-purpose images have.

#!/usr/bin/env bash
# Runs INSIDE the microVM guest, against the guest's own eth0.
set -euo pipefail
IFACE=eth0

# --- 1. Latency + jitter + loss (egress) ------------------------------
# 200ms mean, 50ms jitter, normally distributed, plus 5% packet loss.
tc qdisc add dev "$IFACE" root netem delay 200ms 50ms distribution normal loss 5%
tc qdisc show dev "$IFACE"   # ALWAYS verify -- a wrong qdisc is a silent no-op

# Mutate a running experiment in place with `change` (no teardown):
tc qdisc change dev "$IFACE" root netem delay 100ms loss 1% duplicate 1%

# Reordering REQUIRES a delay to reorder against. `reorder` alone does nothing.
tc qdisc change dev "$IFACE" root netem delay 50ms reorder 25% 50%

# Corruption: flip a random bit in 0.5% of packets. Great for checksum paths.
tc qdisc change dev "$IFACE" root netem delay 20ms corrupt 0.5%

# --- 2. Bandwidth ceiling ---------------------------------------------
# netem shapes delay/loss; tbf is the honest way to cap throughput.
tc qdisc del dev "$IFACE" root
tc qdisc add dev "$IFACE" root tbf rate 1mbit burst 32kbit latency 400ms

# --- 3. netem is EGRESS-only. For inbound, mirror to an ifb device. ----
tc qdisc del dev "$IFACE" root
modprobe ifb numifbs=1
ip link set ifb0 up
tc qdisc add dev "$IFACE" handle ffff: ingress
tc filter add dev "$IFACE" parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0
tc qdisc add dev ifb0 root netem delay 200ms loss 3%

# --- 4. Dependency failure: two very different bugs -------------------
# DROP: the socket hangs until YOUR timeout fires. Tests timeouts + budgets.
iptables -A OUTPUT -p tcp -d 10.100.0.20 --dport 5432 -j DROP
# REJECT+RST: instant "connection refused". Tests retries + circuit breakers.
iptables -A OUTPUT -p tcp -d 10.100.0.20 --dport 6379 -j REJECT --reject-with tcp-reset
# DNS goes dark. The outage nobody tests and everybody has had.
iptables -A OUTPUT -p udp --dport 53 -j DROP

# --- Teardown ---------------------------------------------------------
# (Or just delete the VM. Deleting the VM is the real teardown.)
tc qdisc del dev "$IFACE" root 2>/dev/null || true
tc qdisc del dev "$IFACE" ingress 2>/dev/null || true
iptables -F OUTPUT

Two things worth internalising. First, `netem` only shapes egress — the `ifb` redirect in step 3 is how you get inbound latency, and forgetting it is why half of "my netem experiment did nothing" reports exist. Second, DROP and REJECT test genuinely different code. A blackholed dependency exercises your timeout configuration and your request budget; a reset exercises your retry policy and your breaker. Services routinely handle one gracefully and fall over on the other, and you will not find out which until you inject both.

Resource pressure, disk full, clock skew, and hard kill

The other half. Note the pattern in the cgroup section: you don't just apply pressure, you read the kernel's receipts afterwards. `memory.events`, `cpu.stat`, and the PSI files in `*.pressure` are the difference between "the test passed" and "the test passed because the fault never actually fired."

#!/usr/bin/env bash
# Also runs INSIDE the guest. This guest's kernel, this guest's disk.
set -euo pipefail

# --- cgroup v2: hand the victim a much smaller machine -----------------
grep -q cgroup2 /proc/filesystems || { echo "need cgroup v2"; exit 1; }
echo "+memory +cpu +io" > /sys/fs/cgroup/cgroup.subtree_control
mkdir -p /sys/fs/cgroup/chaos

echo 256M > /sys/fs/cgroup/chaos/memory.max    # hard cap -> OOM kill past this
echo 200M > /sys/fs/cgroup/chaos/memory.high   # soft cap -> throttle + reclaim
echo "50000 100000" > /sys/fs/cgroup/chaos/cpu.max   # 50ms per 100ms = half a CPU

# io.max wants MAJ:MIN, and it differs per image -- never hardcode it.
ROOTDEV=$(lsblk -ndo MAJ:MIN /dev/vda)
echo "$ROOTDEV riops=100 wiops=50" > /sys/fs/cgroup/chaos/io.max

# Move this shell (and everything it spawns) into the cgroup, then start the app.
echo $$ > /sys/fs/cgroup/chaos/cgroup.procs
/opt/app/server &

# The receipts. Read these AFTER the run or your experiment proves nothing.
cat /sys/fs/cgroup/chaos/memory.events    # low high max oom oom_kill counters
cat /sys/fs/cgroup/chaos/memory.pressure  # PSI: some/full stall microseconds
cat /sys/fs/cgroup/chaos/cpu.stat         # nr_throttled, throttled_usec
dmesg | grep -i "killed process" || true  # the OOM killer's confession

# --- Real pressure, not a mocked allocator ----------------------------
stress-ng --vm 2 --vm-bytes 90% --timeout 60s --metrics-brief
stress-ng --cpu 4 --cpu-load 90 --timeout 60s
stress-ng --hdd 4 --hdd-bytes 1G --timeout 60s

# --- Disk full: a real filesystem returning a real ENOSPC -------------
fallocate -l 1G /var/chaos.img
mkfs.ext4 -q -N 1024 /var/chaos.img       # only 1024 inodes, on purpose
mkdir -p /mnt/tiny && mount -o loop /var/chaos.img /mnt/tiny
fallocate -l 1000M /mnt/tiny/ballast      # writes under /mnt/tiny now fail
df -h /mnt/tiny && df -i /mnt/tiny        # note: you can be full on EITHER

# --- Flaky block device: EIO on a schedule (device-mapper) ------------
# "up 8s, down 4s" -- during each down window, IO fails with EIO.
SZ=$(blockdev --getsz /dev/vdb)
dmsetup create flaky --table "0 $SZ flakey /dev/vdb 0 8 4"
mount /dev/mapper/flaky /mnt/flaky
# dm-delay is the gentler sibling: add 200ms to every read.
# dmsetup create slow --table "0 $SZ delay /dev/vdb 0 200"

# --- Clock skew: this guest owns its own wall clock -------------------
timedatectl set-ntp false 2>/dev/null || systemctl stop chrony 2>/dev/null || true
date -s "+90 minutes"   # expire JWTs, blow past cert notAfter, break TOTP
date -s "-2 days"       # replay windows, cache TTLs, cron catch-up storms
# CLOCK_MONOTONIC does NOT jump. That divergence is the bug you're hunting.

# --- Ungraceful death. Only funny inside a microVM. -------------------
echo 1 > /proc/sys/kernel/sysrq
kill -9 1                     # PID 1 dies; with panic=1 the kernel panics
# echo b > /proc/sysrq-trigger  # instant reboot: no sync, no unmount, no mercy
# echo c > /proc/sysrq-trigger  # deliberate panic, for crash-handling paths

Four ways to inject a fault, and what each one actually proves

These are complements more often than rivals — the question is what each one can prove. Descriptions of third-party tools here are qualitative and change fast; verify current capabilities against their own docs before you plan an experiment around them.

• Application-layer fault injection library (in-process hooks, or a proxy like toxiproxy) — What it proves: your code takes the error branch it was written to take, deterministically, in a unit test, in milliseconds. What it can't: anything below your own abstractions — no kernel socket behaviour, no OOM killer, no ENOSPC from a real write, no clock skew. You are testing the failure modes you already thought of.
• Container-native chaos platforms (Chaos Mesh, Litmus, Gremlin and similar) — What they give you: genuinely kernel-level network and cgroup faults applied per-pod, plus scheduling, observability, and blast-radius controls that are much better than a shell script. What stays awkward: the host kernel is still shared, so faults that aren't namespaced — the clock, kernel modules, global fault injectors, the shared block device — are either emulated, restricted, or carry real spillover risk. Check the current docs for exactly which fault types they scope and how.
• MicroVM per experiment (this post) — What you get: a private kernel, private netfilter and tc state, a private block device, and a private clock, so the nastiest faults are both real and structurally contained; plus snapshot-restore for an identical start state and fork for branching one experiment several ways. What you give up: it's one machine, so multi-node consensus and partition semantics need several guests wired together, and the guest kernel is the one the image ships, not necessarily your production kernel.
• A real staging cluster or a production game day — What it proves: the actual system, actual topology, actual data volumes, actual on-call humans and runbooks. Nothing else substitutes for it. What it costs: slow, expensive, hard to make reproducible, and every failed experiment contaminates the next one until somebody rebuilds the environment. This is where you go after the cheap experiments stop finding anything.

Snapshot-restore: the same known-good machine every single run

The unglamorous reason chaos programmes die is not that the faults are hard. It's that experiment number six runs on the wreckage of experiment number five. You filled the disk, you left a `tc` qdisc attached, you set the clock two days back and forgot, you leaked eleven thousand sockets in TIME_WAIT. Now a result is either a genuine finding or residue from the last run, and nobody can tell which, so everyone quietly stops trusting the suite.

Snapshot-restore fixes this by construction. You bake a snapshot of a known-good, fully warmed machine once — dependencies installed, service running, caches primed — and every experiment restores that exact memory and disk state. Not "a freshly provisioned equivalent machine". The same machine, byte for byte, every time. On PandaStack that restore is how a sandbox is created at all: p50 179ms and p99 around 203ms, with the restore step itself about 49ms. A first-ever cold boot before any snapshot exists is roughly 3 seconds, and after that you never pay it again.

Cleanup stops being a script you have to get right, too. There is no `finally` block that unwinds your netem rules and unmounts your loop device and resets your clock. You delete the VM. The disk image, the qdiscs, the netfilter rules, the poisoned clock and the corpse of PID 1 all cease to exist together.

from pandastack import Sandbox

NETEM_LOSS = """
set -euo pipefail
tc qdisc add dev eth0 root netem delay 200ms 50ms distribution normal loss 5%
tc qdisc show dev eth0
"""

DISK_FULL = """
set -euo pipefail
fallocate -l 256M /var/chaos.img
mkfs.ext4 -q /var/chaos.img
mkdir -p /var/lib/app/data && mount -o loop /var/chaos.img /var/lib/app/data
fallocate -l 250M /var/lib/app/data/ballast
df -h /var/lib/app/data
"""

CLOCK_SKEW = """
set -euo pipefail
timedatectl set-ntp false 2>/dev/null || true
date -s "+90 minutes"
date -u
"""


def run_experiment(name: str, fault_script: str) -> dict:
    """One experiment = one microVM = one kernel = one blast radius."""
    with Sandbox.create(
        template="base",
        ttl_seconds=600,                      # backstop: no orphaned experiments
        metadata={"experiment": name, "suite": "chaos"},
    ) as sbx:
        # 1. Known-good start state.
        setup = sbx.exec("bash /srv/app/setup.sh", timeout_seconds=180)
        assert setup.exit_code == 0, setup.stderr

        # 2. Steady-state hypothesis BEFORE the fault. If this fails, the
        #    experiment is void -- you were broken before you broke anything.
        before = sbx.exec("bash /srv/app/probe.sh", timeout_seconds=120)
        assert before.exit_code == 0, f"not steady pre-fault: {before.stderr}"

        # 3. Inject. A failed injection that looks like a passed experiment
        #    is the single most common way a chaos suite lies to you.
        sbx.filesystem.write("/tmp/fault.sh", fault_script)
        fault = sbx.exec("bash /tmp/fault.sh", timeout_seconds=60)
        if fault.exit_code != 0:
            raise RuntimeError(f"injection failed, result is void: {fault.stderr}")

        # 4. Re-probe under fault and collect the kernel's receipts.
        after = sbx.exec("bash /srv/app/probe.sh", timeout_seconds=300)
        evidence = sbx.exec(
            "tc -s qdisc show dev eth0; cat /sys/fs/cgroup/chaos/memory.events 2>/dev/null; dmesg | tail -40",
            timeout_seconds=30,
        )
        return {
            "experiment": name,
            "survived": after.exit_code == 0,
            "stdout": after.stdout,
            "stderr": after.stderr,
            "kernel_evidence": evidence.stdout,
        }
    # VM destroyed here. So are the qdiscs, the full disk, and the wrong clock.


for experiment_name, script in [
    ("netem-loss-5pct", NETEM_LOSS),
    ("disk-full-enospc", DISK_FULL),
    ("clock-skew-90m", CLOCK_SKEW),
]:
    print(run_experiment(experiment_name, script))

Fork: branch the experiment at the interesting moment

Here's the capability that has no container analogue, and it maps onto chaos engineering almost too neatly. Get one machine to the exact moment that matters — the migration half-applied, the cache warm, the queue backed up, the leader elected — then fork it several ways and inject a different fault into each child, from bit-identical state.

That's the difference between "packet loss broke it" and "packet loss broke it and disk-full didn't, from the same starting state, so the difference is real and not setup noise." Copy-on-write memory and a reflinked rootfs mean each child shares pages with the parent until it writes, so branching five ways is cheap in both time and RAM. A same-host fork lands in 400 to 750ms on PandaStack; cross-host is 1.2 to 3.5 seconds because the memory and disk have to travel.

from pandastack import Sandbox

# Drive ONE machine to the interesting moment, once.
base = Sandbox.create(template="base", persistent=True, ttl_seconds=3600)
warm = base.exec(
    "bash /srv/app/setup.sh && bash /srv/app/warm-caches.sh && bash /srv/app/fill-queue.sh",
    timeout_seconds=600,
)
assert warm.exit_code == 0, warm.stderr

FAULTS = {
    "netem-loss":  "tc qdisc add dev eth0 root netem delay 200ms loss 5%",
    "db-blackhole": "iptables -A OUTPUT -p tcp --dport 5432 -j DROP",
    "db-reset":     "iptables -A OUTPUT -p tcp --dport 5432 -j REJECT --reject-with tcp-reset",
    "dns-dark":     "iptables -A OUTPUT -p udp --dport 53 -j DROP",
    "mem-squeeze":  "mkdir -p /sys/fs/cgroup/chaos && echo 128M > /sys/fs/cgroup/chaos/memory.max",
}

results = {}
for name, cmd in FAULTS.items():
    # Each child starts from the parent's exact memory + disk. Not "a similar
    # machine" -- the same machine, branched. Same-host fork: 400-750ms.
    child = base.fork()
    try:
        child.filesystem.write("/tmp/fault.sh", f"set -euo pipefail\n{cmd}\n")
        injected = child.exec("bash /tmp/fault.sh", timeout_seconds=60)
        if injected.exit_code != 0:
            results[name] = {"void": True, "stderr": injected.stderr}
            continue

        probe = child.exec("bash /srv/app/drain-queue.sh", timeout_seconds=300)
        results[name] = {
            "survived": probe.exit_code == 0,
            "exit_code": probe.exit_code,
            "stdout": probe.stdout[-4000:],
            "stderr": probe.stderr[-4000:],
        }
    finally:
        child.kill()   # the fault dies with the VM; nothing to unwind

base.kill()

for name, r in sorted(results.items()):
    print(f"{name:14} survived={r.get('survived')} void={r.get('void', False)}")

Because each experiment is one VM rather than one shared environment, running them in parallel is a loop rather than a scheduling problem. Every sandbox gets its own pre-allocated network namespace and tap device — a PandaStack agent keeps 16,384 pre-allocated /30 subnets, so networking setup isn't the bottleneck; memory and CPU are. Your chaos suite starts to look like your unit test suite: run it all, in parallel, on every merge, and let the interesting failures come to you.

Designing an experiment that can actually fail

The substrate is the easy part. The discipline is what makes the results mean anything, and it's the same discipline whether you run this on microVMs or a cluster.

1. State a steady-state hypothesis in numbers you can measure from outside the system — p99 latency, success rate, queue drain time. "It stays up" is not a hypothesis, it's a hope.
2. Verify steady state BEFORE injecting. If the probe fails pre-fault, the run is void. Half of all confusing chaos results are environments that were already broken.
3. Verify that the fault actually fired. Read `tc -s qdisc show`, `memory.events`, `cpu.stat`, `dmesg`. An experiment where the injection silently failed looks exactly like a system that survived, and is far more dangerous than a failed test.
4. Change one variable per experiment. This is what fork-from-identical-state buys you: the difference between two runs is the fault, not the setup.
5. Bound the run. A TTL on the sandbox and a timeout on every exec, so a hung experiment reaps itself instead of quietly billing you until Monday.
6. Write down what you expected before you look. Chaos engineering's real product is the gap between your mental model and the machine, and you cannot measure that gap after you've already seen the answer.

Where a microVM genuinely doesn't help

If I only listed wins you'd be right not to trust the rest. Some real limits.

• One guest is one node. Split-brain, quorum loss, and asymmetric partitions need several guests and a network you can partition between them. That's buildable — a VM per node, netem and iptables between them — but it's a topology you assemble, not a thing you get for free.
• The guest kernel is the image's kernel, not necessarily your production kernel. Version-specific behaviour — a particular OOM heuristic, a scheduler change, a filesystem quirk — may not reproduce. If the bug you're chasing is kernel-version-specific, match the version or the experiment proves nothing.
• Device emulation is virtio, not your production NIC or NVMe. Faults that live in a specific driver, a specific offload path, or specific hardware behaviour won't show up.
• It doesn't test your operational response. Half the value of a game day is finding out that the runbook is stale, the alert routes to someone who left, and the dashboard nobody opened is wrong. No sandbox tells you that.
• Managed dependencies are still managed. If you want a real managed Postgres in the loop, that's another provisioning step with its own latency — on PandaStack, database creation runs 30 to 90 seconds, so plan for it rather than putting it inside a tight experiment loop.

The way to read all of that: microVM chaos is where the cheap, repeatable, genuinely kernel-level experiments live. It's the layer between "my mock returned an error" and "we broke production on purpose at 10am with everyone watching." That middle layer barely exists on most teams, which is why the same class of failure keeps reaching customers — the faults are too real for a unit test and too disruptive for a shared environment, so nobody tests them at all.

Give each experiment its own kernel and the calculus flips. `tc netem` stops being a career-limiting command and becomes a line in a test file. Filling a disk stops needing a maintenance window. `kill -9 1` becomes something you do forty times before lunch, in parallel, from identical state, and the only thing that dies is a VM that was always going to.