Side-channel attacks in multi-tenant compute, explained

Every isolation boundary you depend on is enforced in what CPU architects call the architectural world: page tables saying which physical memory a process may name, privilege rings saying which instructions it may execute, VM exits saying when a guest must hand control back to the host. That world is specified, documented, and mostly correct. The problem is that it is not the whole machine. Underneath it sits a larger, undocumented, performance-driven one: caches at three levels, TLBs, branch predictors, store buffers, line-fill buffers, load ports, prefetchers. That machine is shared by every thread on the chip — shared precisely because sharing it is what makes the chip fast — and it does not know or care where your security boundaries are. A side channel is what happens when someone reads the shadow your work casts on that shared state, usually by timing their own memory accesses and inferring yours.

I'm Ajay; I built PandaStack, where every sandbox, managed database, and hosted app is its own Firecracker microVM. This post is about the class of attack a VM boundary does not fully solve — an awkward thing for someone who sells VM boundaries to write about, but overclaiming here is worse than the alternative. If you take one thing from it, take this: a microVM removes an enormous, practically-exploited class of shared-fate failures, and it does not make the L3 cache stop being shared. Both halves are true, and anyone who tells you only the first half is selling you something.

There are two computers in your CPU

The architectural computer is the one in the instruction set manual: registers, addresses, and a strict contract about what each instruction does. Isolation lives here. A process cannot name memory outside its page tables. A ring-3 instruction cannot execute a ring-0 operation. A guest cannot touch host physical memory, because a second layer of address translation it does not control (EPT / nested paging) remaps everything it can express. If this were the whole machine, multi-tenancy would be a solved problem and this post would be one paragraph long.

The microarchitectural computer is everything the chip does to make that contract fast: keeping recently-touched lines in cache; remembering which way each branch went so it can run ahead speculatively; holding stores in a buffer so the pipeline doesn't stall; caching page-table walks in the TLB. None of that appears in the architectural contract, which is exactly why it can be optimized aggressively — and almost none of it is partitioned per-tenant, because partitioning it would throw away most of the benefit. Two containers on one host share the L3. Two VMs on one host share the L3. Two hyperthreads on one physical core share considerably more: the L1, the TLBs, the store and fill buffers, the execution ports.

The leak is always a timing difference. The attacker never reads your memory; the attacker reads a clock. If a load that should have taken hundreds of cycles returns in tens, the line was already cached and somebody put it there. If a load that should have hit takes hundreds, somebody evicted it. Repeat across thousands of cache sets, run the statistics, and you can reconstruct which addresses a co-resident workload touched — which, for a table-driven cipher or a secret-dependent branch, can mean reconstructing the secret.

A cache is a machine for remembering what somebody just did. Security wants a machine that forgets. We built the fast one and have been apologizing ever since.

The families that actually matter

The literature is enormous and the naming is chaotic. For an operator, four families cover almost everything, plus a fifth category that is really just "the ones published after this post."

Cache timing: Prime+Probe and Flush+Reload

The oldest and most general family, and the one needing no CPU bug at all — it exploits the cache working as designed. In Prime+Probe the attacker fills a set of cache lines with its own data, lets the victim run, then re-measures its access times: lines that got slow were evicted, which says the victim touched an address mapping to that set. It needs no memory shared with the victim, which makes it the relevant one for cross-VM work — an inclusive shared L3 is enough, and it works across physical cores. Flush+Reload is sharper but needs a shared page (a deduplicated page, a shared library): flush a line, wait, then time reloading it — fast means the victim brought it back. Both are noisy, and both get much harder when the victim runs briefly and the attacker can't control when the interesting work happens.

Transient execution: Spectre and Meltdown

Here speculation is the bug and the cache is merely the wire. Spectre-class attacks train a predictor so a victim, running with its own privileges, speculatively executes an access it would never architecturally perform — reading past a bounds check (v1), or jumping to an attacker-chosen gadget because the branch target buffer was poisoned (v2). The speculative work is rolled back architecturally; the cache footprint it left behind is not, and that footprint is read out with the cache timing above. Meltdown-class attacks are the more shocking cousin: on affected parts a faulting load could transiently return data from a privileged address before the fault landed, letting a user process read kernel memory outright. Meltdown was largely fixed in silicon on later parts and mitigated by KPTI on the rest; Spectre is structural to speculation and gets mitigated variant by variant, indefinitely.

MDS: sampling out of the shared buffers

Microarchitectural Data Sampling — the RIDL, Fallout, and ZombieLoad family, plus TSX Asynchronous Abort — is different in character and worse for multi-tenancy. You don't choose an address; you sample whatever happens to be flowing through a shared internal buffer: line-fill buffers, the store buffer, load ports. It's a firehose you filter rather than a pointer you dereference — unreliable for a targeted secret, but it crosses boundaries the architectural world thinks are solid, including the VM boundary, because those buffers are per-physical-core structures shared by both hyperthreads. This is the concrete reason "just turn off SMT" appears in every serious hardening guide. Two tenants on the two threads of one core share far more machine than either believes.

L1TF, and the ongoing drip

L1 Terminal Fault (Foreshadow) is the one written for the virtualization case: on affected parts a guest could craft page-table entries causing a terminal fault whose transient path exposed whatever sat in the L1 data cache — including lines belonging to the host or another guest. Mitigation is as blunt as it sounds: invert PTE bits so a guest can't aim the fault, flush L1D on every VM entry, and again, don't let two tenants share a core. Beyond these four families the drip continues — Retbleed, Downfall, Inception, branch history injection, whatever lands next quarter. Treat any list of variants, including this one, as a snapshot with a short shelf life.

What the mitigations are, and what they cost

The defenses come in three layers, all of which cost something. The first is microcode: vendors ship updates adding instructions and MSRs (buffer-clearing behaviour, indirect-branch controls) that the kernel then uses. Microcode without a kernel that knows to use it does approximately nothing, which is why "we're patched" has to mean both. The second is kernel mitigations: KPTI unmaps most kernel memory during user execution to blunt Meltdown, costing a TLB flush at every syscall boundary; retpolines and IBRS/eIBRS constrain indirect-branch speculation; IBPB and STIBP stop predictor state being inherited across a context switch or shared with a sibling thread; a VERW sequence clears the MDS-exposed buffers on transitions; L1D is flushed on VM entry where L1TF applies. The third is scheduling policy — the only layer that closes SMT sharing completely: turn SMT off, or use core scheduling so only threads from one trust domain share a physical core.

None of this is free. KPTI taxes syscall-heavy workloads. Indirect-branch mitigations tax anything with many indirect calls, which is to say most interpreted-language runtimes. L1D flush taxes VM-exit-heavy workloads. And SMT off is the big one: you visibly halve your logical CPU count and give up real throughput for it. All of which makes the kernel command line a tempting place to type something you will regret.

What each isolation boundary actually buys you

Line the options up honestly. Each row gets strictly better against the shared-software classes; only the last row gets better against the shared-silicon ones.

• Shared process — The worst case: same address space, same page tables, same everything. "Isolation" is a language runtime's promise, and an escape is a bug in a JIT rather than a kernel. Side channels barely matter because there is barely a boundary to leak across.
• Shared-kernel container — Better, but the tenant population is one syscall-reachable kernel away from shared fate: a kernel CVE is an everyone-on-the-box event. You also still share every cache, predictor, and buffer. Namespaces are a bookkeeping boundary the kernel agrees to honour, not a wall the hardware enforces.
• MicroVM — Structurally better: each tenant runs its own guest kernel, so a kernel bug stays in that guest, and the host is reached only through hardware-mediated VM exits into a tiny device model instead of hundreds of syscalls. The shared-kernel class is gone outright. The caches, TLBs, and predictors are the same silicon they always were.
• Separate physical host — The only complete answer, and why clouds sell dedicated hosts to people with regulators. No shared cache, no shared predictor, no co-residency to establish. You pay in utilization — precisely the trade multi-tenancy exists to avoid.

So be precise about what a microVM does. It eliminates the shared-kernel failure class — the one that actually shows up in incident reports — and raises the cost of everything else considerably. What it does not do is make a cross-core Prime+Probe against a shared L3 stop working, because that channel is a property of the cache hierarchy, not of the software running on top of it. Virtualization is a very large improvement, not an ending. The full ladder of options is in /blog/code-isolation-hierarchy.

Calibrating the threat, honestly

Now the part left out of the vendor deck. These attacks are slow, noisy, and demanding. They require co-residency — the same physical host, and for the SMT variants the same physical core — which on a fleet with any scheduling entropy is its own research problem. They require the victim to be doing something interesting with a secret while the attacker measures: secret-dependent table lookups, a key-dependent branch, a signature operation. And they require enough samples to beat the noise floor of a machine that is, by construction, busy doing other things.

The gap between a vulnerability's launch website and its exploit's data rate remains one of the great comedies of modern security. There is a logo. There is a domain name. There is a paper with a very good name. And then there is the actual measurement loop, which would like the victim to please keep performing the same secret-dependent operation continuously for a while, without rebooting, on the same core, in a quiet lab. Meanwhile, the thing that actually breached somebody last quarter was an API token in a build log.

That is not an argument for ignoring side channels; it's an argument for ordering your work correctly. For almost every sandbox operator the realistic loss event is a kernel CVE reached from a shared-kernel container, a credential left where untrusted code could read it, or an egress path that let a compromised guest send out what it found. Side channels are why you also keep microcode current and think hard about SMT — not why you rebuild your architecture. For the escape-path version of this calibration, /blog/vm-escape-attacks-explained covers where hypervisor escapes really come from.

Auditing what your host is actually doing

Linux exposes the kernel's own view of every known issue and its mitigation state in sysfs, one file per issue, and it is the fastest honest answer to "are we exposed?" for a given box. Run this on a host before you trust it with two tenants.

#!/usr/bin/env bash
# What is this host ACTUALLY doing about microarchitectural side channels?
# Read-only. Safe to run anywhere. Answers three questions:
#   1) is the microcode current, 2) what does the kernel think it mitigated,
#   3) did someone quietly turn it all off for a benchmark in 2023.
set -uo pipefail

echo "== cpu =="
lscpu | grep -E 'Model name|Vendor|Thread\(s\) per core|Core\(s\) per socket|Socket'

echo
echo "== microcode revision (kernel + patch level must BOTH be current) =="
grep -m1 '^microcode' /proc/cpuinfo || echo "microcode: not reported (arm64, or a VM guest)"

echo
echo "== kernel mitigation state, one line per known issue =="
for f in /sys/devices/system/cpu/vulnerabilities/*; do
  echo "  $(basename "$f"): $(cat "$f")"
done

echo
echo "== SMT: the single biggest lever for hostile multi-tenancy =="
cat /sys/devices/system/cpu/smt/control 2>/dev/null || echo "smt/control: absent (no SMT on this part)"
cat /sys/devices/system/cpu/smt/active  2>/dev/null || true

echo
echo "== anything still reporting plain 'Vulnerable'? =="
grep -H . /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null \
  | grep -i 'vulnerable' \
  || echo "  none -- every issue reports a mitigation or 'Not affected'"

echo
echo "== did someone disable mitigations on the kernel cmdline? =="
grep -o 'mitigations=[a-z]*' /proc/cmdline \
  || echo "  no mitigations= override (good)"
grep -o 'nosmt' /proc/cmdline || true

You can run exactly that inside a sandbox to see what a tenant sees — which is a useful and slightly humbling exercise, because the guest's answer is not the host's answer.

from pandastack import Sandbox

# The audit script from above, as a string.
AUDIT = open("audit-mitigations.sh").read()

# A throwaway guest with its own kernel. 179ms p50 to be live, so this is
# a cheap thing to run on every host image you build.
sbx = Sandbox.create(
    template="base",
    ttl_seconds=300,
    metadata={"job": "cpu-mitigation-audit"},
)
try:
    sbx.filesystem.write("/work/audit.sh", AUDIT)
    r = sbx.exec("bash /work/audit.sh", timeout_seconds=60)
    print(r.stdout)

    # The guest kernel's opinion, captured for your image-build records.
    vulns = sbx.exec(
        "grep -H . /sys/devices/system/cpu/vulnerabilities/* | sed 's#.*/##'",
        timeout_seconds=30,
    ).stdout

    # IMPORTANT: this is the GUEST's view. It is a function of the CPU features
    # the hypervisor chose to expose plus the guest kernel's own mitigations.
    # It does NOT tell you whether the HOST flushes L1D on VM entry, whether
    # the HOST's microcode is current, or whether the HOST runs with SMT on.
    # Those three facts live on the host and must be audited on the host.
    for line in vulns.splitlines():
        if "Vulnerable" in line:
            print("guest reports unmitigated:", line)
finally:
    sbx.kill()

Practical guidance for a sandbox operator

1. Keep microcode and host kernels current, as one unit. Microcode without a kernel that uses the new controls buys little, and vice versa. Bake the audit above into your image pipeline so a stale host fails the build rather than quietly joining the fleet.
2. Do not disable mitigations for a benchmark. If you must, do it on a machine that cannot reach production and delete the flag afterwards. The realistic failure mode isn't a dramatic decision — it's a temporary flag that became permanent because the person who typed it changed teams.
3. For genuinely hostile multi-tenancy, turn SMT off or use core scheduling. This is the one control that closes the entire sibling-thread family — MDS, L1TF's worst case, port contention — instead of papering over variants one at a time. It costs real throughput and is still the right default when the tenants sharing a core would happily hurt each other.
4. Keep sandboxes short-lived. Side channels are statistical: they need measurement time and stable co-residency. A workload that exists for ninety seconds is a far worse target than a VM up for eleven months. Short lifetimes are a mitigation, not just cost control.
5. Deny egress by default. A leaked bit is only a breach if it can leave. Default-deny outbound with an explicit allowlist, blocking link-local metadata and RFC1918 ranges, means an attacker who does extract something has nowhere to send it and no lateral path to try instead.
6. Don't put secrets on a host a sandbox can be co-resident with. No cloud API keys, signing keys, or customer database credentials on machines running untrusted guests. This one survives every future variant, because it removes the target instead of hardening the channel.

Four of those six are cheap. The expensive one — SMT off — is exactly the one that stops being optional as your tenants get less trustworthy. That's the shape of the trade: side-channel hardening is mostly hygiene, with one genuine throughput decision at the centre of it.

Where PandaStack fits, and what it doesn't claim

PandaStack runs every sandbox, managed database, and hosted app as its own Firecracker microVM: a KVM guest with its own kernel, a minimal virtio device model, a seccomp-filtered VMM, and a per-sandbox network namespace out of a pool of 16,384 pre-allocated /30 subnets. That removes the shared-kernel class outright — one tenant's code never executes against another tenant's kernel, and a guest kernel CVE is a single-guest event, not a fleet event. It also makes egress policy per-sandbox rather than fleet-wide, which is what makes rule five above implementable instead of aspirational.

The lifetime argument is the underrated one. Because every create is a snapshot restore rather than a boot — roughly a 49ms restore step inside about 179ms p50 end to end, with copy-on-write memory shared until first write — a sandbox is cheap enough that keeping it alive is the unusual choice. Run the job, take the output, kill the VM. An attacker needing stable co-residency and a long measurement window, against a target that exists for one task on a host they did not choose, is having a much worse day than one attacking a long-lived container. That mitigation fell out of the performance work rather than a security feature, which is my favourite kind.

What PandaStack does not claim is side-channel immunity, because nobody running multi-tenant compute on shared silicon honestly can. The L3 is shared. The predictors are shared. If your threat model includes a well-resourced adversary willing to work for cross-core cache inference against your specific secret, the answer is dedicated hardware, and always has been. A microVM removes the failure class that actually breaks people and makes everything else far more expensive and noisier. Good trade — just not a magic one.

To go deeper: /blog/vm-escape-attacks-explained covers where hypervisor escapes really come from and why device emulation is the classic culprit; /blog/code-isolation-hierarchy lays out the full ladder from language sandbox to separate host; and /blog/microvm-cpu-pinning-noisy-neighbor takes on scheduling and core placement — the same physical sharing seen through a performance lens instead of a security one.