Spectre, Meltdown, and What a microVM Actually Protects You From

Every fast CPU built in the last thirty years works by guessing. It guesses which way a branch will go, it guesses that a load will hit, it starts executing down the guessed path, and when the guess turns out wrong it quietly throws the architectural results away and pretends nothing happened. That last clause is the interesting one. It throws away the architectural results — the registers, the flags, the memory writes. It does not throw away the cache lines it warmed, the buffers it filled, or the predictor entries it trained on the way. The CPU guessed, it guessed about someone else's memory, and it left the receipt in a drawer that anybody on the machine can time.

I'm Ajay; I build PandaStack, where every sandbox, managed database and hosted app runs as its own Firecracker microVM on a KVM host. So read this as opinionated — and read the thesis carefully, because it is the uncomfortable one for someone selling microVMs: hardware virtualization is a genuinely strong boundary against software attacks, and it is not, by itself, a mitigation for transient-execution leaks. KVM plus a guest kernel does not make Spectre go away. What makes Spectre go away is microcode, kernel mitigations left switched on, and a tenancy policy that decides who is allowed to be co-resident with whom.

This post stays in that lane. If you want the wider taxonomy of side channels from first principles — cache timing, Prime+Probe, the whole family tree — /blog/side-channel-attacks-multi-tenant-compute-explained is the survey. If you want the SMT decision argued properly, /blog/smt-hyperthreading-side-channel-microvm-isolation does that at length and I am deliberately not re-litigating it here. What follows is the transient-execution lane specifically: what the families are, why the VM boundary is the wrong layer to fix them at, how to read what your host is actually doing, and what the fix costs you.

What "transient execution" actually means

Every attack in this family has exactly two halves, and keeping them separate is the difference between understanding the problem and repeating a headline. The first half is the transient window: some sequence of instructions runs speculatively, touches data it is not architecturally allowed to touch, and is then squashed. Nothing in your program's visible state changes. The second half is the covert channel: the squashed instructions left a measurable side effect — most commonly a cache line that is now warm, sometimes a port that was busy, sometimes a TLB entry — and a second piece of code times that effect to reconstruct the secret one bit at a time.

The reason this class refuses to die is that the first half is not a bug in the usual sense. It is the microarchitecture doing exactly what it was built to do, at the speed it was built to do it. The permission check and the speculative load are not serialized with respect to each other, because serializing them would cost most of the performance of the last two decades. So the mitigations are all, in one way or another, retrofits: put a fence here, flush a predictor there, clear a buffer on the way out, stop the two threads on this core from sharing that structure. It is a tax, and it is levied on the machine you already bought.

The families, and which boundary each one crosses

• Spectre v1 (bounds check bypass) — the victim's own code speculatively runs past an array bounds check that it would never architecturally pass. Crosses whatever boundary the victim code sits on, including into the kernel via a syscall handler. Mitigated in software, per gadget, forever.
• Spectre v2 (branch target injection) — the attacker poisons shared indirect-branch prediction state so the victim speculates into a gadget of the attacker's choosing. This is the one that most directly crosses privilege and VM boundaries, because the branch predictors historically did not care who trained them.
• Meltdown (rogue data cache load) — a faulting load transiently returns data from a privileged address before the fault is delivered, letting an unprivileged process read kernel memory outright. Fixed in silicon on later parts, mitigated with page-table isolation on the rest.
• Spectre v4 (speculative store bypass) — a load speculatively bypasses an older store to the same address, reading stale data. Mostly a same-address-space concern: it matters when a sandbox inside the guest is meant to contain code, which is exactly what a language-level sandbox is.
• L1TF / Foreshadow — written for the virtualization case. On affected parts a guest could craft page-table entries whose terminal fault exposed whatever happened to be sitting in the L1 data cache, including host or other-guest lines. Mitigation is PTE inversion plus flushing L1D on VM entry.
• MDS — RIDL, Fallout, ZombieLoad, and TSX Asynchronous Abort. You do not pick an address; you sample whatever is flowing through shared internal buffers (line-fill buffers, the store buffer, load ports). Unreliable for a targeted secret, but it crosses boundaries the architectural world thinks are solid, including the VM boundary.
• SRBDS and MMIO stale data — the same buffer-sampling shape aimed at special-register reads and at stale data left in MMIO-adjacent structures. Both are mitigated by microcode plus the kernel's buffer-clearing sequence, and both are boring right up until the microcode is missing.
• Retbleed, Inception / SRSO — return instructions and the return stack buffer turn out to be indirect branches wearing a hat, and can be trained the same way. Mitigations here are the ones that make kernel maintainers visibly tired: return thunks, RSB stuffing, safe-RET sequences.
• Downfall / Gather Data Sampling, and register-file data sampling — the newer arrivals, leaking from the gather instruction's internal buffers and from the register file respectively. Both are microcode-mitigated. Both are proof that this list is a snapshot with a short shelf life, not a finished inventory.

Notice what almost every entry has in common: the leaking structure is per-core or per-socket silicon that predates the idea that two mutually hostile tenants might share it. The branch predictor was not built with a tenant field. The line-fill buffer does not consult a VMID. Adding that tagging after the fact is precisely what the microcode updates do — which is why microcode, not your choice of hypervisor, is the thing doing the mitigating.

Why KVM is a strong boundary and still not a mitigation

Be precise about what KVM enforces, because the strength is real and its shape explains the gap exactly. A guest runs under a second layer of address translation it does not control — EPT on Intel, NPT on AMD — so there is no guest instruction that names host physical memory. Firecracker narrows what remains: a device model small enough to read in an afternoon, a seccomp filter on the VMM process, a jailer that drops it into its own namespaces and cgroup. That combination genuinely removes the shared-kernel failure class. One tenant's code never executes against another tenant's kernel; a guest kernel CVE is a one-guest event rather than a fleet event. /blog/firecracker-vmm-security-model has the detail.

Now look at what every one of those mechanisms operates on: addresses, instructions, and privilege transitions. None of them operates on microarchitectural state. There is no bit in an EPT entry that says how warm a cache line may be. There is no VMCS control that stops the branch target buffer from remembering what the last guest taught it. The isolation is built one storey above the floor where the sharing happens, and no amount of configuration moves it down.

A hypervisor is a very good lock on a very good door. Speculation is the building's ventilation system, and it was installed by a different contractor who was optimising for airflow.

There is one place where the VMM does become part of the story, and it is easy to get backwards. Firecracker CPU templates mask CPUID so a snapshot taken on one host restores on another — necessary, and covered in /blog/firecracker-cpu-templates-explained and /blog/firecracker-cpuid-masking-explained. But CPUID is also how a guest kernel learns which mitigation controls exist. Mask away the architectural-capabilities bits and you can produce a guest that cheerfully reports "Not affected" because nobody told it the hardware had opinions. The guest is not lying; it is answering a question about a CPU it was shown rather than the CPU it is on.

Read your host's actual exposure

Linux publishes the kernel's own verdict on every issue it knows about, one file per issue. It is the fastest honest answer to "is this host safe to put two tenants on?", it is read-only, and it takes one command. Run it on the host, not in a guest.

# The whole picture, one command. Read-only, safe anywhere.
grep . /sys/devices/system/cpu/vulnerabilities/*

# A recent, well-patched server part. Yours will differ -- and the file list
# itself grows every time a new family gets a name, a paper and a logo:
#   gather_data_sampling:Not affected
#   itlb_multihit:Not affected
#   l1tf:Not affected
#   mds:Not affected
#   meltdown:Not affected
#   mmio_stale_data:Not affected
#   reg_file_data_sampling:Not affected
#   retbleed:Not affected
#   spec_rstack_overflow:Not affected
#   spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl
#   spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
#   spectre_v2:Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; RSB filling; PBRSB-eIBRS: SW sequence; BHI: BHI_DIS_S
#   srbds:Not affected
#   tsx_async_abort:Not affected

# The same command on a four-year-old host. This is what "mitigated in
# software, at a price" looks like -- note the suffixes:
#   l1tf:Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable
#   mds:Mitigation: Clear CPU buffers; SMT vulnerable
#   meltdown:Mitigation: PTI
#   mmio_stale_data:Mitigation: Clear CPU buffers; SMT vulnerable
#   retbleed:Mitigation: IBRS
#   spectre_v2:Mitigation: IBRS; IBPB: conditional; STIBP: conditional; RSB filling
#   tsx_async_abort:Mitigation: Clear CPU buffers; SMT vulnerable

# And the line you never want to see. The kernel knows the fix and the
# microcode never turned up to help:
#   mmio_stale_data:Vulnerable: Clear CPU buffers attempted, no microcode

# The gate for a rollout script. Catches plain "Vulnerable" AND the
# "SMT vulnerable" suffix, which is deliberate -- both are findings.
grep -i vulnerable /sys/devices/system/cpu/vulnerabilities/* || echo "clean"

# How many issues does this kernel even know about? An old kernel on a new
# CPU has FEWER files, not better news.
ls /sys/devices/system/cpu/vulnerabilities/ | wc -l
uname -r

How to read those strings without fooling yourself

• "Not affected" — this kernel believes this silicon does not have this issue. It is a claim by the kernel about the CPU, and its accuracy depends on the kernel being new enough to have heard of both the issue and the part. A 2019 kernel on a 2024 CPU produces a lot of confident, worthless "Not affected".
• "Mitigation: <something>" — the issue applies and the kernel is doing work about it. This is the good outcome and it is also where your throughput is going. The named mechanism tells you which tax you are paying: PTI is page-table isolation, IBRS/eIBRS and IBPB are indirect-branch controls, "Clear CPU buffers" is the VERW buffer-overwrite sequence, "PTE Inversion" plus "conditional cache flushes" is the L1TF pair.
• "Vulnerable" on its own — nothing is being done. Either the mitigation was disabled on the command line, or the kernel is too old to have one. Both are your problem and both are fixable today.
• "Vulnerable: ... no microcode" — the worst of the honest states. The kernel is ready and the CPU's patch level is not. This is a package-and-reboot problem, not a kernel problem, and it is the single most common finding on hosts that were imaged once and never rebuilt.
• "; SMT vulnerable" as a suffix — the mitigation works against everything except a sibling hardware thread on the same core. The kernel is telling you that the remaining hole is a tenancy decision, not a patching one. That decision is its own post: /blog/smt-hyperthreading-side-channel-microvm-isolation.
• "KVM: Mitigation: ..." prefixes, seen on itlb_multihit and inside the l1tf string — these are specifically about the host-to-guest direction. If you run VMs, these are the lines that concern your tenants rather than your own processes.

Microcode is half of every mitigation

Nearly every mitigation in the modern set is a joint effort: the vendor ships microcode that adds an MSR or changes a buffer's behaviour, and the kernel ships code that knows to use it. Either one alone does approximately nothing. "We're fully patched" from someone who has only run apt upgrade on the kernel is a sentence with a hole in the middle of it.

# 1) What revision is running right now -- and do ALL cores agree?
#    More than one line here means a late load went partly wrong.
awk '/^microcode/ {print $3}' /proc/cpuinfo | sort -u
# 0x2b000603

cat /sys/devices/system/cpu/cpu0/microcode/version   # same answer, no parsing

# 2) Was it loaded EARLY, from the initramfs, before the kernel finished
#    bringing up features? Early is the only load you should ship.
dmesg | grep -i microcode
# microcode: updated early: 0x2b000181 -> 0x2b000603, date = 2024-05-15
# microcode: Microcode Update Driver: v2.2.

# 3) Is the vendor package installed, and did it make it into the initramfs?
dpkg -l intel-microcode amd64-microcode 2>/dev/null | tail -n +6   # Debian/Ubuntu
rpm -q microcode_ctl linux-firmware 2>/dev/null                    # RHEL family
lsinitramfs /boot/initrd.img-$(uname -r) | grep -c kernel/x86/microcode

# 4) Host-side KVM knobs that only exist because of this family:
cat /sys/module/kvm_intel/parameters/vmentry_l1d_flush   # never | cond | always
cat /sys/module/kvm/parameters/nx_huge_pages             # Y | N | auto

Late loading — pushing new microcode into a running CPU through the reload interface — exists, and every time I have watched somebody reach for it, it was because a reboot was politically expensive. It has genuine hazards: features the kernel already probed at boot do not get re-probed, and a partially applied load leaves you with cores at different revisions, which is the one line in the check above nobody thinks to look at. Take the reboot. On a platform where a sandbox is created by restoring a snapshot in well under a second, draining a host is a scheduling problem rather than an outage.

The kernel command line, and the line you should never type

cat /proc/cmdline
# BOOT_IMAGE=/boot/vmlinuz-6.8.0-40-generic root=UUID=... ro mitigations=auto,nosmt

# The umbrella switch, in the three settings that matter:
#   mitigations=auto        the default: every mitigation the kernel deems
#                           necessary for THIS cpu, SMT left alone
#   mitigations=auto,nosmt  the same, plus SMT off. The paranoid default for
#                           genuinely hostile multi-tenancy.
#   mitigations=off         see below. Please don't.

# Per-issue overrides, for when "auto" is not the policy you want:
#   spectre_v2=on               force the full indirect-branch stack
#   spectre_v2_user=on          IBPB + STIBP for user-to-user, unconditionally
#   spec_store_bypass_disable=on
#   l1tf=full,force             flush L1D unconditionally on VM entry
#   mds=full,nosmt              clear buffers, and refuse to run with SMT
#   tsx_async_abort=full,nosmt
#   tsx=off                     removes TAA by removing the feature
#   gather_data_sampling=force
#   kvm-intel.vmentry_l1d_flush=always

# The one line to never type on a host that runs other people's code:
#   mitigations=off
# It works. That is the problem.

What the mitigations actually cost

I am not going to give you a percentage, and you should be suspicious of anyone who does without naming their workload, their kernel, their CPU stepping and their SMT setting. The reason is structural rather than coy: these mitigations do not tax computation. They tax transitions and indirect branches. So the cost of your fleet's mitigations is a function of how often your code crosses a boundary, which is a property of your workload and nobody else's.

• KPTI (page-table isolation) — unmaps most of the kernel while userspace runs, so every syscall and interrupt switches page tables. The bill arrives per transition; PCID/ASID support on the CPU makes it much cheaper but not free. A process doing millions of tiny reads pays enormously; one doing a long matrix multiply barely notices.
• Retpolines / IBRS / eIBRS — constrain indirect-branch speculation. Retpolines make every indirect call slower in software; enhanced IBRS moves it into hardware and is cheaper. The bill lands on indirect-call-heavy code, which means interpreters, dynamic dispatch and anything with a vtable in its hot loop. Your Python workload is more exposed to this cost than your compiled one.
• IBPB on context switch — flushes predictor state so the next process starts cold. Excellent isolation, and it means the branch predictor gets no warm start at all. The bill lands on high context-switch-rate workloads, which is exactly what a busy host running many small tasks is.
• "Clear CPU buffers" (the VERW sequence) — a fixed cost paid on returns to userspace and on VM transitions. Small per event, and events are numerous. Again: transitions, not cycles.
• L1D flush on VM entry — throws away the L1 data cache on the way into the guest, so the guest starts from a cold L1 after every exit. This is the one that specifically bills virtualization: a guest with a chatty virtio device that exits frequently pays far more than one that sits in a compute loop.
• SMT off — the blunt instrument. You visibly halve your logical CPU count and give up real throughput. It is also the only control that closes the entire sibling-thread family at once instead of chasing variants. Argued properly in the SMT post; not re-argued here.

So the honest instruction is: measure it on your own workload. Not a microbenchmark, and specifically not one chosen after you already knew which answer you wanted. Boot the same image twice on the same host SKU with only the kernel command line differing, run the thing you actually ship, and record the mitigation state alongside the numbers so the result is still interpretable in six months.

from pandastack import Sandbox

# Two shapes of load, because the mitigation tax lands on transitions and
# indirect branches, not on arithmetic. Run BOTH or you will conclude
# whatever your first benchmark happened to be shaped like.
BENCH = open("bench.sh").read()

sbx = Sandbox.create(
    template="base",
    ttl_seconds=900,
    metadata={"job": "mitigation-cost", "cmdline": "mitigations=auto,nosmt"},
)
try:
    sbx.filesystem.write("/work/bench.sh", BENCH)

    # Transition-heavy: small reads, many syscalls, lots of context switches.
    # This is where KPTI, IBPB and the buffer-clear sequence actually bill you.
    print(sbx.exec("bash /work/bench.sh syscall", timeout_seconds=600).stdout)

    # Compute-bound: the same silicon, almost none of the same tax.
    print(sbx.exec("bash /work/bench.sh compute", timeout_seconds=600).stdout)

    # Record what the GUEST believed it was running on, next to the numbers.
    # This is not the host's answer -- see the caveat below -- but a benchmark
    # without its mitigation state attached is a number without units.
    print(sbx.exec("grep . /sys/devices/system/cpu/vulnerabilities/*").stdout)
finally:
    sbx.kill()

Co-residency is the precondition, so tenancy is a control

Every attack in this family needs the attacker's code executing on the same physical machine as the victim's, and several of them need the same physical core. That is not a footnote; it is the load-bearing assumption. Which means your scheduler is a security component, and "who gets placed next to whom" is a policy question that most platforms answer by accident, via whatever the bin-packer felt like doing at three in the morning.

1. Separate trust tiers onto separate hosts, not separate namespaces. If you have a paid tier of known customers and a free tier of anonymous signups, those are two different populations of risk and putting them on the same silicon is a decision, whether or not anyone made it deliberately.
2. Keep untrusted workloads short-lived. These attacks are statistical: they need stable co-residency plus enough samples to beat the noise floor of a busy machine. A workload that exists for ninety seconds and then evaporates is a much worse target than a VM that has been up since the last kernel CVE.
3. Make placement unpredictable to the tenant. If an attacker can reliably land on the same host as a chosen victim by creating sandboxes in a loop, co-residency stops being a precondition and becomes an API call. Spreading creates across the fleet is a security property as well as a capacity one.
4. Do not keep secrets on hosts that run untrusted guests. No cloud API keys, no signing keys, no customer database credentials on machines executing other people's code. This is the only item on the list that survives every future variant unchanged, because it removes the target instead of hardening the channel.
5. Make the SMT decision explicitly, once, and write down why. On or off, core scheduling or not — the failure mode is not choosing wrong, it is never choosing at all and discovering the default during an audit.

Boundaries, lined up against transient execution specifically

Here is the part that makes vendor comparison charts awkward. Against software escapes these four options are a clear ladder, each rung meaningfully better than the last. Against transient-execution leaks the ladder is almost flat, and only the last rung is a different thing entirely. Characterisations of other people's systems below are qualitative on purpose — verify anything load-bearing against their current documentation rather than my summary of it.

• Shared-kernel container — Against software escapes: the weakest of the four. One syscall-reachable kernel bug is an everyone-on-the-box event, and namespaces are a bookkeeping boundary the kernel agrees to honour rather than a wall the hardware enforces. Against transient execution: no help whatsoever, and slightly worse than the others in practice because the shared kernel is itself a large, attractive Spectre v1 gadget surface reachable directly from tenant code.
• gVisor or a comparable userspace kernel — Against software escapes: a real improvement, because most syscalls are serviced by a userspace kernel instead of the host's, shrinking the host kernel surface a tenant can reach. Against transient execution: unchanged. The interception happens in software; the branch predictors, buffers and caches underneath are identical silicon, and the extra transitions are themselves places mitigations get billed.
• MicroVM on KVM (Firecracker et al.) — Against software escapes: structurally better again. Each tenant gets its own guest kernel, the host is reached only through hardware-mediated VM exits into a tiny device model, and the shared-kernel failure class is gone outright. Against transient execution: better in one narrow, real way — L1TF-class and MDS-class exposure between guests is what the host's L1D flush and buffer clearing specifically address, and those only apply because there is a VM boundary to flush at. Otherwise: same caches, same predictors, same story.
• Separate physical host — Against software escapes: complete, trivially. Against transient execution: the only row that genuinely changes the answer, because it removes co-residency, which is the precondition for all of it. You pay in utilization — which is precisely the cost multi-tenancy exists to avoid, which is why this is sold as a premium and not as a default.

Read that list twice, because the shape of it is the whole argument. Three of the four rows differ enormously in one column and barely at all in the other. If a vendor's isolation pitch does not distinguish those two columns, they are either not thinking about it or hoping you won't. The full software-escape ladder is in /blog/code-isolation-hierarchy, and /blog/vm-escape-attacks-explained covers where hypervisor escapes really come from.

What PandaStack does here, and what it doesn't

What we do is the boring, honest half. Every sandbox, managed database and hosted app is its own Firecracker microVM: a KVM guest with its own kernel, a minimal virtio device model, a seccomp-filtered VMM, and a per-sandbox network namespace out of a pool of 16,384 pre-allocated /30 subnets. That removes the shared-kernel class outright and makes egress policy per-sandbox rather than fleet-wide, which is what makes "a leaked bit is only a breach if it can leave" implementable instead of aspirational. Host-side, the mitigation state is a property of the machine image, the audit above runs against it, and mitigations stay on.

The lifetime argument is the underrated one, and it fell out of performance work rather than a security roadmap. Because every create is a snapshot restore rather than a boot — roughly 179ms at p50, about 203ms at p99, against roughly three seconds for a genuine cold boot — a sandbox is cheap enough that keeping one alive is the unusual choice. Run the job, take the output, destroy the VM. An attacker who needs stable co-residency and a long measurement window, against a target that exists for one task on a host they did not choose, is having a substantially worse day than one attacking a container that has been up since spring.

The honest asterisks. Fork defaults to placing the child on the parent's host, because that is what makes a same-host fork land in 400–750ms instead of the 1.2–3.5s a cross-host fork costs — a performance decision with a co-residency consequence, benign here only because a fork tree is one tenant's own trust domain. Guest kernels are 5.10 on Ubuntu 24.04 userspace, which is a deliberate boot-time choice with its own tradeoffs (/blog/firecracker-guest-kernel-5-10-vs-6-1-explained argues both sides). And the big one: PandaStack does not claim immunity to microarchitectural leakage, because nobody running multi-tenant compute on shared silicon honestly can. The L3 is shared. The predictors are shared. If your threat model includes a well-resourced adversary willing to work for cross-core inference against your specific secret, the answer is dedicated hardware — and since PandaStack self-hosts on any Linux KVM host, the version of that answer we can actually offer is "run it on machines only you are on", not a checkbox.

A microVM removes the failure class that actually appears in incident reports and makes everything else considerably more expensive and noisier. That is a good trade. It is not a magic one, and the difference matters most to exactly the customers who ask the best questions. /blog/how-to-vet-a-code-execution-vendor-security is the list of those questions, including the ones I would rather not be asked.

The operator checklist

1. Run the sysfs sweep on every host image before it joins the fleet, and fail the build on anything reporting plain "Vulnerable" or "no microcode". Compare the file count across hosts too — fewer files means an older kernel, not fewer problems.
2. Ship microcode and kernel together, loaded early from the initramfs, and verify all cores report the same revision. Treat late loading as an emergency measure with known hazards, not a routine.
3. Fingerprint the mitigation state per host — microcode revision, SMT state, the full sysfs sweep — hash it, store it, and alert on change. Hosts come back from hardware maintenance different from how they went in, and a diff notices that faster than a person does.
4. Grep /proc/cmdline for mitigations= on every host, every day. The flag you are looking for was typed by somebody who no longer works there.
5. Measure the mitigation cost on your real workload, with both a transition-heavy and a compute-heavy shape, and record the mitigation state next to the numbers.
6. Decide tenancy deliberately: trust tiers on separate hosts, short lifetimes for untrusted work, unpredictable placement, and no secrets on hosts running other people's code.
7. Make the SMT call, write down the reasoning, and revisit it when the tenant population changes rather than when an auditor asks.

None of that is exciting, and that is rather the point. The transient-execution families have excellent branding and terrible data rates; the mitigations have no branding at all and cost you real throughput every single day. The tax is not optional, though, because the alternative is a machine that is very fast at guessing about other people's memory. Pay it, measure it, and spend the remaining worry on the credential someone left in a build log — which is, still, what actually breached somebody last quarter.