Hyper-Threading and microVM Isolation: the SMT Decision

There is a line in every capacity spreadsheet that nobody argues about, because nobody notices it. You take the fleet's logical CPU count, divide by the vCPUs you sell per sandbox, apply an overcommit ratio somebody chose in a hurry, and that is your density. The arithmetic is fine. The first number is the problem: on almost every x86 host you will ever rent, roughly half of those logical CPUs are not cores. They are the second hardware thread of a core you already counted — and that second thread is not a private machine. It is a second queue into a kitchen that already has one.

That sharing is the entire performance argument for simultaneous multithreading, and it is also, unmodified, the entire security problem with it. I'm Ajay; I build PandaStack, where every sandbox, managed database and hosted app is its own Firecracker microVM. This post is deliberately narrow: not a survey of side channels — /blog/side-channel-attacks-multi-tenant-compute-explained already does the taxonomy, and you should read that if you want the Spectre and MDS families explained from first principles — but the specific question of what two sibling threads share, why a VM boundary does nothing about it, and the four-rung operational decision at the end that costs real money whichever rung you pick.

A physical core, a logical CPU, and the thing in between

Start with the vocabulary, because the whole decision is downstream of getting it straight. A physical core is a complete execution engine: front end that fetches and decodes, a scheduler that issues micro-operations to a set of execution ports, the ports themselves (integer ALUs, load units, store units, vector units), an L1 data cache, an L1 instruction cache, an L2, TLBs, and the branch prediction machinery that keeps the whole thing fed. A logical CPU — what Linux calls a processor, what your provider almost certainly calls a vCPU — is a hardware thread: an architectural context that the core can run. With SMT enabled, one physical core presents two of them (four or eight on some non-x86 parts). Each thread gets its own register file, its own program counter, its own interrupt state, and its own line in the kernel's run queue. Neither gets its own execution resources, because there is only one set.

That is not a design flaw; it is the design. A single instruction stream stalls constantly — a cache miss here, a branch mispredict there, a dependency chain that leaves half the ports idle for a few cycles. Those bubbles are dead silicon. SMT fills them by keeping a second thread's instructions ready to issue whenever the first one is waiting. When your workload stalls a lot, this is close to free throughput. When your workload is a well-tuned vectorized kernel that already saturates the ports, the second thread has nothing to fill and mostly adds contention. This is why every honest statement about SMT's performance is workload-shaped and why the second half of this post insists you measure your own rather than borrowing a percentage from a blog post about someone else's compiler.

The map from logical CPUs to physical cores is not something you have to guess at. Linux publishes it, and it is the first thing to look at on any host you are about to make multi-tenant.

# Which logical CPUs are actually the same physical core?
# lscpu -e is the fastest read: equal CORE values mean sibling threads.
lscpu -e=CPU,NODE,SOCKET,CORE,ONLINE

# CPU NODE SOCKET CORE ONLINE
#   0    0      0    0 yes      <- core 0, first thread
#   1    0      0    0 yes      <- core 0, second thread  (sibling of cpu0)
#   2    0      0    1 yes      <- core 1, first thread
#   3    0      0    1 yes      <- core 1, second thread
#   ...

# The same answer straight from sysfs, per CPU. This is the file to script
# against, because it does not depend on lscpu's column formatting:
cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list   # e.g. "0,1"
cat /sys/devices/system/cpu/cpu0/topology/core_cpus_list         # newer alias
cat /sys/devices/system/cpu/cpu0/topology/core_id                # the core index

# Is SMT on at all, and will this kernel let you turn it off?
cat /sys/devices/system/cpu/smt/active    # 1 = sibling threads are online
cat /sys/devices/system/cpu/smt/control   # on | off | forceoff | notsupported

# The three numbers people confuse with each other, in one line:
lscpu | grep -E 'Thread\(s\) per core|Core\(s\) per socket|^CPU\(s\):'

# Build the sibling map -- these are the pairs you must never split across
# two trust domains, whichever rung of the ladder you end up on:
for c in /sys/devices/system/cpu/cpu[0-9]*; do
  echo "$(basename $c) -> $(cat $c/topology/thread_siblings_list 2>/dev/null)"
done | sort -u

A vCPU is not a core, and a guest vCPU is not even a thread

Two layers of the same confusion stack on a sandbox platform. At the bottom, when a cloud provider sells you an instance with sixteen vCPUs, that has conventionally meant sixteen hardware threads on eight physical cores — check your provider's current documentation, because the definition varies by instance family and some newer families sell cores outright. At the top, a Firecracker guest's vCPU is a host thread; it is not pinned to anything by default, and the host scheduler moves it around. /blog/firecracker-vcpu-scheduling-model has the threading model in detail and /blog/microvm-cpu-pinning-noisy-neighbor covers what that costs you in tail latency.

The security consequence of that second layer is the one people miss. If a guest vCPU is just a host thread and you have not told the scheduler otherwise, then which physical core it lands on — and therefore which stranger's thread is sitting on the other half of that core — is an emergent property of Linux's load balancer. It changes millisecond to millisecond. You are not choosing your co-tenant at the core level; you are letting the scheduler choose, and the scheduler is optimising for cache locality and run-queue balance, not for trust domains. That is the default, and defaults are policy whether or not anybody decided them.

What sibling threads share, and what they don't

The exact partitioning varies by microarchitecture and vendor, and some structures are statically split between threads rather than competitively shared. But the general shape has been stable for two decades and it is more than most people assume.

• Shared: the execution ports and the scheduler that issues to them. This is the whole point of SMT — one set of ALUs, load units, store units and vector units, fed by two instruction streams. Contention here is directly observable as latency by either thread.
• Shared: the L1 data and instruction caches, and on essentially every current part the L2 as well, since L2 is per-core. Two siblings evict each other's lines in the cache closest to the pipeline, where timing differences are largest and cleanest.
• Shared: the TLBs, including the second-level TLB. Address-translation pressure from one thread degrades the other, and the pattern of that degradation is information about which pages the other thread touched.
• Shared: branch prediction structures — direction predictors, the branch target buffer, and the return stack buffer on affected parts. This is the substrate the entire Spectre v2 mitigation stack (IBPB, STIBP, retpolines) exists to police, and STIBP exists specifically because of the sibling case.
• Shared: the store buffer, line-fill buffers and load ports — the internal plumbing that the MDS family samples out of. These are per-core structures, not per-thread ones.
• Shared: the core's power and frequency budget. A sibling running heavy vector code can pull the core's clock down, which is a performance problem and, if you squint, another timing channel.
• Not shared: the architectural register file, the program counter, the local APIC state. Each thread has a complete architectural context. This is exactly why the operating system sees two CPUs and why the illusion holds — the architectural world really is duplicated. The microarchitectural world underneath it is not.

Compare that list against what two separate cores share: the L3 and the memory controller, essentially. Cross-core cache attacks are real and they work through that shared inclusive L3, but the attacker is fighting through a large, noisy, socket-wide structure with every other workload on the machine adding interference. A sibling thread is not fighting through anything. It is inside the core, on the other side of the same L1, contending for the same ports, cycle by cycle. It is a different order of magnitude of access, and treating the two as one category — "caches are shared" — is how the SMT question gets waved away.

Two cores share a building. Two threads share a desk. The security guard at the front door has opinions about the first situation and no jurisdiction over the second.

Why a VM boundary is strong for memory and weak for timing

It is worth being precise about what KVM actually enforces, because the strength of that enforcement is real and the shape of it explains the gap exactly. A guest runs with a second layer of address translation it does not control — EPT on Intel, NPT on AMD — so every physical address the guest can express is remapped by tables only the host can write. There is no guest instruction that names host physical memory. Firecracker narrows the remaining surface further: a device model small enough to read in an afternoon, a seccomp filter on the VMM process, a jailer that drops it into its own namespaces and cgroup. That is a genuinely strong boundary, it is hardware-enforced, and it is the reason the shared-kernel failure class simply does not exist for microVM tenants.

Now notice what every one of those mechanisms operates on: addresses, instructions, and privilege transitions. Not one of them operates on time. There is no field in an EPT entry that says how many cycles a load may appear to take. There is no VMCS control that stops the sibling thread from noticing that the multiplier was busy last cycle. Port arbitration happens in hardware, per cycle, with no notion of a VMID, because the arbiter was designed to maximise instructions per clock and has never heard of your tenancy model. The isolation is built one layer above the layer where the sharing happens, and you cannot fix that with configuration — it is where the mechanism lives.

If anything, virtualization adds signal rather than removing it. A VM exit is a large, expensive, extremely timeable event. The guest kernel's behaviour is more regular than a general-purpose userspace process. And the co-tenant relationship on a sandbox platform is more stable than on a general compute fleet, because sandboxes are pinned to hosts by volume affinity, fork locality, and the plain fact that the scheduler put them there. None of this makes microVMs a bad choice — the alternative shares an entire kernel — but the honest claim is memory-shaped, and it should be stated that way.

The attacks that specifically need a sibling

Skip the taxonomy; the useful cut is not "which paper had the best logo" but "which of these stops working when the sibling thread goes away". Three shapes matter.

• L1TF / Foreshadow — the virtualization-flavoured one. On affected Intel parts a guest can craft page-table entries so that a terminal fault's transient path exposes whatever currently sits in the core's L1 data cache, including lines belonging to the host or another guest. The mitigation flushes L1D on VM entry, which is a point-in-time action. With SMT enabled that is not enough, and the reason is structural rather than subtle: your sibling keeps running while your VM runs, so it keeps refilling the L1 you just flushed. You cannot fix a continuous exposure with a discrete flush.
• The MDS family — RIDL, Fallout, ZombieLoad, TSX Asynchronous Abort, and later the MMIO stale data variants. These sample from the per-core buffers rather than reading a chosen address: line-fill buffers, the store buffer, load ports. The mitigation clears those buffers on transitions out of your context. Same structural problem as above — the sibling never transitions, it is simply running, and a co-resident thread is not a context switch you can hook.
• Port contention — the PortSmash shape, and the one that needs no CPU bug at all. You measure the latency of your own carefully chosen instructions and infer which execution ports your sibling is keeping busy, and therefore roughly what instruction mix it is executing. There is no data leak primitive, no speculative window, nothing to patch, because nothing is broken: you are observing the resource sharing that SMT is. Against a code path whose instruction mix depends on a secret — a classic square-and-multiply, a non-constant-time comparison — an instruction-mix trace is signal.

You do not have to take my word for any of this, because the kernel says it out loud, on your host, right now, in a file. This is the single most useful thing in this post.

#!/usr/bin/env bash
# What does THIS kernel say about sibling-thread exposure, right now?
# Entirely read-only. The phrase you are hunting for is "SMT vulnerable".
set -uo pipefail

echo "== per-issue mitigation state =="
for f in /sys/devices/system/cpu/vulnerabilities/*; do
  echo "  $(basename "$f"): $(cat "$f")"
done

# Typical output on a fully patched Intel host WITH SMT enabled:
#   l1tf:            Mitigation: PTE Inversion; VMX: conditional cache
#                    flushes, SMT vulnerable
#   mds:             Mitigation: Clear CPU buffers; SMT vulnerable
#   tsx_async_abort: Mitigation: Clear CPU buffers; SMT vulnerable
#   mmio_stale_data: Mitigation: Clear CPU buffers; SMT vulnerable
#   spectre_v2:      Mitigation: Retpolines; IBPB: conditional;
#                    STIBP: always-on; RSB filling; ...
#
# The SAME host after `echo off > /sys/devices/system/cpu/smt/control`:
#   l1tf:            Mitigation: PTE Inversion; VMX: cache flushes,
#                    SMT disabled
#   mds:             Mitigation: Clear CPU buffers; SMT disabled
#
# Read what changed. Nothing was patched. No microcode moved. The exposure
# clause disappeared because the sibling thread disappeared.
#
# Exact strings vary by CPU vendor, model and kernel version -- treat the
# above as illustrative and read your own host's files.

echo
echo "== the operational one-liner: is anything still sibling-exposed? =="
grep -l 'SMT vulnerable' /sys/devices/system/cpu/vulnerabilities/* 2>/dev/null \
  || echo "  nothing reports 'SMT vulnerable'"

echo
echo "== SMT state =="
echo "  active:  $(cat /sys/devices/system/cpu/smt/active 2>/dev/null || echo n/a)"
echo "  control: $(cat /sys/devices/system/cpu/smt/control 2>/dev/null || echo n/a)"

echo
echo "== and did someone disable mitigations for a benchmark in 2023? =="
grep -o 'mitigations=[a-z,]*' /proc/cmdline || echo "  no override on the cmdline"
grep -o 'nosmt[a-z=]*' /proc/cmdline || true

This is also why Firecracker's own production host setup guidance has long included disabling SMT on hosts running untrusted workloads. Not as an optimisation, not as belt-and-braces — as the thing that makes the mitigations above complete rather than partial. Check their current documentation for the exact wording and the accompanying host recommendations, which evolve with each new variant.

The honest threat model

Now the counterweight, because a post that only lists exposures is marketing with worse manners. These attacks are hard. They need co-residency on the same physical core, not merely the same host. They need sustained measurement time. They need the victim to be doing something repeatable and secret-dependent during that window. They need the attacker to distinguish signal from the noise floor of a machine that is, by construction, busy doing other things for other people. Published demonstrations are usually laboratory-shaped: known victim, known code path, quiet machine, a lot of patience. The gap between the paper and a working attack on a production fleet is enormous and it is not closing quickly.

And yet. Look at that list again from the perspective of a company whose product is renting out slices of a machine to strangers. Co-residency is not an obstacle the attacker must overcome; it is the thing being sold, for a few cents, with an API. The attacker controls one side of the sibling pair completely — they write the measuring workload, they choose when it spins, they retry as often as they like. Measurement time is not scarce either: a code sandbox that runs a build lives for minutes, and an AI agent sandbox chewing through a long task can live for an hour, which is a very different proposition from a function that returns in 200 milliseconds. The three conditions that make these attacks impractical in a corporate data centre are exactly the three conditions a public sandbox platform provides on demand.

The attacker also does not need to win reliably. They need to be able to buy attempts in bulk, which is what a per-second billing model with a free tier is, viewed unkindly. This is roughly the same economics that make credential stuffing work: a terrible success rate is fine when attempts are cheap and you can make a great many of them while you sleep.

There is a real limiting factor, though, and it is the one worth acting on first: something valuable has to be co-resident. If the hosts running untrusted tenants hold no credentials, no signing keys and no control-plane secrets, then the population of things a hostile sibling can sample is other tenants' workloads. That is still a genuine problem if you sell to anyone who has a compliance function, but it is a much smaller and much more bounded problem than the alternative, and getting secrets off tenant-facing hosts costs you nothing but a refactor. Do that regardless of which rung you pick below; it is the one mitigation that survives every future variant, because it removes the target rather than hardening the channel.

The ladder: four rungs, increasing in price

There are exactly four positions here. Everything else is a combination or a euphemism.

Rung 1: leave SMT on and accept the risk

This is defensible, and I want to say that plainly before the rest of the post makes it sound otherwise. If every tenant on the host is you — internal CI, your own build fleet, a single-tenant deployment for one customer on their own hardware — then the trust domain is the whole host and sibling sharing costs you nothing in isolation because there is nothing to isolate from. Likewise if the workloads genuinely have no cross-tenant secret worth a research project's worth of effort. Most compute in the world is in this category and disabling SMT on it would be a pure loss.

The failure mode of rung 1 is not that it is wrong; it is that it is usually undecided. "We left SMT on because our tenants are all internal and here is the boundary at which that stops being true" is a position with an owner and a trigger. "Nobody looked" is the same configuration with none of the credit, and it reads very differently in an incident review or a security questionnaire. If you are going to stay on rung 1 — and you might well be right to — write down why, with a date on it and a named condition that would move you.

Rung 2: turn SMT off

The blunt instrument, and the one every hardening guide reaches for, because it closes the entire sibling family at once — L1TF's worst case, all of MDS, port contention, cross-thread branch predictor games — instead of chasing variants individually as they are published. There is no residual gap to reason about, no configuration that can silently stop working, and the sysfs files change their minds on the spot to prove it happened.

# Runtime, reversible: offline every sibling thread right now.
echo off | sudo tee /sys/devices/system/cpu/smt/control
cat /sys/devices/system/cpu/smt/active     # -> 0
nproc                                      # -> half of what it was

# Runtime, one-way until reboot (a later `echo on` is refused):
#   echo forceoff | sudo tee /sys/devices/system/cpu/smt/control

# Boot-time, which is the version you actually want baked into the image:
#   nosmt                    -> siblings offlined at boot, re-enablable
#   nosmt=force              -> and refuse to bring them back online
#   mitigations=auto,nosmt   -> apply the default mitigations AND disable SMT
#
# GRUB_CMDLINE_LINUX="... mitigations=auto,nosmt"

# Make it an image-build gate rather than a wiki page nobody opens:
if [ "$(cat /sys/devices/system/cpu/smt/active)" != "0" ]; then
  echo "POLICY DRIFT: SMT is enabled on a tenant-facing host" >&2
  exit 1
fi
echo "SMT off, as policy requires"

# Firmware-level disable (BIOS/UEFI) removes the siblings from the topology
# entirely, so nothing in software can bring them back. Slower to change,
# which is either the point or the problem depending on your week.

The cost, stated as honestly as I can: your logical CPU count halves, visibly and immediately. What that does to your throughput is not something I or anyone else can tell you as a number, because it depends entirely on how much your workload stalls. A memory-latency-bound service and a saturated vector kernel will give you wildly different answers on the same silicon, and both answers will be real. Anyone quoting you a single percentage for "the cost of disabling SMT" measured it on code that is not yours. Measure it on code that is.

Rung 3: core scheduling

The interesting rung, and the least known. Core scheduling keeps SMT enabled but teaches the kernel scheduler that some tasks must never occupy sibling threads of the same physical core simultaneously. Tasks carry a cookie; two tasks may share a core only if their cookies match. When a core has one runnable task and nothing with a matching cookie to pair it with, the sibling is forced idle rather than being handed to a stranger. You keep SMT's benefit whenever two same-domain tasks are runnable together and you pay for it in forced idle when they are not.

The mechanics are a single prctl. You need a kernel built with CONFIG_SCHED_CORE — mainline since 5.14 — and the feature switches on the first time anything creates a cookie. Cookies are inherited across fork, which is the property that makes this practical for a VMM: stamp the cookie, then exec Firecracker, and every vCPU thread it spawns carries the cookie without further work.

// Core scheduling: keep SMT on, but refuse to co-schedule two trust domains
// on the sibling threads of one physical core. Requires a kernel built with
// CONFIG_SCHED_CORE (mainline since 5.14); the feature activates the first
// time a cookie is created.
//
//   grep -w CONFIG_SCHED_CORE /boot/config-$(uname -r)
package sched

import (
	"fmt"

	"golang.org/x/sys/unix"
)

const (
	prSchedCore = 62 // PR_SCHED_CORE

	schedCoreCreate    = 1 // PR_SCHED_CORE_CREATE
	schedCoreShareTo   = 2 // PR_SCHED_CORE_SHARE_TO   (push ours onto target)
	schedCoreShareFrom = 3 // PR_SCHED_CORE_SHARE_FROM (pull target's onto us)

	pidTypePid  = 0 // PIDTYPE_PID  -- that one thread
	pidTypeTgid = 1 // PIDTYPE_TGID -- the whole process, threads included
)

// NewCookie stamps a fresh core-scheduling cookie onto pid's whole process.
// Children inherit it across fork, so a Firecracker process started after
// this call carries the cookie into every vCPU thread for free.
func NewCookie(pid int) error {
	_, _, errno := unix.Syscall6(uintptr(unix.SYS_PRCTL), prSchedCore,
		schedCoreCreate, uintptr(pid), pidTypeTgid, 0, 0)
	if errno != 0 {
		return fmt.Errorf("PR_SCHED_CORE_CREATE(%d): %w", pid, errno)
	}
	return nil
}

// ShareTo copies OUR cookie onto pid -- the call you actually want on a
// sandbox host. Get the granularity right: one cookie per TENANT, pushed
// onto every Firecracker process that tenant owns. NOT one cookie per VM.
// Per-VM cookies are strictly safe and strictly wasteful: two microVMs
// belonging to the same customer would force each other idle for no
// isolation benefit whatsoever, and you would pay SMT-off prices while
// still carrying SMT-on risk everywhere you forgot to set a cookie.
func ShareTo(pid int) error {
	_, _, errno := unix.Syscall6(uintptr(unix.SYS_PRCTL), prSchedCore,
		schedCoreShareTo, uintptr(pid), pidTypeTgid, 0, 0)
	if errno != 0 {
		return fmt.Errorf("PR_SCHED_CORE_SHARE_TO(%d): %w", pid, errno)
	}
	return nil
}

Three honest caveats before you reach for it. First, forced idle is a real cost whose size depends on your tenant concurrency: a dense host with many tenants runnable at once pairs well and pays little, while a host where one bursty tenant is the only thing running pays close to SMT-off prices for SMT-on complexity. Second, the kernel's own documentation — Documentation/admin-guide/hw-vuln/core-scheduling.rst is the authoritative reference, and you should read it rather than this paragraph — is explicit that core scheduling is not a complete substitute for disabling SMT, with residual considerations around kernel-mode execution and interrupts running on a sibling. Third, and most operationally dangerous: a cookie you failed to set looks exactly like safety. There is no cheerful sysfs file that says "core scheduling is protecting you". If you build this, build the verification with it, and treat an unstamped VMM process as a page rather than a curiosity.

Rung 4: dedicated cores per tenant

The expensive answer that actually works. Allocate whole physical cores to a tenant with a cpuset, and — this is the entire trick — always allocate both siblings of a core into the same cpuset, never split one across two customers. Do that and SMT becomes free again inside the tenant's own boundary, where its threads were already allowed to see each other. The tenant gets the throughput; nobody gets a stranger on the other half of their core.

What changes operationally is your unit of accounting. Cores become the thing you allocate, place, and sell, and your bin-packer has to reason about pairs rather than a flat pool of interchangeable threads. You lose overcommit on those cores, you eat fragmentation when a tenant's demand does not divide neatly by two, and a dedicated core that its owner is not using is a core you are paying for and nobody is running on. This is precisely the trade every cloud makes when it sells dedicated hosts to customers with regulators, priced accordingly, and it is why that product is expensive. It is not a clever trick that gets you isolation for free. It is isolation, at cost, which is the only kind there is.

The four rungs, compared

• SMT on (the default) — Throughput: the highest available for stall-heavy mixed workloads; this is what SMT is for and on the right workload it is close to free. Isolation: sibling threads share L1, L2, TLBs, predictors, internal buffers and execution ports across trust domains, and your kernel says so in sysfs. Operational cost: zero, which is exactly why this configuration survives unexamined for years.
• SMT off — Throughput: you give up the SMT gain entirely and your logical CPU count halves; whether that costs a little or a lot depends wholly on how much your workload stalls, so measure rather than importing someone else's number. Isolation: closes the entire sibling family in one move, and the "SMT vulnerable" clauses disappear from sysfs on the spot. Operational cost: one boot flag, plus re-tuning every capacity number you ever computed from thread counts.
• Core scheduling — Throughput: keeps SMT's benefit whenever two same-cookie tasks are runnable together, pays forced idle when they are not, so the cost tracks your tenant concurrency rather than being fixed. Isolation: sibling threads only ever run one trust domain, with residual gaps the kernel documents rather than hides. Operational cost: the highest in engineering terms — cookie assignment, inheritance, granularity choices, and verification that is genuinely awkward to build.
• Dedicated cores per tenant — Throughput: SMT stays on and stays useful within a tenant's own boundary; what you lose is overcommit and clean bin-packing, not the SMT gain itself. Isolation: the strongest of the four short of separate physical hosts, because no core is ever split between two customers. Operational cost: the highest in money — cores become the unit of allocation, idle dedicated cores are yours to pay for, and you should price the tier accordingly. This is the answer that works, and its price is honest.

Measure the cost on your workload, not on someone else's

Every rung above rung 1 has a throughput cost, and you cannot make a competent decision about a cost you have not measured. Some discipline about how to measure it, because this is a benchmark that is easy to get wrong in a way that flatters whichever answer you already wanted.

1. Benchmark your real workload, not a synthetic. SMT's benefit is proportional to stall cycles, so a memory-latency probe and a hand-tuned vector kernel will give you opposite verdicts on the same host and both will be true. Run the builds, the agent sessions, the queries your customers actually run.
2. Watch tail latency, not just aggregate throughput. SMT typically raises total throughput while making any individual thread slower, because that thread is now sharing a core. If your product is a p99 — and for a sandbox platform, create latency is the product — the aggregate number is the wrong one to optimise.
3. Include the control-plane path. On PandaStack a create is a snapshot restore, roughly 179ms p50 and 203ms p99 with the restore step itself around 49ms; a same-host fork is 400–750ms. Those are host-side work competing for the same cores as tenant guests, so an SMT change moves them too, and they are the numbers customers feel first.
4. A/B at the host level, with real traffic. Take two identically specced hosts, flip one, and route production work at both. A single-host microbenchmark with nothing else running tells you about an empty machine, and an empty machine is not the thing you are worried about.
5. Re-measure on every CPU generation. The SMT gain, the mitigation set, and the sysfs strings all move between microarchitectures. A measurement from your previous instance family is a historical document, not a fact about your fleet.

What a guest can see about all this: nothing

Worth checking from the other side, because it changes how you communicate the decision. Firecracker presents the guest a normalised CPU topology; the guest does not get a view of the host's physical core layout. Verify the exact behaviour against Firecracker's current documentation for your version, but the operational upshot is stable: a tenant cannot tell whether they are sharing a physical core with a stranger, and cannot audit your answer either.

from pandastack import Sandbox

# What can a tenant learn about the core they are sharing? Almost nothing --
# and that is a property of the hypervisor's design, not an oversight.
sbx = Sandbox.create(template="base", ttl_seconds=300)
try:
    print(sbx.exec("lscpu -e=CPU,CORE,SOCKET").stdout)
    # The guest sees a flat topology: N logical CPUs, N distinct cores,
    # one thread per core. It looks like a machine with SMT disabled
    # whether or not the host has SMT disabled.

    print(sbx.exec("cat /sys/devices/system/cpu/smt/control 2>/dev/null "
                   "|| echo 'smt/control: absent'").stdout)
    # Usually absent or "notsupported". The guest kernel is telling the
    # truth about the machine it was handed, which is not the machine it
    # is running on.

    print(sbx.exec(
        "grep -H . /sys/devices/system/cpu/vulnerabilities/* | sed 's#.*/##'"
    ).stdout)
    # Guest-side mitigations, chosen by the guest kernel from the CPU
    # features the hypervisor exposed. This says NOTHING about whether the
    # HOST has SMT enabled, flushes L1D on VM entry, or has current
    # microcode. Those three facts live on the host and are audited there.
finally:
    sbx.kill()

Which cuts both ways. Your tenants cannot verify your SMT posture, so if it is part of what you are selling, it has to be something you state and evidence rather than something they can check. And a screenshot of clean guest sysfs proves nothing about the host — a point worth remembering the next time a vendor sends you one.

Making the call

The decision is not really a security decision; it is a tenancy decision wearing a security costume. Six steps, in order.

1. Name your tenants out loud. Are they you, your colleagues, customers under contract with a named legal entity behind them, or strangers with a stolen credit card and an API key? The first two cases make rung 1 defensible. The last one does not: if anyone can buy a sandbox in thirty seconds, sibling sharing between trust domains is not a position you will enjoy defending later.
2. Get anything worth stealing off tenant-facing hosts. No control-plane credentials, no signing keys, no customer database passwords on a machine running untrusted guests. This is cheap, it is unglamorous, and unlike everything else on this list it survives every future variant, because it removes the target rather than hardening the channel.
3. Pick a rung deliberately and write it down. With a date, an owner, and the condition that would move you up — "when we open self-serve signup", "when we take our first customer with a regulator". A decision you can point at is worth more than a configuration that happens to be correct.
4. Measure the cost before you price it. Run the A/B above on your real workload. Then, if isolation costs you cores, put that in the price list rather than absorbing it silently until someone notices the margin.
5. Verify it continuously, in the image build. Read smt/control and grep the vulnerabilities directory as a build gate, fail the build on drift, and alert if a running host disagrees with policy. Configuration that is only correct on the day you set it is a story with a second act.
6. If you pay for isolation, sell it. Dedicated cores are a real product with a real cost and customers who genuinely need them will pay for them. Giving it away to everyone is the option that makes the security decision look like a pure loss on the P&L, which is how it eventually gets reversed by someone who was not in this conversation.

Where PandaStack lands, and what it doesn't claim

Every sandbox, managed database and hosted app on PandaStack is its own Firecracker microVM: a KVM guest with its own kernel, a minimal virtio device model, a seccomp-filtered VMM, and a dedicated network namespace drawn from a pool of 16,384 pre-allocated /30 subnets. That architecture removes the shared-kernel failure class outright — one tenant's code never executes against another tenant's kernel — and it makes egress policy a per-sandbox property rather than a fleet-wide one. Those are the claims I am comfortable making, and they are the claims that cover the failure modes that actually appear in incident reports.

What it does not do is un-share a physical core, because nothing running on shared silicon does. The SMT decision sits with whoever operates the hosts, it is made per fleet and per tenancy posture, and it is exactly the four-rung decision above rather than a checkbox in a product. Anyone who tells you their virtualization layer solved sibling-thread side channels has either invented new hardware or is describing dedicated cores without mentioning the invoice.

The one structural thing that does help, and it fell out of performance work rather than a security review, is lifetime. Because a create is a snapshot restore rather than a boot — around 179ms p50 — keeping a sandbox alive is the unusual choice rather than the default one. Run the job, take the output, kill the VM. An attacker who needs stable co-residency on a specific core, with a specific victim, for long enough to beat the noise floor, is having a considerably worse time against a workload that exists for one task on a host they did not choose than against a container that has been up since March. Short lifetimes are a mitigation. They are just a mitigation that also happens to be the cheapest way to run the platform, which is my favourite category of engineering.

To go further: /blog/side-channel-attacks-multi-tenant-compute-explained is the general survey this post deliberately avoided repeating; /blog/microvm-cpu-pinning-noisy-neighbor covers core placement through a performance lens instead of a security one; /blog/firecracker-vcpu-scheduling-model explains why a guest vCPU is just a host thread in the first place; and /blog/firecracker-cpu-templates-explained covers what the guest is told about the CPU it is running on, which is a surprisingly load-bearing question once you start caring about this stuff.