GPU Passthrough and microVMs: The Honest Answer

This question arrives in some form almost every week, usually from someone building an AI product: can I attach a GPU to a Firecracker microVM? They want the thing everyone wants — the isolation of a hardware-virtualized guest with its own kernel, plus a real accelerator inside it, so untrusted model code can run CUDA without running it on a machine that matters.

The answer is no. Not "it's tricky", not "it depends on your kernel version", not "there's a patch". Firecracker does not do GPU passthrough, and the reason it does not is the same reason you picked Firecracker in the first place. I would rather say that plainly than sell you a workaround that doesn't exist, so the first half of this post is the no, in detail, and the second half is the architecture that people who asked this question end up shipping anyway.

Why Firecracker can't, and why that's deliberate

Firecracker was written for a specific job: run enormous numbers of small, untrusted, short-lived guests on shared hardware with the smallest plausible amount of emulated code between the guest and the host. Every device a VMM emulates is code that parses attacker-controlled input from inside the security boundary. So Firecracker emulates almost nothing — a block device, a network device, a vsock, a serial port, a few MMIO windows — and it reaches them over virtio-MMIO, which means a fixed register block at a known physical address rather than a bus the guest walks.

There is no PCI bus in a Firecracker guest. The kernel command line typically carries `pci=off`, and the guest doesn't even probe for one. Device assignment — handing a real physical device to a guest — is built on top of PCI and IOMMU plumbing: config space, base address registers, interrupt remapping, MSI-X tables, a VFIO container the VMM maps into guest memory. None of that machinery exists in Firecracker, and adding it would mean adding back the single largest chunk of surface the project exists to avoid. I've written about the transport choice in more detail in /blog/firecracker-mmio-vs-pci-transport-explained; GPU passthrough is the sharpest consequence of it.

So this isn't a missing feature waiting on someone's weekend. It's a design decision with a coherent rationale, and the correct reaction is not to wait for it but to check whether you needed the GPU inside the sandbox at all. Usually you don't, which we'll get to.

What GPU passthrough actually involves

It's worth understanding what you're asking for, because the mechanics explain both why Firecracker skipped it and why it's a poor fit for disposable sandboxes even where it works. On a hypervisor that supports it — QEMU/KVM, Cloud Hypervisor — passthrough means detaching a physical PCIe device from the host's own driver and handing the device's memory and interrupts directly to a guest.

Unbinding the device and binding vfio-pci

The host has to stop driving the card. You find the device's PCI address, unbind whatever driver claimed it, and bind `vfio-pci` instead, which is the kernel's userspace device-assignment framework. From that moment the host cannot use the GPU at all: no display, no CUDA on the host, no monitoring through the vendor driver. The card belongs to whichever process opens the VFIO container.

# 1. Find the card and see who currently drives it.
lspci -nnk | grep -A3 -i nvidia
# 41:00.0 3D controller [0302]: NVIDIA Corporation ... [10de:20b5]
#         Kernel driver in use: nvidia
#         Kernel modules: nouveau, nvidia

# 2. Look at the IOMMU group. Everything in this directory travels together.
readlink -f /sys/bus/pci/devices/0000:41:00.0/iommu_group
# /sys/kernel/iommu_groups/57
ls /sys/kernel/iommu_groups/57/devices/
# 0000:41:00.0   <- just the GPU here. On a consumer board you may also see
#                   the audio function, a bridge, and an unrelated NIC.

# 3. Hand it to VFIO. The host loses the card at this point.
echo 0000:41:00.0 > /sys/bus/pci/devices/0000:41:00.0/driver/unbind
echo vfio-pci     > /sys/bus/pci/devices/0000:41:00.0/driver_override
echo 0000:41:00.0 > /sys/bus/pci/drivers_probe

# 4. Confirm. "Kernel driver in use: vfio-pci" means it is assignable.
lspci -nnk -s 0000:41:00.0

IOMMU groups, and why a bad group ruins your day

The IOMMU group is the kernel's unit of isolation: the smallest set of devices that the platform can guarantee are isolated from each other in terms of DMA. If your GPU shares a group with a USB controller and a network card — common on consumer boards where the PCIe topology doesn't provide proper isolation between downstream ports — you cannot pass just the GPU. You pass the whole group or nothing, because passing part of a group would let the guest DMA its way into devices the host still owns. Server platforms with proper PCIe ACS support usually give each slot its own group; desktop boards frequently do not, which is the origin of every forum thread about ACS override patches. Those patches work by lying to the kernel about isolation guarantees the hardware doesn't make, which is fine for a homelab and disqualifying for multi-tenant.

BAR mapping, reset quirks, and exclusive ownership

Once assigned, the VMM maps the device's BARs — the memory windows through which the driver talks to the card, including the large aperture that covers video memory — into the guest's physical address space, and routes MSI/MSI-X interrupts into the guest. The guest then loads a real vendor driver against real hardware. Two consequences matter for sandboxing. First, the device is exclusively owned by that guest for its entire lifetime; there is no timesharing, no oversubscription, no second tenant on the card. Second, getting the card back cleanly requires a working reset. Function-level reset is not universally well behaved on GPUs, and a card that doesn't reset properly comes back in an unusable state until the host reboots.

Now hold that next to a microVM's economics. A sandbox that exists for 400 milliseconds, holding an accelerator that costs as much as a car, exclusively, with a reset dance between tenants that might wedge the host. Even in a world where Firecracker supported VFIO, this would be a strange thing to build.

What a Firecracker machine actually looks like

The absence is easier to see than to describe. Here is the whole shape of a Firecracker guest configuration: a machine config, one or more drives, a network interface, a boot source. There is no devices array, no PCI topology, no place a device address would go.

{
  "boot-source": {
    "kernel_image_path": "/var/lib/pandastack/kernels/vmlinux-5.10",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  },
  "machine-config": {
    "vcpu_count": 8,
    "mem_size_mib": 4096,
    "smt": false
  },
  "drives": [
    { "drive_id": "rootfs", "path_on_host": "/var/lib/pandastack/vms/abc/clone.ext4",
      "is_root_device": true, "is_read_only": false }
  ],
  "network-interfaces": [
    { "iface_id": "eth0", "host_dev_name": "tap0" }
  ],
  "vsock": { "guest_cid": 3, "uds_path": "/var/lib/pandastack/vms/abc/v.sock" }

  // Note what is NOT here and cannot be added: no "devices", no PCI bus,
  // no VFIO group, no device address. The boot args say pci=off because
  // there is no bus for the guest to probe.
}

That configuration is the whole hardware universe your guest inhabits. It's a short list, and its shortness is the product.

The alternatives, fairly

If you genuinely need the device visible inside an isolated guest, there are real options — with real costs. I'm describing designs, not benchmarking claims; every one of these projects moves faster than a blog post, so verify capabilities against their own current documentation before you commit.

• Firecracker microVM — GPU access: none (no PCI bus, no VFIO). Isolation: guest kernel under KVM, minimal device model. Use it for: untrusted CPU work — agent code, notebooks, builds, preprocessing — where fast create and a small VMM matter more than device access.
• Cloud Hypervisor + VFIO — GPU access: yes, PCIe device assignment. Isolation: guest kernel under KVM, larger device model than Firecracker. Use it for: a Rust VMM that keeps some of the minimalist spirit while supporting passthrough; verify current device-assignment support and caveats in its own docs.
• QEMU/KVM + VFIO — GPU access: yes, the mature and best-documented path. Isolation: guest kernel under KVM, but a very large VMM and device model. Use it for: long-lived GPU VMs where flexibility and hardware coverage outweigh a bigger trusted computing base.
• Kata Containers + GPU device plugin — GPU access: yes, via passthrough into the Kata guest. Isolation: a VM boundary presented behind a container UX and OCI runtime. Use it for: Kubernetes shops that want a hardware boundary without rewriting their scheduling; the PCIe machinery is still underneath, with the same IOMMU and reset constraints.
• Container + NVIDIA container runtime — GPU access: yes, the overwhelmingly common answer. Isolation: namespaces and cgroups only — shared host kernel and shared GPU driver. Use it for: trusted or semi-trusted code where the operational simplicity is worth having no hardware boundary.
• MIG / vGPU / time-slicing — GPU access: shared, a slice of one card per tenant. Isolation: vendor-implemented partitioning; MIG advertises hardware-level separation of compute and memory, which is the vendor's claim and should be read in their docs and weighed against your threat model. Use it for: pooling one expensive card across many tenants when you accept vendor-level partitioning as your boundary.

What "GPU isolation" even means

There's an uncomfortable fact under all of these options, and it's worth stating even-handedly because it applies to approaches I like as much as to ones I don't.

The GPU driver is a small operating system that you did not audit and cannot read. Whatever your isolation story, it is somewhere in the trust boundary.

A modern accelerator driver is a large privileged kernel module handling command submission, memory management, scheduling, and firmware that runs on the device itself. In a container, that module is shared by every tenant on the host and sits directly in the host kernel — a bug there is a host compromise. Under passthrough you move the driver into the guest, which is a genuine improvement: the module now sits inside a boundary you can throw away. But the device is still doing DMA into guest memory, still running vendor firmware, and you're now relying on the IOMMU to contain a device that a hostile guest is programming directly.

Vendor partitioning — MIG in particular — is a real engineering effort and the claims are specific about separating compute units, memory, and caches. It is also, unavoidably, a boundary implemented by the vendor in hardware and firmware you cannot inspect, evaluated by people who are not you. That may be entirely acceptable for your threat model. It is a different proposition from "a guest kernel under KVM with a VMM I can read", and the honest move is to name which one you're relying on rather than blur them into the single word "isolated".

The split architecture that works today

Here's the part that's actually useful. Almost every team that asks about GPU passthrough for sandboxes is building the same shape: untrusted code — an AI agent's generated Python, a customer's notebook, user-supplied preprocessing, an evaluation harness — that at some point wants to run a model. The instinct is to put the whole thing in one isolated box with a GPU in it. The better decomposition is to split on trust and put the accelerator on the other side of an API.

Untrusted orchestration and glue code goes in a microVM, where isolation is the point and create latency is part of the product. The GPU work goes on separate, trusted hosts that speak an inference API and nothing else. The sandbox gets an HTTP endpoint and a scoped token; it never sees a device node, never loads a driver, and never needs to.

from pandastack import Sandbox

# The untrusted half. No GPU in here -- deliberately. This VM exists to run
# code we did not write, for as long as one task takes, and then die.
INFERENCE_URL = "http://inference.internal:8000/v1/generate"

def run_agent_step(agent_code: str, prompt: str, scoped_token: str) -> dict:
    sbx = Sandbox.create(
        template="base",
        ttl_seconds=300,                  # hard backstop, enforced by the platform
        metadata={"job": "agent-step"},
    )
    try:
        sbx.filesystem.write("/work/step.py", agent_code)

        # The sandbox reaches the GPU pool the same way any other client does:
        # an authenticated HTTP call to a service on trusted hosts. The token
        # is scoped to one tenant and expires with the task, so the worst a
        # rooted guest gets is the ability to spend its own quota.
        result = sbx.exec(
            "cd /work && INFERENCE_URL=%s TOKEN=%s python3 step.py"
            % (INFERENCE_URL, scoped_token),
            timeout_seconds=240,
        )
        return {"ok": result.exit_code == 0, "output": result.stdout}
    finally:
        sbx.delete()

# The GPU never enters the blast radius, because it was never in the VM.
# Creating that VM is a snapshot restore: ~179ms p50, ~203ms p99.

The reason to like this isn't that it routes around a limitation. It's that it's the right architecture even on a hypervisor that supports passthrough, for reasons that have nothing to do with Firecracker.

1. GPU capacity is expensive and lumpy. Accelerators want to be pooled, queued, and batched across many requests to stay busy. Stranding one inside a per-request sandbox is the single most effective way to waste it — you've taken your scarcest resource and given it the utilization profile of your burstiest workload.
2. The untrusted code almost never needs the device. It needs an inference result. Once you notice that, the device requirement dissolves into an API requirement, and API requirements are easy to authenticate, rate-limit, audit, and cache.
3. Batching lives on the trusted side. Continuous batching, KV-cache reuse, and multi-request scheduling all require a server that sees many requests at once. A sandbox with its own GPU sees exactly one, forever.
4. The blast radius shrinks in both directions. A compromised sandbox reaches an authenticated endpoint with a scoped token and a quota, not a device node and a driver. And a wedged GPU host doesn't take a tenant's sandbox with it — the failure domains separate cleanly.
5. The two halves scale on different curves. Sandbox demand is spiky and cheap to satisfy: a create is a snapshot restore, and networking comes from a pool of 16,384 pre-allocated subnets per host. GPU demand is smooth and expensive. Coupling them forces the expensive one to follow the spiky one.

When you really do need a GPU in the guest

Some workloads don't decompose. If a customer brings you their own CUDA kernels, or a training job needs sustained device access for hours, or you're renting out raw accelerator time as the product, then an API in front of the GPU isn't a boundary — it's the thing you're selling access to. In those cases you want passthrough, and you want a VMM that does it: QEMU/KVM for maturity and hardware coverage, Cloud Hypervisor if you want a smaller Rust VMM and its device-assignment support fits, or Kata if you want that boundary underneath a Kubernetes-shaped hole.

Just size the unit correctly when you do. A passed-through GPU is a long-lived, exclusively-owned resource with a rough handover between tenants; treat it like leasing a machine, not like spawning a sandbox. Bill it by the hour, schedule it with real reservations, and put a reset-and-verify step between tenants that you actually test. The disposable, sub-second, thousands-per-host model that microVMs are good at is a different product, and mixing the two mostly produces the costs of both.

So: no, Firecracker can't attach a GPU, and that is unlikely to be the sentence that decides your architecture. The useful question was never "which hypervisor lets me put a GPU in the sandbox" — it was "which half of my system actually needs the accelerator". For nearly everyone asking, the answer is the half that was already trusted.