Signing and Attesting microVM Templates

Container images spent a decade growing a supply chain: content-addressed digests, cosign signatures, SBOM attachments, SLSA provenance, admission controllers that refuse an unsigned image. VM templates got a bucket and a hope. Somewhere in most microVM fleets there is a multi-gigabyte file that becomes the kernel and userspace of every guest you boot, and the strongest thing anyone can say about how it got there is that the sync job exited zero.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, so read this as opinionated and as partly self-critical. Our template seeds are published per generation to object storage with an immutable pointer and a SHA256 manifest, which is the right shape — and while writing this post I went and read our own manifest struct and found a gap in it, which I will show you rather than tidy away.

First, the inventory — and it is not just the rootfs

Ask an engineer what needs signing in a VM image pipeline and you will hear "the image" — meaning the root filesystem. That answer is roughly a third of the artifact set on any platform that boots from snapshots, and the missing two thirds are the interesting ones. Here is what one template generation actually consists of on our hosts, which is close enough to any Firecracker fleet to be useful.

• The guest kernel — an uncompressed vmlinux, 5.10 in our case. It is the code that executes first and with the most authority. It is also, on most fleets, provisioned separately from the rootfs by a different script written at a different time, which is exactly how things escape an inventory.
• The template rootfs — rootfs.ext4, the disk built from the Dockerfile and reflinked per sandbox. This is the file everyone remembers.
• The snapshot disk — clone.ext4, the frozen post-boot filesystem that every restore actually starts from. It is not byte-identical to rootfs.ext4; boot wrote to it.
• The snapshot VM state — vm.state, Firecracker's serialized vCPU registers and device topology. Small, structured, and it decides what devices the guest believes it has.
• The snapshot memory — vm.mem, the whole of a booted guest's RAM. Multiple gigabytes. The single most security-relevant file in the set and the one most likely to be unsigned, because it is too big to be convenient.
• The streaming sidecars — vm.mem.header (which chunks are non-zero) and vm.mem.prefetch (the hot chunk set). Tiny metadata files that steer what gets fetched and what gets zero-filled. An attacker who can edit only these has a surprisingly good day.
• The identity and sizing metadata — identity.json, which freezes the guest IP, MAC and gateway that were true at bake time, and meta.json, which pins cpu, memory_mb and disk_gb. The guest's whole network identity is a JSON file on a bucket.
• The pointer — whatever object says which generation is current. Sign every artifact perfectly and leave the pointer unauthenticated and you have built a very rigorous system for booting whichever old generation an attacker prefers.

Now the sharp version. On a snapshot-restore platform, signing the rootfs is signing the cover of a book you are not going to read. We do not boot rootfs.ext4 into a fresh kernel on create — we restore a snapshot: about 179ms p50 end to end, with the load step near the middle of that. A cold boot from the rootfs happens roughly once per template per host, around 3 seconds, and then never again until the next bake. The file your signature covers is the one your platform was specifically engineered to stop executing.

A rootfs signature attests to a filesystem. A snapshot is a kernel that has already finished thinking about that filesystem, with the conclusions in RAM. Signing the first and not the second is signing the ingredients and shipping someone else's cake.

Why the memory image is the artifact you should be nervous about

Take the attacker's view for a paragraph. A tampered rootfs is a normal filesystem attack: you plant a binary, change a unit file, add a key to authorized_keys, and then you wait for something to execute it. Every boot-time defence in the industry — Secure Boot, dm-verity, an initramfs that checks a hash tree — exists to catch precisely that, on the way up.

A tampered vm.mem skips all of it, because there is no way up. Restore does not boot; it reinstates a kernel mid-stride. Page tables, the kernel's own text and data, sshd's already-parsed configuration, whatever credentials a process was holding, the state of every LSM — all of it is file content now, and the restore path's job is to install those bytes into guest RAM as fast as physics allows. No integrity check runs, because the code that would have run it is itself part of the image being installed. You cannot verify a kernel using that kernel.

It gets slightly worse when the memory is streamed rather than downloaded. We serve guest page faults on demand out of object storage — userfaultfd on the host, 4 MiB range GETs against the bucket — so pages arrive lazily, individually, over the guest's whole lifetime. That is excellent for boot latency and it means the integrity question is no longer "was the file correct when we installed it" but "is every chunk still correct at the moment we fetch it, hours later." Whole-file hashing has a real answer to that and it is not a comfortable one: you would have to hash the whole file first, which is the download you just spent a year eliminating.

Sign a manifest, not a pile of files

The mechanism that fixes this is old and boring: one manifest per generation, listing every file and its SHA256, and exactly one signature — over the manifest. Signing each file individually sounds more thorough and is strictly worse. Per-file signatures let a valid signed file from generation seven sit next to valid signed files from generation nine, and every individual check passes while the set as a whole is a chimera nobody ever baked or tested. A manifest signature says something the per-file version cannot: these files, this set, this generation, together.

Three properties make it work. The manifest must be complete — if a file is not in it, it is not covered, and the omission is invisible at verification time. It must be bound to an identity: template name, generation, schema version, source commit, so that a valid manifest for a different template cannot be swapped in. And the generation must be immutable, with a separate mutable pointer saying which generation is current, so "upgrade" and "tamper" are distinguishable operations.

That last part is a shape most fleets already have and do not think of as security. Ours publishes each seed under a per-generation prefix, uploads the payload and manifest first, and flips a CURRENT pointer last — so a host that reads CURRENT always sees a fully-uploaded generation. That was designed as a race fix. It happens to be exactly the substrate signing needs, which is the usual story: digest-pinning arrives for operational reasons and the signature is a layer you add on top, not an architecture you rebuild.

#!/usr/bin/env bash
# Build a SHA256 manifest over EVERY file in one template generation.
# Runs on the bake host, right after the snapshot is captured, BEFORE upload.
set -euo pipefail

GEN_DIR="$1"      # /var/lib/pandastack/seeds/base/20260908T1142Z
TEMPLATE="$2"     # base | code-interpreter | agent | browser | postgres-16
cd "$GEN_DIR"

# The inventory, written down once, in one place. Every argument about what
# needs signing should end at this array -- and note that a file missing from
# it is not "unsigned", it is INVISIBLE. Verification cannot fail on a line
# that does not exist.
FILES=(
  vmlinux           # guest kernel 5.10 -- the code that actually executes
  rootfs.ext4       # template disk (cold boot only, but sign it anyway)
  clone.ext4        # snapshot disk -- what every restore really starts from
  vm.state          # vCPU + device state
  vm.mem            # a booted kernel's entire RAM. yes, this one especially
  vm.mem.header     # zero-chunk index: a lie here changes what is zero-filled
  vm.mem.prefetch   # hot-chunk trace
  identity.json     # baked guest IP / MAC / gateway
  snap-meta.json    # sizes the restore path cross-checks
  meta.json         # cpu / memory_mb / disk_gb the guest is pinned to
)

for f in "${FILES[@]}"; do
  [ -f "$f" ] || { echo "manifest: missing artifact: $f" >&2; exit 1; }
done

# One canonical format, sorted by name so the bytes are stable across runs.
sha256sum "${FILES[@]}" | LC_ALL=C sort -k2 > SHA256SUMS

# Bind the digest list to an IDENTITY. A bare SHA256SUMS is a checksum of
# "some files"; this makes it a checksum of THIS generation of THIS template,
# which is what stops a perfectly valid manifest for another template from
# being dropped in its place.
cat > generation.json <<EOF
{
  "template":     "$TEMPLATE",
  "generation":   "$(basename "$GEN_DIR")",
  "schema":       4,
  "firecracker":  "v1.16.0",
  "guest_kernel": "5.10",
  "base_image":   "ubuntu-24.04",
  "source_commit":"$GIT_COMMIT",
  "dockerfile":   "templates/$TEMPLATE/Dockerfile",
  "sums_sha256":  "$(sha256sum SHA256SUMS | cut -d' ' -f1)"
}
EOF

# generation.json is now the ONE file that needs a signature. Everything else
# is reachable from it by digest.
sha256sum generation.json

cosign on blobs, not images

Most sigstore documentation assumes you have an OCI image and a registry to attach signatures to. A rootfs.ext4 and a 4 GiB memory dump are neither. The blob subcommands cover this case and they are less advertised than they deserve: sign-blob and verify-blob operate on an arbitrary file, and since our arbitrary file is a small JSON manifest, the cost is independent of how many gigabytes the generation weighs.

# --- sign, on the bake host ------------------------------------------------
# Keyless: identity comes from the CI OIDC token, the certificate is minted by
# Fulcio and logged to Rekor. There is no long-lived private key to steal,
# which is genuinely the best property in this whole post.
COSIGN_EXPERIMENTAL=1 cosign sign-blob \
  --yes \
  --bundle generation.json.bundle \
  generation.json

# Key-based, for the air-gapped and the sceptical. The key can live in a KMS
# or on a hardware token -- --key accepts kms:// URIs as well as a file.
cosign sign-blob \
  --key "awskms:///alias/pandastack-template-signing" \
  --bundle generation.json.bundle \
  generation.json

# --- verify, on the HOST, before install -----------------------------------
# Keyless verification is an ASSERTION ABOUT WHO SIGNED, not just "a signature
# exists". Pin both the identity and the issuer or you are accepting any
# certificate Fulcio ever minted for anyone.
cosign verify-blob \
  --bundle generation.json.bundle \
  --certificate-identity-regexp '^https://github\.com/pandastack/infra/\.github/workflows/bake-templates\.yml@refs/heads/main$' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  generation.json

# Key-based verification, offline. --insecure-ignore-tlog is required when
# there is no transparency log to reach, and it means exactly what it says:
# you keep the signature check and lose the "was this ever published" check.
cosign verify-blob \
  --key /etc/pandastack/template-signing.pub \
  --bundle generation.json.bundle \
  --insecure-ignore-tlog=true \
  generation.json

# Flags in this tool have churned across releases. Pin a cosign version in
# both the signer and the verifier and check them against the version you
# pinned, not against a blog post -- including this one.

Keyless versus key-based is usually presented as a preference. It is closer to a question about where verification happens. Keyless is genuinely lovely in CI: the signing identity is the workflow, the certificate is short-lived, there is no key material anywhere for anyone to exfiltrate, and the Rekor entry gives you a public record that this generation was published at this time by that workflow. It is markedly less lovely on a host in a locked-down VPC that is not supposed to reach the public internet, or on a truly air-gapped one, where verification now depends on services outside your failure domain. The honest arrangement I would defend is keyless in CI plus a key-based counter-signature from a KMS for the hosts, verified offline — two signatures because they answer to two different audiences.

Attestation: recording how the thing was baked

A signature says these bytes are the bytes we published. It says nothing about where they came from, and "where did this template come from" is the question that actually arrives — usually months later, usually phrased as "why does the browser template have that package in it," usually while somebody is on a call. An in-toto statement with a SLSA provenance predicate answers it, and it costs one more file in the same pipeline.

The important detail for templates specifically is the subject list. For a container image the subject is one digest. For a template generation it is the whole set — kernel, disks, snapshot state and memory — because the thing you are making a claim about is a bootable machine assembled from several files, not any one of them. Attest the set, and the resolved dependencies field is where you record which base image, which Dockerfile, and which Firecracker binary took the snapshot.

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [
    { "name": "vmlinux",    "digest": { "sha256": "0f3c...a91b" } },
    { "name": "clone.ext4", "digest": { "sha256": "77d2...4e08" } },
    { "name": "vm.state",   "digest": { "sha256": "b410...cc7f" } },
    { "name": "vm.mem",     "digest": { "sha256": "e9a7...1d33" } },
    { "name": "identity.json", "digest": { "sha256": "5511...09ae" } }
  ],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://pandastack.ai/buildtypes/microvm-template/v1",
      "externalParameters": {
        "template": "browser",
        "dockerfile": "templates/browser/Dockerfile",
        "platform": "linux/amd64",
        "repository": "https://github.com/pandastack/infra",
        "ref": "refs/heads/main"
      },
      "internalParameters": {
        "snapshot_taken_by": "firecracker v1.16.0",
        "guest_kernel": "5.10",
        "memory_class": "4KiB",
        "bake_host_class": "isolated-runner"
      },
      "resolvedDependencies": [
        { "uri": "pkg:docker/ubuntu@24.04",
          "digest": { "sha256": "3d1b...77c2" } },
        { "uri": "git+https://github.com/pandastack/infra@a1b2c3d",
          "digest": { "sha1": "a1b2c3d4e5f60718293a4b5c6d7e8f9012345678" } }
      ]
    },
    "runDetails": {
      "builder": { "id": "https://github.com/pandastack/infra/.github/workflows/bake-templates.yml" },
      "metadata": { "invocationId": "bake-2026-09-08T11:42:03Z-browser" }
    }
  }
}

Two things this buys that are not security, and which are frankly why it survives contact with a roadmap. It makes a template's contents diffable across generations, so "what changed between the generation that worked and the one that didn't" is a query rather than an archaeology project. And it makes the bake reproducible enough to argue about — if the recorded base image digest and source commit produce a different rootfs today, that is information, and it is information you cannot obtain at all if all you kept was a timestamp.

The crux: verify on the host, at install, before first boot

This is the part people get wrong, and they get it wrong in a way that produces green checkmarks. A verification step in the publish pipeline proves the artifact was correct at the moment CI signed it — which nobody was disputing. CI is the one participant in this system with no motive and no opportunity. The threat you actually care about is everything downstream: a bucket object modified after publication, a truncated sync that leaves a plausible-looking file, a stale generation served from a cache, a compromised mirror, a host that pulled while the pointer was mid-flip, or the perfectly ordinary case of a network blip during a multi-gigabyte transfer.

A signature check that only ever runs in the pipeline that produced the signature is not a control. It is a unit test with a certificate.

So the check belongs on the host, at install time, in the sync path, before the artifact is ever visible to the boot path. Concretely: download into a staging directory, verify the manifest signature, verify every digest in the manifest against the bytes on local disk, and only then move the generation into place with an atomic rename. On mismatch, refuse — leave the previous generation installed and let the host keep serving what it already trusts. Our seed sync already does the checksum half of this: it downloads into a staging directory, compares against tar_sha256 and the recorded byte length, and on mismatch abandons the seed entirely and falls back to a local cold bake. The signature is the layer that goes around it.

#!/usr/bin/env bash
# seed-install: runs on the AGENT HOST, in the sync path, before anything in
# this generation is reachable by the boot path.
set -euo pipefail

TEMPLATE="$1"
SEEDS=/var/lib/pandastack/seeds/$TEMPLATE
STAGE=$(mktemp -d "$SEEDS/.stage.XXXXXX")
trap 'rm -rf "$STAGE"' EXIT

# 1. Resolve the pointer, then NEVER trust it again. Everything below is
#    addressed by the generation id, so a pointer that flips mid-sync gives
#    us a failed verify, not a mixed generation.
GEN=$(gcloud storage cat "gs://$BUCKET/seeds/$TEMPLATE/CURRENT" | tr -d '\n')
[ -n "$GEN" ] || { echo "empty CURRENT pointer" >&2; exit 1; }
PREFIX="gs://$BUCKET/seeds/$TEMPLATE/$GEN"

gcloud storage cp -r "$PREFIX/*" "$STAGE/"
cd "$STAGE"

# 2. Signature over the manifest. This is the only cryptography here; the
#    rest is arithmetic that the signature makes meaningful.
cosign verify-blob \
  --key /etc/pandastack/template-signing.pub \
  --bundle generation.json.bundle \
  --insecure-ignore-tlog=true \
  generation.json \
  || { echo "REFUSE: signature failed for $TEMPLATE/$GEN" >&2; exit 2; }

# 3. The manifest must describe the generation we asked for. Without this, a
#    correctly-signed OLD generation is a completely valid rollback attack.
python3 - "$TEMPLATE" "$GEN" <<'PY' || exit 2
import json, sys
m = json.load(open("generation.json"))
want_tpl, want_gen = sys.argv[1], sys.argv[2]
if m["template"] != want_tpl or m["generation"] != want_gen:
    sys.exit(f"REFUSE: manifest is {m['template']}/{m['generation']}")
PY

# 4. The signed manifest commits to SHA256SUMS; SHA256SUMS commits to the
#    bytes. Check both links or the chain has a hole in the middle.
EXPECT=$(python3 -c 'import json;print(json.load(open("generation.json"))["sums_sha256"])')
ACTUAL=$(sha256sum SHA256SUMS | cut -d' ' -f1)
[ "$EXPECT" = "$ACTUAL" ] || { echo "REFUSE: SHA256SUMS not the signed one" >&2; exit 2; }

sha256sum -c SHA256SUMS --quiet \
  || { echo "REFUSE: artifact digest mismatch in $TEMPLATE/$GEN" >&2; exit 2; }

# 5. Only now does it become visible to the boot path. Rename is atomic, so
#    there is no window where a half-verified generation is installable.
mv "$STAGE" "$SEEDS/$GEN"
ln -sfn "$SEEDS/$GEN" "$SEEDS/.current.new" && mv -T "$SEEDS/.current.new" "$SEEDS/current"
trap - EXIT
echo "installed $TEMPLATE/$GEN"

# Exit 2 leaves the PREVIOUS generation in place and serving. A host that
# refuses a bad update and keeps running is a host that verified something.

Three details in that script earn their place. Resolving the pointer once and addressing everything by generation id afterwards, so a flip during the sync produces a clean failure instead of a Frankenstein install. Checking that the signed manifest names the generation you asked for, which is the entire defence against a rollback to a correctly-signed old template — the one with the CVE you patched last month. And the atomic rename at the end, because a verification that runs while the boot path can already see the files is a race, not a gate.

The uncomfortable bit: whoever holds the key holds everything

Time to say the quiet part. A signing key sitting on the bake host, signing every generation automatically as it is produced, is a single point of total compromise. Not a partial one. That key authenticates the kernel and the memory image of every guest your platform boots, so an attacker who obtains it can publish a template that every host in the fleet will accept, verify happily, and restore into production — and your verification gate will not merely fail to stop it, it will attest to it.

Worse, automated signing quietly redefines what the signature means. If the pipeline signs whatever it builds without a human in the loop, the signature does not assert "a person reviewed this template". It asserts "our CI ran". That is still worth having — it is a strong statement about origin and it stops every attack that involves modifying an artifact after publication — but be precise about the claim, because the gap between what a signature means and what people assume it means is where the incident lives.

• Key in a file on the bake host — Stops: bucket tampering, partial syncs, a compromised mirror. Does not stop: anyone who can run a command on the bake host, which on most teams is a longer list than expected, and which includes every build step that runs there.
• Key in a KMS or HSM, bake host holds only sign permission — Stops: key exfiltration outright; the key cannot leave. Does not stop: an attacker on the bake host signing arbitrary bytes while their access lasts. It converts a permanent compromise into a bounded one, which is a real improvement and not a solution.
• Keyless OIDC in CI — Stops: key theft entirely, since there is no long-lived key, and gives you a public transparency-log record of every signing event. Does not stop: someone who can push to the branch the workflow trusts. Your signing identity is now your branch protection.
• Two-person promotion — Stops: any single compromised identity publishing a template the fleet accepts, because CI signs a candidate and a separate, human-gated key signs the promotion to CURRENT. Costs: a human in the loop of every template release, which teams accept for production and abandon for staging within about three weeks.

My honest position: the highest-leverage move is not a better key, it is a smaller signing surface. Sign in a workflow that does nothing else, from a branch nobody pushes to directly, on a runner that does not also run tests, with a key that can only sign template manifests and nothing else in the organisation. Most signing-key compromises are not cryptographic events. They are somebody's build step reading an environment variable on a machine that had too many jobs.

Revocation, honestly: you mostly move a pointer

Every design like this eventually meets the question "what do we do when a signed template turns out to be bad", and the honest answer is less impressive than the rest of the post. You do not revoke it in any cryptographic sense. You move CURRENT to a known-good generation, you delete the bad objects from the bucket, and you garbage-collect the generation from every host's disk. That is not revocation, it is cleanup with good intentions and a change-management ticket.

The reason it does not fully work is that the artifact has already done its job. A host that installed the bad generation yesterday has it on local disk and a verified signature that still checks out — the signature was never wrong, the template was. Any sandbox already restored from it is running. And because forks are copy-on-write descendants of a snapshot, the lineage outlives the file: you can delete every byte of a template from every bucket and every disk, and the memory pages it contributed are still resident in running guests that were forked from it. There is no operation in the system that reaches into a live VM and removes an ancestor's pages.

What you can build is honest and mostly bookkeeping. A denylist of generation ids the host checks after signature verification and before install, so a refetch cannot reinstall a burned generation. A monotonic generation counter, so a signed older manifest cannot pass the freshness check — this is the piece most people skip and it is the one that turns "we moved the pointer" into something an attacker cannot undo by serving an old object. Short-lived signing certificates, so the window in which a stolen identity can sign anything at all is measured in minutes. And an inventory that answers "which live sandboxes descend from generation X", because the actual remediation for a bad template is draining and recreating the guests, and you cannot drain what you cannot enumerate.

Four rungs, and what each one actually stops

Softest to hardest. Each rung is strictly more work than the one above it, and the honest advice is that rung two is where most of the value is and rung three is where most of the remaining value is; rung four is for the fleet whose template pipeline is itself a target.

• Unsigned bucket sync — Stops: nothing, though it feels fine because the sync exits zero. A truncated download, a mid-flip read, a modified object and a correct object are all indistinguishable to the host. Detects a failure only when a guest misbehaves later, which is the worst possible moment and the hardest to attribute. Cost: zero. Very popular.
• Checksum manifest, verified at install — Stops: truncation, partial syncs, silent corruption, storage bit-rot, and the mixed-generation install where half your files are from last week. This is the single biggest jump in the list and it needs no keys, no CI changes and no cryptographic ceremony. Does not stop: anyone who can write to the bucket, because they can write a matching manifest too.
• Signed manifest, verified on the host — Stops: a tampered artifact from anyone without the signing key, including someone with full write access to your storage. Adds rollback protection if — and only if — you check that the signed manifest names the generation you asked for. Does not stop: a compromised bake host, or a bad template that was faithfully built from bad inputs. Cost: a key-management problem you now own forever.
• Signed manifest plus provenance attestation — Stops: the same attacks as above, and additionally answers "which Dockerfile, which commit, which builder, which base image" months later without archaeology, which is what turns a suspicious template into a bounded investigation. Enables policy: refuse any generation not built from main by the one workflow allowed to bake. Does not stop: a malicious change that passed review and was hermetically, verifiably, provably baked into the artifact exactly as written.

The summary

The inventory is the part to get right, and it is bigger than the rootfs: the guest kernel, the template disk, the snapshot's disk and vCPU state, the snapshot's memory, the streaming sidecars that decide what gets fetched, the identity metadata that freezes the guest's IP and MAC, and the pointer that says which generation is live. On a snapshot-restore platform the memory image is the one that matters most and the one most likely to be left out, because it is big and hashing it is inconvenient. Inconvenience is not a threat model.

The mechanism is unglamorous: one SHA256 manifest per generation covering every file, one signature over the manifest, published under an immutable per-generation prefix with a separate pointer, in-toto provenance alongside it recording how it was built. cosign's blob subcommands do the signing without needing a registry, keyless in CI and key-based for hosts that should not depend on the public internet to boot.

And the placement is the whole point. Verify on the host, in the sync path, into a staging directory, before an atomic rename makes the generation visible to the boot path — and refuse on mismatch, which is survivable precisely because the previous generation keeps serving. Be honest that the signing key is a single point of total compromise and shrink its surface accordingly, and be honest that revocation is mostly moving a pointer, garbage-collecting, and knowing which running guests descend from the generation you no longer trust. That is a smaller claim than "our supply chain is secure". It is also one you can actually defend on a call.