On-prem without regrets: shipping your product as a microVM

Sooner or later a sales call ends with a sentence that rearranges your architecture: "we love it, and it can never touch the internet." The speaker is a bank, a hospital group, a defence prime, a clearing house, or a government department. They are not negotiating. They also have the budget, which is the whole problem — the customers most willing to pay enterprise prices are exactly the ones least willing to let data leave their building.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, and I keep watching SaaS teams discover this the same way. They sell the deal, then find out that "enterprise tier" is not a pricing page — it is the moment a cloud company becomes a software distribution company again, a discipline most of the industry deliberately abandoned around 2012 and has been quietly relieved about ever since.

The enterprise tier is where you rediscover shipping software

Strip away the deal romance and the requirement is brutally concrete. You must produce an artefact that runs in a datacentre you will never see, on hardware you did not choose, operated by people who do not work for you, patched on a schedule you do not control, upgraded through a change advisory board, and supported without shell access. Every assumption your platform team lives on — we can deploy on Friday, we can read the logs, we can roll back in ninety seconds, there is exactly one version in production — is gone.

The naive answer is "we already have containers, we'll give them a Helm chart." That is a reasonable first instinct and it is where most of these projects go wrong, so it deserves a proper argument rather than a sneer. But before the artefact question, there is a trust question, and it is the more interesting half.

The two-sided trust problem

Most isolation writing on this site is about protecting tenants from each other inside your infrastructure. On-prem inverts it. Inside the customer's network, you are the untrusted party — a vendor binary with a listening socket, sitting on a segment that also carries patient records or trade orders. And from your side, the customer is the untrusted party too: they can read your binary, patch it, run it on a host you have never tested, and then open a P1 about the results.

• What the customer must believe — Your software cannot roam their network. It cannot exfiltrate anything, deliberately or by a default that someone forgot to switch off. It is not a lateral pivot into a flat internal VLAN if it gets popped. They can inspect what it is made of, they can bound what it can reach, and they can remove it completely and know that it is gone.
• What you must be able to do — Answer "what is customer X running" with a hash rather than a guess. Reproduce their environment on your own bench. Support a failure without SSH, without a screen share at 2am with an operator who has never used a terminal, and without inheriting responsibility for their kernel, their storage layer, or their proxy.
• The awkward overlap — Both sides want the artefact to be opaque enough to be safe and transparent enough to be trusted. Those are not actually in conflict, but the answer has to be structural, not contractual. Nobody's security team has ever been calmed by a paragraph in an MSA.

A microVM image answers both halves with the same object. It is a sealed, bit-exact artefact: a kernel you built, a rootfs you built, and a machine configuration, all addressed by hash. It brings its own kernel, so it does not care what the host runs and the host does not have to trust it with a shared one. And its network posture is declarative — a virtual NIC, a tap device, and a firewall rule set that is a file a human can read in a meeting. "Here is the complete list of things this appliance can talk to" is a sentence you can actually say, and then prove.

A container is a polite suggestion to the kernel. A microVM is a smaller kernel that isn't theirs.

Why Docker image + Helm chart is the weaker artefact

A container image is not a machine. It is a tarball of userspace plus a promise about a kernel it does not ship. On your own clusters that promise is cheap, because you own the kernel it refers to. On a customer's cluster you have signed up to a support matrix that expands with every deal, and here is what expands it:

• The kernel. Their node kernel, their vendor patch set, their cgroup version, their seccomp defaults, and whether AppArmor or SELinux is enforcing this week.
• The CNI. Calico, Cilium, an OpenShift SDN, or something a platform team wrote in 2021. NetworkPolicy semantics and enforcement points differ, so your network posture is now their configuration, not your artefact.
• Admission control. Mutating webhooks that inject sidecars, proxies, or a security agent into your pod. Pod Security Admission at restricted, which quietly removes capabilities you needed. Gatekeeper policies that reject your spec at 4pm on a Friday.
• The registry. An air-gapped mirror that rewrites image references, and in doing so breaks digest pinning — the one mechanism that told you what they were running.
• Storage and scheduling. A CSI driver with different fsync behaviour, a StorageClass that does not do what your database assumed, a node with taints that leave one of your pods Pending forever.
• Their Kubernetes distribution and version, which your on-call engineer now has to learn instead of learning your product.

The compounding effect is that every customer becomes a snowflake, and your support burden multiplies rather than adds. "Works on my cluster" is not a support tier. Meanwhile the trust half fails too: from the customer's point of view, your chart is a set of processes on their nodes, sharing their kernel, next to a service account token, reachable across whatever their CNI actually enforces as opposed to what the YAML claims.

None of this makes Kubernetes bad. It is an excellent runtime when you operate the cluster. Delivery is a different job, and in delivery the property you want is that the artefact contains its own answers.

What a sealed artefact actually contains

One directory. A kernel, a rootfs image, a machine configuration, an SBOM, a release descriptor, a manifest of hashes, and a signature over that manifest. Nothing in it resolves anything from a network at run time.

#!/usr/bin/env bash
# build-appliance.sh - produce one sealed, signed, on-prem artefact.
set -euo pipefail

VERSION="${1:?usage: build-appliance.sh <version>}"
OUT="dist/acme-appliance-${VERSION}"
mkdir -p "$OUT"

# Reproducibility inputs: a pinned base digest, a pinned package snapshot,
# and a build clock derived from the commit rather than from "now".
export SOURCE_DATE_EPOCH="$(git log -1 --format=%ct)"

docker buildx build \
  --platform linux/amd64 \
  --build-arg VERSION="$VERSION" \
  --build-arg SOURCE_DATE_EPOCH="$SOURCE_DATE_EPOCH" \
  --output "type=tar,dest=$OUT/rootfs.tar" \
  -f appliance/Dockerfile .

# Flatten to a bootable ext4 image. No registry, no CNI, no admission
# controller, no node config to argue about: the customer runs a file.
./scripts/tar2ext4.sh "$OUT/rootfs.tar" "$OUT/rootfs.ext4"
cp kernel/vmlinux-6.1-pinned  "$OUT/vmlinux"
cp appliance/machine-config.json "$OUT/machine-config.json"

# The SBOM covers the guest kernel too, which a container SBOM never does.
syft "$OUT/rootfs.ext4" -o spdx-json > "$OUT/sbom.spdx.json"

jq -n --arg v "$VERSION" --arg c "$(git rev-parse HEAD)" \
      --arg t "$(date -u -d "@$SOURCE_DATE_EPOCH" +%Y-%m-%dT%H:%M:%SZ)" \
  '{product:"acme-appliance",version:$v,commit:$c,built_at:$t,kernel:"6.1-pinned"}' \
  > "$OUT/release.json"

# Address every byte the customer will run, then sign the manifest.
# Signing the manifest rather than the tarball keeps the signature valid
# across repackaging: ISO, USB stick, internal artifact store, courier.
( cd "$OUT" && sha256sum vmlinux rootfs.ext4 machine-config.json \
    sbom.spdx.json release.json > manifest.sha256 )

cosign sign-blob --key "$COSIGN_KEY" \
  --output-signature "$OUT/manifest.sha256.sig" "$OUT/manifest.sha256"

tar -C dist -cf "dist/acme-appliance-${VERSION}.tar" "acme-appliance-${VERSION}"
sha256sum "dist/acme-appliance-${VERSION}.tar"

Two details that look pedantic and are not. First, the kernel is inside the manifest, because on-prem the kernel is part of your product and part of your CVE surface. Second, the signature covers the manifest, not the outer tarball — customers will repackage your artefact into their internal artifact store, their approved ISO process, or a USB stick carried by a person with a badge, and every one of those repackagings would invalidate a signature over the container.

Provenance, or: what version is customer X actually running?

In a container-and-chart world this question has no reliable answer. The tag was mutable. The mirror rewrote the reference. A webhook injected a sidecar. Someone patched the deployment to pin an older image during an incident in March and never told anyone. Your support engineer is debugging a system that does not correspond to any build you ever produced.

With a sealed image the answer is a hash, and the customer can produce it themselves in one command. That single property changes support economics more than any tooling you can buy. It also gives the auditors what they are going to ask for regardless: an SBOM in SPDX or CycloneDX, a signed attestation binding that SBOM to the exact image digest, and a documented chain from a commit to the bytes on the machine. If you can hand a security reviewer a manifest, a signature, an SBOM covering kernel and userspace, and a build script they can read, you have skipped roughly six weeks of questionnaire.

Air-gap mechanics: no phone-home, offline licences, sneakernet updates

Nothing calls home, and you prove it rather than promise it

License phone-home is an instant deal-killer, and telemetry that is off "by default" is a deal-killer as soon as someone greps your binary for a URL and finds one. Offline licence validation is the standard answer: a small signed blob containing customer identity, entitlement, seat or node count, and an expiry, verified in-guest with a public key baked into the image. It is not DRM and it will not survive a determined adversary with a debugger. It is not supposed to. It is supposed to be auditable, to work with no DNS, and to give both sides an unambiguous statement of what was licensed.

The stronger move is to have the appliance assert its own posture at boot and expose the result on a health endpoint the customer can scrape. If your telemetry endpoint is reachable from inside their air-gapped segment, that is a finding — and it is much better for you to report it than for their next audit to.

// cmd/preflight/main.go - runs inside the guest at boot, before the product.
// It proves three things the customer's security team asked about, and one
// thing our own support team will need six months from now.
package main

import (
	"crypto/ed25519"
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"fmt"
	"io"
	"net"
	"net/http"
	"os"
	"strings"
	"time"
)

// Baked in at build time. The private half never leaves the release box,
// so handing this binary to a customer leaks nothing.
var vendorPub ed25519.PublicKey

type license struct {
	Customer string    `json:"customer"`
	Seats    int       `json:"seats"`
	NotAfter time.Time `json:"not_after"`
	Sig      string    `json:"sig"` // ed25519 over the canonical body
}

// 1. The guest is bit-for-bit what we shipped.
func verifyManifest(dir string) error {
	raw, err := os.ReadFile(dir + "/manifest.sha256")
	if err != nil {
		return err
	}
	for _, line := range strings.Split(strings.TrimSpace(string(raw)), "\n") {
		f := strings.Fields(line)
		if len(f) != 2 {
			continue
		}
		want, name := f[0], f[1]
		got, err := sha256File(dir + "/" + name)
		if err != nil {
			return err
		}
		if got != want {
			return fmt.Errorf("drift: %s is %s, we shipped %s", name, got[:12], want[:12])
		}
	}
	return nil
}

// 2. The licence validates with no DNS and no call to us.
func verifyLicense(path string) (*license, error) {
	raw, err := os.ReadFile(path)
	if err != nil {
		return nil, err
	}
	var l license
	if err := json.Unmarshal(raw, &l); err != nil {
		return nil, err
	}
	sig, err := hex.DecodeString(l.Sig)
	if err != nil {
		return nil, fmt.Errorf("licence signature malformed: %w", err)
	}
	body, _ := json.Marshal(struct {
		Customer string    `json:"customer"`
		Seats    int       `json:"seats"`
		NotAfter time.Time `json:"not_after"`
	}{l.Customer, l.Seats, l.NotAfter})
	if !ed25519.Verify(vendorPub, body, sig) {
		return nil, fmt.Errorf("licence signature invalid")
	}
	if time.Now().After(l.NotAfter) {
		return nil, fmt.Errorf("licence expired %s", l.NotAfter.Format("2006-01-02"))
	}
	return &l, nil
}

// 3. Nothing in here can phone home. Assert it, do not promise it.
func assertNoEgress(hosts []string) error {
	for _, h := range hosts {
		if c, err := net.DialTimeout("tcp", h, 2*time.Second); err == nil {
			c.Close()
			return fmt.Errorf("egress reachable: %s (air-gap posture violated)", h)
		}
	}
	return nil
}

func main() {
	const dir = "/opt/appliance"
	checks, failed := map[string]string{}, 0

	run := func(name string, err error) {
		if err != nil {
			checks[name] = "FAIL: " + err.Error()
			failed++
			return
		}
		checks[name] = "ok"
	}
	run("manifest", verifyManifest(dir))
	_, lerr := verifyLicense(dir + "/license.json")
	run("licence", lerr)
	run("phone_home", assertNoEgress([]string{
		"telemetry.acme.example:443",
		"licence.acme.example:443",
	}))

	rel, _ := os.ReadFile(dir + "/release.json")
	http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "application/json")
		if failed > 0 {
			w.WriteHeader(http.StatusServiceUnavailable)
		}
		_ = json.NewEncoder(w).Encode(map[string]any{
			"release": json.RawMessage(rel),
			"checks":  checks,
		})
	})
	if failed > 0 {
		fmt.Fprintf(os.Stderr, "preflight failed: %v\n", checks)
	}
	log(http.ListenAndServe("0.0.0.0:8443", nil))
}

func sha256File(p string) (string, error) {
	f, err := os.Open(p)
	if err != nil {
		return "", err
	}
	defer f.Close()
	h := sha256.New()
	if _, err := io.Copy(h, f); err != nil {
		return "", err
	}
	return hex.EncodeToString(h.Sum(nil)), nil
}

func log(err error) { fmt.Fprintln(os.Stderr, err) }

Updates when the machine cannot reach you

The update arrives as the same sealed artefact, transported by whatever their process allows: an internal artifact store, a one-way data diode, or a person carrying media through a door with a card reader. The critical design decision is that an update is an atomic image swap, not an in-place package upgrade.

1. Verify the signature over the manifest offline, with a public key the customer already has from the original install.
2. Stage the new kernel and rootfs beside the running ones. Nothing is replaced yet.
3. Run any data migration against a clone or snapshot first, and abort the whole update if it fails there. Storage migrations are the one part of this that genuinely is not atomic, so they get the paranoid treatment: forward-only, idempotent, and preceded by a backup the operator has confirmed.
4. Flip a single boot pointer to the new image and restart the VM. This is the actual cutover, and it is one rename.
5. Keep N-1 on disk. Rollback is flipping the pointer back, which an operator can do at 3am from a runbook without a call to you.
6. Garbage-collect old images only after the customer confirms, on their timeline, not yours.

Debugging with no SSH, ever

You will not get access. Not "probably not" — the security control that made them buy an on-prem appliance is precisely the one that forbids a vendor tunnel. So the support bundle is a day-one feature, not something you bolt on after the first sev1. It should be generated inside the guest, written to a path the customer controls, and shipped only when they decide to ship it.

• Identity — release.json, the manifest verification result, and the licence status. If the manifest does not verify, that is the first line of your triage and it saves a week.
• Kernel and boot — dmesg, boot args, and the machine configuration. Because the guest kernel is yours, this is a kernel you built and can reason about, rather than a stranger's node.
• Runtime — the guest journal, your service logs with structured redaction, and resource counters over time so you can tell "slow" from "out of memory at 04:12".
• Network posture — the effective firewall rules and any egress denial log. A denied connection you did not expect is the single highest-signal artefact in the bundle.
• Configuration — with every secret replaced by a hash, so you can prove two installs differ without ever holding their credentials.
• A bundle manifest — a plain listing of everything included, so the customer's reviewer can approve the send in minutes instead of escalating it.

Make the collector support a dry run that prints exactly what it would gather, and an extra-redaction mode for the customers who need it. A support bundle that a security team refuses to release is worth nothing, and you find that out during the incident.

The dev loop: test the artefact you are actually shipping

The failure mode here is testing a rebuilt approximation of the artefact rather than the artefact. Boot the real bytes in a disposable sandbox, run the customer's install path against them, and fork the healthy install once per upgrade path you promise to support.

from pandastack import Sandbox

VERSION = "4.7.0"
ART = f"dist/acme-appliance-{VERSION}"

sbx = Sandbox.create(template="base", ttl_seconds=1800)

# Upload the exact bytes the customer receives - not a fresh build.
for name in ("vmlinux", "rootfs.ext4", "machine-config.json",
             "manifest.sha256", "manifest.sha256.sig", "license.json"):
    sbx.filesystem.write(f"/opt/appliance/{name}", open(f"{ART}/{name}", "rb").read())

# 1. Signature check, run exactly as the customer's operator would run it.
r = sbx.exec(
    "cosign verify-blob --key /opt/appliance/vendor.pub "
    "--signature /opt/appliance/manifest.sha256.sig "
    "/opt/appliance/manifest.sha256",
    timeout_seconds=60,
)
assert r.exit_code == 0, r.stderr

# 2. Boot the appliance with egress cut, which is the only honest way to
#    test an air-gapped install, and read the guest's own verdict.
sbx.exec("/opt/appliance/run-appliance.sh --offline --wait-healthy 120",
         timeout_seconds=300)
health = sbx.exec("curl -sf http://10.0.0.2:8443/healthz", timeout_seconds=30)
print(health.stdout)
assert '"phone_home": "ok"' in health.stdout

# 3. Snapshot the verified 4.7.0 install once, then fork it per upgrade
#    path. Testing 4.7.0 -> 4.8.1 should not mean rebuilding 4.7.0 again.
sbx.snapshot()
for target in ("4.8.0", "4.8.1-hotfix"):
    child = sbx.fork()
    child.filesystem.write(
        "/opt/appliance/incoming.tar",
        open(f"dist/acme-appliance-{target}.tar", "rb").read(),
    )
    up = child.exec("/opt/appliance/apply-update.sh /opt/appliance/incoming.tar",
                    timeout_seconds=900)
    bundle = child.exec("/opt/appliance/support-bundle.sh --dry-run", timeout_seconds=60)
    print(target, up.exit_code, bundle.stdout.splitlines()[-1])
    child.kill()

sbx.kill()

A same-host fork lands in 400-750ms, so a matrix of upgrade-path machines is cheap enough to run on every release candidate rather than once a quarter. At $0.054 per active vCPU-hour and $0.0162 per GiB-hour billed per second, the matrix costs roughly what the minutes cost — the VMs that are not running are not costing anything, which is what makes "one VM per upgrade path per RC" a sane policy instead of a budget conversation.

Four ways to ship into a customer's datacentre

• Docker image + Helm chart — Trust posture: your processes share the customer's kernel and sit on their CNI, so isolation is their configuration rather than your artefact. Support burden: highest; every distribution, CNI, admission policy, and registry mirror is a variable, and every customer becomes a snowflake. Update model: rolling, partially applied by definition, with mutable tags and mirrors obscuring what is actually running. Customer acceptance: easy for a platform-mature customer who already runs Kubernetes, genuinely hard for a regulated one whose reviewers ask what kernel it shares.
• OVA-style full VM — Trust posture: strong, and the format their virtualisation team already trusts; the boundary is a hypervisor they operate. Support burden: moderate, but images are large and slow to move through an air gap, and boot is a full general-purpose OS with a full general-purpose CVE surface. Update model: usually replace-the-VM, which is atomic and fine; the friction is that multi-gigabyte artefacts make patch cadence worse. Customer acceptance: often the highest of the four, because "here is a VM" is a request their change process has answered a thousand times.
• MicroVM image — Trust posture: hardware-virtualised boundary with your own guest kernel and a declarative, auditable network posture; small enough that the attack surface is describable in a paragraph. Support burden: lowest per-customer, because the artefact contains its own kernel and userspace and does not negotiate with the host; the cost moves to you maintaining that kernel. Update model: atomic image swap with N-1 rollback, verified offline by signature. Customer acceptance: good once you have cleared the KVM-capable host requirement, which is the real gate — see the caveats below.
• SaaS-only (refuse the deal) — Trust posture: irrelevant, since their data never arrives. Support burden: lowest possible; one version, your infrastructure, deploy on Friday. Update model: continuous, invisible, entirely yours. Customer acceptance: zero with the regulated buyers, which is the point — this is the option you are giving up when you decide the enterprise tier is worth building, and it is a legitimate strategic answer to say no.

The usual honest caveat applies to the first two rows: virtualisation platforms, Kubernetes distributions, and packaging tools change fast, so check the current documentation for whatever your customer actually runs before you promise anything based on a blog post, including this one.

The parts that are genuinely worse

Three costs, and you should name all of them in the first technical call rather than discovering them in week six.

• Somebody on their side needs a KVM-capable Linux host. Bare metal is ideal; a Linux VM inside their existing virtualisation estate works only if nested virtualisation is enabled, which is frequently off by policy and sometimes off by licensing. This is a deal-shaped question, not an implementation detail — ask it early, in writing, of someone who administers the hypervisor rather than someone who buys the software.
• The guest kernel is now your responsibility. You choose the version, you track its CVEs, you decide what to backport, and you answer "does CVE-XXXX-YYYY affect your appliance?" for customers who will not accept "we don't think so." The mitigating half is that a minimal guest lets you answer most of those with "we do not ship that module" — but only if you actually know what you ship, which is what the SBOM is for.
• Air-gapped means slow patching, and that is a real security cost you should say out loud. You have traded exposure-to-the-internet for exposure-to-time: the interval between your fix and their install is measured in weeks or quarters, gated by a change board. Compensate by narrowing the guest, publishing an advisory feed they can pull on their own schedule, and making updates boring enough that applying one is not itself a risk decision.

There is also a commercial cost that engineering cannot design away. An enterprise tier funds a release train, an SBOM and signing pipeline, a support-bundle triage rotation, and a version-support policy with real end dates. If the pricing does not cover those, the tier will quietly consume the roadmap instead, one bespoke customer at a time.

What you get in return is that the hardest sentence in the enterprise sales cycle — "what exactly will be running on our network, and what can it reach?" — has a boring, checkable answer. A kernel and a rootfs, addressed by hash, signed, with an SBOM covering both, on a virtual NIC whose entire reachable set is a file. The customer's reviewer can read it. Your support engineer can reproduce it. Nobody has to trust anybody, which is the only trust model that has ever survived contact with a bank.