Shrinking PCI DSS Scope With Per-Transaction microVMs

Everyone who has been through a PCI DSS assessment remembers the moment the auditor's idea of “the system” turned out to be much larger than theirs. You walked in expecting to discuss the payment service. You walked out with a spreadsheet that also listed the CI runners, the log aggregator, the bastion host, the Kubernetes control plane, a Redis instance nobody has owned since a reorg, and the laptop of a contractor who left in March. That spreadsheet is scope, and scope is the single biggest determinant of what PCI DSS costs you.

I'm Ajay; I built PandaStack, an open-source Firecracker microVM platform, and teams building payment products keep asking whether a stronger isolation boundary shrinks what they have to defend. The honest answer has two halves. It can genuinely reduce the number of systems in scope and how long cardholder data exists on any of them, and — more usefully — it turns a segmentation argument that runs twenty pages into one that fits on a diagram. It cannot make a system that touches a primary account number stop touching one. Vendor material here is remarkably bad at drawing that line.

What “scope” actually means, and why it spreads

The cardholder data environment — the CDE — is the people, processes and technologies that store, process, or transmit account data. If scope stopped there this would be a short post. Two further categories come with it, and they are the ones that eat organizations alive.

• Systems that store, process, or transmit account data — the payment service, the queue the payload passes through, the worker that decrypts it, and the log line somebody added that echoes the request body.
• Connected-to systems — anything with connectivity to the CDE, whether or not it ever sees a card number: the monitoring agent, the shared broker, internal DNS, the service mesh control plane. Connectivity is the criterion, not intent.
• Security-impacting systems — anything that can affect the CDE's security: CI/CD, because it decides what code runs there; your identity provider, because it decides who gets in; your image build pipeline, secrets manager, config management, and admin jump host. None of these ever see a PAN, which is exactly why they get missed.

Scope propagates along connectivity and along administrative control, and the second path is what turns a tidy diagram into a spreadsheet. One Jenkins deploying both the payment service and the marketing site makes that Jenkins — and its plugins, credentials, and node — security-impacting. Every system in any bucket inherits the requirement set: hardening evidence, patch timelines, change records, integrity monitoring, logging with retention, access control with MFA, vulnerability scans, penetration testing. Multiply by system count; that product is your assessment budget.

Segmentation is not a product; it is an argument you defend

PCI DSS does not require segmentation. It offers it: don't segment and your whole network is in scope; segment and the excluded portion can stay excluded — provided you can show the segmentation actually works. That proviso does enormous work. Segmentation is a claim about your own infrastructure that you substantiate with data flow diagrams, configuration evidence, and penetration testing aimed specifically at the segmentation controls, at a cadence that is more frequent for service providers than for other entities. Verify the current requirement numbering and frequency against the version you are assessed against; these details have moved between versions and will move again.

Internalize what the assessor is asking. Not “is this secure?”, which is unanswerable, and not “do you trust your platform team?”, which is irrelevant. It is: name the control that prevents access from crossing this line, show me its configuration, show me the test that proved it holds. Separate hardware answers that in a sentence. So does a separate cloud account with no peering, or a VLAN whose firewall ruleset you can dump to a text file. Other controls answer it in twenty pages of architecture narrative, and twenty pages is where assessments go to die.

Why a shared kernel makes that conversation hard

A container is a process with namespaces, cgroups, seccomp filters and usually a mandatory access control profile, layered over one host kernel. On a given node, the workload handling card data and the workload resizing marketing images issue syscalls into the same kernel. That isolation is real and useful. It is also the hard kind of control to evidence: enforcement is a software policy inside a shared runtime rather than a boundary you can point at.

Watch what happens when you write the argument down. You enumerate node pools and the taints keeping CDE pods on their own nodes — and explain what stops a future deployment from tolerating them. You explain your NetworkPolicy, and whether your CNI enforces it in the mode you actually run, and what covers traffic leaving through the shared ingress controller. Then automounted service account tokens, the CSI driver, the log agent reading every pod's stdout, the metrics sidecar with a Unix socket, the admission controller, the kubelet. Each is a reasonable control; together they are a long argument that any single change silently invalidates.

The container escape that should worry you at assessment time is not a kernel CVE. It is a well-meaning platform engineer adding a `hostPath` mount to a DaemonSet on a Thursday afternoon, six weeks after the diagram was signed, to debug something unrelated. The PCI Security Standards Council's cloud and virtualization guidance has long carried the same theme — mixed-trust workloads on shared infrastructure pull the shared components into scope — so read the current version rather than trusting my summary. The technical half of the argument is in /blog/why-docker-is-not-a-sandbox.

What a hardware-virtualized boundary changes

A Firecracker microVM is a different category of thing. The guest has its own kernel, booted from your image. Its memory is isolated by the CPU's second-level address translation under KVM, not by a namespace. Its devices are a small set of virtio devices implemented by a minimal userspace VMM running unprivileged under its own seccomp filter. Its network is a dedicated namespace and TAP with host-side rules the guest cannot see, let alone edit. Crossing to the host means defeating the hypervisor, not finding a gap in a namespace config — it is the model AWS uses to run untrusted code from unrelated customers on shared hardware.

But the property that matters for scope is not “more secure” in the abstract. It is that the boundary is small, describable, and enumerable. You can list the interface in a paragraph: these virtio devices, this vsock or SSH channel, this TAP with this host-side allowlist, this snapshot image. That is a diagram a QSA can read and a target a penetration tester can be pointed at. Compare it to “here are the fourteen Kubernetes objects whose combined configuration constitutes our boundary.” Same intent; wildly different cost to defend.

The shape: a worker that exists for one transaction

A card payload arrives at your edge. Instead of routing it to a long-lived worker in a shared cluster, you create a microVM for that single transaction, hand it the payload and one short-lived credential scoped to one acquirer endpoint, let it authorize, take back a token and a decision, and destroy the machine. Per tenant works the same way for batch shapes like settlement files — the unit is whatever gives a clean answer to “whose data was in this machine?”

Four properties fall out. There is no persistent storage in the guest, so “we do not store account data” becomes a statement about the architecture rather than about everyone remembering to delete things. The guest holds no credential reaching anything else you own, so a compromised parser gets one transaction and one destination instead of an enumeration primitive. Lifetime is bounded by a TTL a reaper enforces, so retention is a scheduler property. And one-job-one-machine makes attribution exact.

The historical objection is cost: a VM per transaction sounds like trading a millisecond of container dispatch for half a minute of provisioning. Snapshot-restore removes that. On PandaStack nothing is cold-booted in the request path — every create restores a pre-baked snapshot at p50 179ms and p99 203ms, the restore step itself around 49ms, with the roughly three-second cold boot paid once at bake time. Each agent also pre-allocates 16,384 /30 subnets, so your ceiling is host CPU and memory rather than network slots.

The guest-side fences come first. Card data should not arrive until the machine has proven it is the machine you think it is.

#!/bin/bash
# Runs INSIDE the per-transaction guest, before any card data arrives.
# Every line here exists because of a specific way PANs escape a worker.
set -euo pipefail

# 1. No core dumps. A segfault in an ISO 8583 codec or a JSON parser would
#    otherwise write the process address space -- including the decrypted
#    PAN it was holding -- to disk, and on most distributions core_pattern
#    pipes that image straight to a crash handler that helpfully forwards it.
ulimit -c 0
echo '|/bin/false' > /proc/sys/kernel/core_pattern

# 2. No swap. Anonymous memory holding cardholder data must not be paged to
#    a device that outlives the process that owned it.
swapoff -a 2>/dev/null || true

# 3. Scratch space is a small tmpfs, noexec, and dies with the guest. There
#    is no persistent volume in this machine, which is how "we do not store
#    cardholder data" becomes a statement about the architecture rather than
#    a statement about everyone remembering to delete things.
mkdir -p /work/scratch
mount -t tmpfs -o size=32m,noexec,nosuid,nodev tmpfs /work/scratch

# 4. Assert the egress fence FROM THE INSIDE before touching data. The host
#    enforces it; this proves it. A job that finds the open internet fails
#    closed instead of quietly processing a live transaction on a guest
#    somebody accidentally attached to the default route.
probe() { curl -sS -o /dev/null --max-time 4 "$1" 2>/dev/null; }

if probe https://connectivity-check.example.net/generate_204; then
  echo "FATAL: general egress reachable; refusing to process" >&2
  exit 90
fi
if ! probe "$ACQUIRER_URL/healthz"; then
  echo "FATAL: acquirer endpoint unreachable; refusing to process" >&2
  exit 91
fi

# 5. Now, and only now, the actual work. One payload in, one verdict out.
exec /usr/local/bin/authorize --in /work/in.json --out /work/scratch/out.json

Then the orchestration. The sandbox is pure: one payload in, one verdict out, no credentials, no route, no persistence. The trusted orchestrator — which does hold the database handle and the key material — takes the token and writes the audit row.

from pandastack import Sandbox
import json

FENCES = open("fences.sh").read()   # the script above


def authorize_once(txn_id: str, tenant: str, payload: bytes) -> dict:
    """Run ONE authorization in a machine that exists for one authorization.

    Note what is deliberately absent from this guest: your primary database
    DSN, your object-store credential, your KMS admin key, any other
    cardholder's data, and any route to the internet other than the acquirer.
    A memory-corruption bug in the codec here owns a throwaway machine
    holding one transaction, with one place to send it, for ninety seconds.
    """
    sbx = Sandbox.create(
        template="base",
        ttl_seconds=120,          # a retention control, not just a safety net
        metadata={
            # NON-CARD-DATA ONLY. Metadata is queryable, comes back from list
            # APIs, and is printed in operational logs that live outside the
            # CDE. A transaction id and a tenant slug belong here. A PAN, a
            # cardholder name, an expiry, or anything resembling track data
            # does not -- not once, not "just for debugging", not truncated.
            "txn": txn_id,
            "tenant": tenant,
            "class": "cde",
        },
    )
    try:
        sbx.filesystem.write("/work/fences.sh", FENCES)
        sbx.filesystem.write("/work/in.json", payload)

        run = sbx.exec("bash /work/fences.sh", timeout_seconds=60)

        # FAIL CLOSED. Exit 90/91 are the egress assertions; anything else is
        # a crash, a timeout, or a declined authorization. A declined
        # transaction is a support ticket. A leaked PAN is a letter you have
        # to write to your acquirer, and those are much longer.
        if run.exit_code != 0:
            raise AuthorizationFailed(txn_id, run.exit_code, run.stderr[-1000:])

        # ONLY the token and the decision cross the boundary. The PAN was
        # decrypted inside this guest and stops existing along with it.
        out = json.loads(sbx.filesystem.read("/work/scratch/out.json"))
        return {
            "decision": out["decision"],        # approved | declined
            "token": out["token"],              # the thing you may keep
            "acquirer_ref": out["acquirer_ref"],
            "guest_id": sbx.id,                 # for the audit row, below
        }
    finally:
        # Guest memory, tmpfs scratch, the decrypted payload, and any
        # half-written buffer cease to exist as one event. There is no
        # cleanup routine to get right on the unhappy path, which matters
        # because the unhappy path is where cleanup routines are not run.
        sbx.kill()

The TypeScript equivalent, in its batch shape, with the audit write that makes the whole thing defensible.

import { Sandbox } from "@pandastack/sdk";

// Settlement is the batch shape of the same idea: one guest per tenant per
// file. "Tenant A's settlement file influenced tenant B's" stops being a
// scenario you rule out by code review and starts being one you rule out by
// there not being a shared machine for it to happen on.
const TEMPLATE = "base";
const TEMPLATE_GENERATION = process.env.CDE_TEMPLATE_GENERATION!; // pins code

export async function settleTenantBatch(
  tenant: string,
  batchId: string,
  file: Buffer,
) {
  const sbx = await Sandbox.create({
    template: TEMPLATE,
    ttlSeconds: 900,
    metadata: { tenant, batch: batchId, class: "cde" },
  });

  try {
    await sbx.filesystem.write("/work/fences.sh", FENCES);
    await sbx.filesystem.write("/work/batch.csv", file);

    const run = await sbx.exec("bash /work/fences.sh", { timeoutSeconds: 840 });
    if (run.exitCode !== 0) {
      throw new Error(
        `batch ${batchId} failed closed (${run.exitCode}): ` +
          run.stderr.slice(-1000),
      );
    }

    const raw = await sbx.filesystem.read("/work/scratch/summary.json");
    const summary = JSON.parse(raw.toString());

    // The audit entry names the MACHINE and the batch. Never a PAN, never a
    // truncated PAN "for support", never a hash of one -- sixteen digits
    // with a known prefix and a Luhn check is not a keyspace, it is a
    // weekend project.
    await audit.write({
      event: "settlement.batch.processed",
      tenant,
      batchId,
      guestId: sbx.id,
      template: TEMPLATE,
      templateGeneration: TEMPLATE_GENERATION,
      egressPolicy: "allow:acquirer-only",
      recordCount: summary.count,
      at: new Date().toISOString(),
    });

    return summary;
  } finally {
    await sbx.kill();
  }
}

Logging, clocks, and the audit trail

PCI DSS spends a whole requirement family on audit logging, and the substance is unglamorous: record access to account data and administrative actions, capture who, what, when, where and outcome, protect logs from modification, synchronize clocks to a trusted time source, review them, and retain them for a period commonly cited as twelve months with the most recent three immediately available. Verify the specifics against the current standard. The per-guest model makes “who and what” nearly free: one job maps to one machine, so an audit row names a guest id, a template generation and an egress policy, and never has to name a card.

-- One row per guest that ever held cardholder data. This is the table a QSA
-- reads instead of grepping an interleaved worker log for an identifier you
-- should not have been logging in the first place.
CREATE TABLE cde_job_audit (
  guest_id       uuid PRIMARY KEY,        -- the microVM, one job, one life
  txn_id         text        NOT NULL,    -- your reference, not the card's
  tenant         text        NOT NULL,
  template       text        NOT NULL,
  template_gen   text        NOT NULL,    -- which baked image ran: pins code
  egress_policy  text        NOT NULL,    -- 'allow:acquirer-only'
  created_at     timestamptz NOT NULL,
  destroyed_at   timestamptz,             -- still NULL past TTL+grace = page
  outcome        text        NOT NULL     -- approved|declined|failed_closed
);

CREATE INDEX ON cde_job_audit (tenant, created_at DESC);
CREATE INDEX ON cde_job_audit (created_at DESC) WHERE destroyed_at IS NULL;

-- Nothing in this table is cardholder data, and the cheapest way to keep
-- that true is to have no column that could hold any. If someone proposes
-- adding `last_four` "just for reconciliation", that column is now in scope
-- and so is every replica, backup, and analytics copy of this table.

Two operational traps. The first is clocks, and it is specific to snapshot-restore architectures: a guest restored from a snapshot resumes with whatever the clock said at bake time, so a machine baked in June and restored in August believes it is June. Every timestamp it writes is wrong, TLS handshakes may fail on certificate validity, and the audit trail you were relying on quietly becomes fiction. PandaStack syncs the guest clock on restore, resume and wake for exactly this reason; if you are building on Firecracker yourself, put clock synchronization on the restore path before anything else goes there.

The second is that logs must leave the guest, because the guest is about to be destroyed — which makes your log pipeline a path out of the CDE, and everything on it inherits scope if account data can ride along. Emit structured events with an explicit allowlist of fields, not free-form strings you later run a redaction regex over. Regex redaction works beautifully until a delivery note contains a sixteen-digit order reference, or an exception message carries a serialized request object your framework helpfully attached. An allowlist fails toward missing data; a denylist fails toward a breach notification.

Shared cluster vs. long-lived VM vs. microVM per transaction

Same workload, three topologies. Verify any specific claim about a platform — isolation mode, network policy enforcement, tenancy model — against that platform's own current documentation; these details vary by version and plan, and they change.

• Segmentation evidence — Shared cluster: a composite argument spanning node pools, taints, NetworkPolicy, admission control and node agents, correct today and invalidated by a config change nobody flags as security-relevant. Long-lived dedicated VM: a clean boundary, easy to describe and well understood to test. MicroVM per transaction: the same clean boundary, and the machine behind it is younger than the assessment meeting.
• Scope footprint — Shared cluster: the node, its kernel, every co-resident workload, the orchestrator and the node agents all have a claim on being in scope. Long-lived dedicated VM: the VM, its host, its network path. MicroVM per transaction: the host and hypervisor are still in scope — that never goes away — but no co-resident tenant workload and no long-lived worker state comes with them.
• Blast radius of a parser bug — Shared cluster: code execution in a process sharing a kernel with everything on the node, holding whatever the pod's service account carries. Long-lived dedicated VM: a hardware boundary, but the attacker inherits every transaction that machine has processed and will process. MicroVM per transaction: a throwaway guest holding one transaction, one credential, one reachable endpoint.
• Account data residue — Shared cluster: temp files, page cache, core dumps routed to a host crash handler, swap and aggregated log pipelines all outlive the job. Long-lived dedicated VM: the same categories, bounded by a lifetime measured in months. MicroVM per transaction: memory and disk stop existing together at job end, so residue is bounded by lifetime rather than by a cleanup routine running correctly on the unhappy path.
• Cost and latency — Shared cluster: effectively free per job, which is why everyone starts here and why the scope spreadsheet gets so long. Long-lived dedicated VM: cheap per job, expensive per tenant, idle capacity is the business model. MicroVM per transaction: creates restore a baked snapshot at p50 179ms and p99 203ms — a fifth of a second in front of an acquirer round trip that will take longer than that anyway.

What this does not do

Per-transaction microVM isolation is one technical control addressing one family of risks. Several important things sit entirely outside it, and the consequence of believing an oversell here is regulatory.

• It concentrates the CDE, it does not eliminate it — the host, the hypervisor, the control plane scheduling the guests and the network path are all in scope, and now more important, not less. A narrow stable boundary instead of a wide shifting one is a good trade, not a vanishing act.
• It produces no AOC, SAQ, or validated segmentation — those come from an assessment. The architecture makes several assertions cheap to make truthfully and cheap to evidence; someone still has to make them and have them assessed.
• It says nothing about cryptography or key management — where keys live, how they rotate, whether an HSM is involved, whether you sit inside a validated P2PE solution. A microVM boundary does not encrypt anything.
• Your image bake pipeline is now a security-impacting system — whatever produces the template your CDE guests restore from decides what code runs in the CDE. Pin the generation so an audit row can say which image ran.
• A snapshot of a guest that held account data is account data — a snapshot is a memory image plus device state plus a disk. If a PAN was in a parser buffer when you captured it, those bytes are in the file. Snapshot the clean template and restore per job. Hibernation is a snapshot with a friendlier name.
• If someone else runs the hardware, they are in your scope as a service provider — you will need their attestation and a responsibility matrix. Self-hosting changes who holds those obligations; it does not remove them.

The biggest scope reduction is still not touching the data

If you can arrange for account data to never reach your systems at all — hosted payment fields, an iframe or redirect to a validated provider, tokenization at the gateway before anything of yours sees a PAN — do that instead. Nothing here competes with it. The eligibility rules for the shorter self-assessment questionnaires are precise and worth reading with your acquirer, because whether card data touches your page context can be the difference between a short questionnaire and a full report on compliance. That decision is usually available early and rarely available later.

Per-transaction isolation earns its place when that option is closed. You are a facilitator or processor and handling account data is the product. You do card-present or terminal integration and the raw data genuinely arrives. You run tokenization yourself because the tokens need to be yours. Or your acquirer's interface speaks ISO 8583 over a private link and was last redesigned when the answer to “what parses this?” was a C program that is still the answer. The data is coming; the only question is what it lands in.

The summary

Scope is the cost function of PCI DSS, and it spreads along connectivity and administrative control. Segmentation is the lever, but it is a claim you substantiate rather than a product you install — and shared-kernel platforms make that substantiation long, brittle, and dependent on configuration owned by people who do not know they own a compliance boundary. Give each transaction or tenant its own machine, put no credential and no route in it beyond what that job needs, assert the fences from the inside before data arrives, and destroy it at the end. You get a smaller CDE with a shorter-lived population, an audit trail that names machines instead of cards, and “we do not store account data” as a property of the architecture. You do not get compliance, because nobody sells that. The architecture just means the part of the conversation about how your workloads are separated takes an afternoon instead of a quarter.