What FedRAMP Asks of a Platform That Runs Untrusted Code

Somebody in sales forwards an email that ends with a question mark. "Can you get FedRAMP?" The engineering answer people usually give is a shrug and a number of months. The useful answer starts with what the regime actually demands, because for a platform whose product is executing other people's code, roughly a third of it maps onto your architecture unusually well and another third maps onto it badly in ways that are genuinely interesting.

Before anything else: PandaStack is not FedRAMP authorized. We are not in process, we have no agency sponsor, and we are not listed in the marketplace. Nothing below is a claim otherwise. What this post is: the control regime as it actually reads, mapped onto a Firecracker microVM fleet — snapshot-restore on every create, no warm pool, guest memory paged out of object storage — with the places it fits and the places it does not called out by name. If you are evaluating whether you could ever sell into government, or building something similar, the mismatches are the part worth knowing about in advance.

The framing error to discard first is that FedRAMP is SOC 2 with more paperwork. SOC 2 lets you write your own controls and then proves you followed them. FedRAMP hands you the baseline — NIST 800-53 tailored by the programme, north of three hundred controls at Moderate — makes you document how each one is implemented in a System Security Plan, has an accredited third-party assessor test them, and then puts you on a monthly reporting obligation that does not stop for as long as you hold the authorization. It is less an audit than a subscription.

The authorisation boundary is a drawing, and it is the whole exam

The single artefact that determines how much of the next eighteen months hurts is the boundary diagram. Everything that stores, processes or transmits federal information is inside it. Inside means: in your inventory, in your monthly scans, in your SSP, in your continuous monitoring, in your incident reporting obligations. Outside means it is either an external service you document and accept residual risk on, or a service carrying its own authorization that you inherit from.

The mistake is drawing the boundary around the things that look like servers. Here is the one specific to a snapshot-restore platform, and it caught me the first time I thought about it properly. When streaming restore is enabled, a guest does not get its memory copied to the host before it runs. It starts running, touches a page, and that page fault becomes an HTTPS range request to an object-storage bucket, which returns a four-megabyte chunk that gets installed into the guest's address space while the vCPU waits. The bucket is not a backup target. It is guest RAM, in the live execution path of a running machine.

So the bucket is inside the boundary, with everything that follows: encryption and key management, access logging, who at the cloud provider can read it, what happens to it on decommission. If you had it filed as "artefact storage" in a diagram somewhere, you had it wrong, and an assessor who understands the architecture will find that faster than one who does not.

Apply the same test to everything else and be prepared for the answers. The Postgres holding orgs, sandboxes and metadata: inside. The ClickHouse taking the event stream: inside, and it is holding audit records, which brings its own control family. The billing integration: external, and what crosses the line is usage metadata rather than customer data, which is a sentence you will write and defend. The pipeline that builds a binary and hot-swaps it onto every production host: inside, and see the change-control section, because that one is a problem.

The inventory, and an asset class that lives forty seconds

CM-8 asks for a complete inventory of system components. FedRAMP makes it concrete: an integrated inventory workbook with every asset carrying a unique identifier, an IP, its function, its owner, whether it is virtual, whether it is scanned and by what — updated monthly as a continuous-monitoring deliverable. The control was written with a fleet of named servers in mind, and it shows.

Our fleet is not that. A create is a restore of a baked snapshot with a p50 around 179 milliseconds, there is no warm pool of idle machines, and a sandbox can be created, run a test suite and be reaped inside a minute. Enumerating those as assets is not merely tedious, it is category-incorrect: by the time a row reaches a spreadsheet the asset has been destroyed and its /30 subnet handed to someone else's workload.

I have first-hand evidence of how badly the ad-hoc version of this goes. The nearest thing we have to a record that a VM ever existed is a control-plane table called sandbox_lifecycle: one row per successful create, with a created_at and a deleted_at. It was built for onboarding email and billing reconciliation, not for compliance, which is exactly the sort of provenance that should make you suspicious. For a period nothing ever wrote deleted_at. User-initiated deletes went through one code path; the agent's own idle reaper killed VMs autonomously without ever touching the control-plane database. The result was three hundred and sixty-five rows in production describing sandboxes that looked permanently alive and had not existed for weeks.

The fix was unglamorous: a reconcile loop that periodically asks every active agent which sandboxes still exist and stamps deleted_at on every "live" row whose VM is gone. That background job is the difference between an inventory and a fiction, and the generalisable lesson is that any inventory derived from control-plane intent will drift, because the things that destroy your assets are not always the things you asked to destroy them.

The way out is not to enumerate the guests. It is to move the unit of inventory up a level, the way the container world eventually did. The enumerated assets become the hosts, the network pool, the control-plane services and the small set of baked template images — plus an enforced statement that a guest can only ever be a restore of one of those images. The ephemeral instances then become a class with a population count and a lifetime distribution rather than forty thousand rows, and the assessor's question changes from "list them" to "prove nothing else can run", which is a question you can actually answer.

And the scanning problem sitting underneath it

RA-5 wants authenticated vulnerability scans on a monthly cadence, with findings tracked to remediation deadlines that vary by severity. You cannot credential-scan a machine that lived forty seconds. So the artefact you scan is the template rootfs and the baked seed, on the same cadence, with the remediation clock running against the image rather than against any instance.

There is a cost here the control never anticipated. Patching an image in this architecture means re-baking the template snapshot, and re-baking invalidates snapshots derived from the old one. A routine "fix this CVE" becomes a fleet-wide artefact regeneration plus a distribution step to every host. That is fine — it is just something to budget for, because in a conventional estate a patch is a package upgrade and here it is a build with a fan-out.

-- Monthly ConMon inventory delta for an ephemeral fleet.
--
-- The unit of inventory is NOT the individual VM. It is the template image,
-- plus a population count and lifetime distribution for the class of guests
-- restored from it. This query produces that, per org, for a reporting month.
--
-- sandbox_lifecycle is the control-plane record: one row per successful
-- create, stamped with deleted_at when the VM is gone. deleted_at is written
-- by a reconcile loop that asks every agent which sandboxes still exist --
-- NOT derived from user intent, because the agent's own idle reaper kills
-- VMs without ever telling the control plane.
SELECT
    template,
    count(*)                                            AS created_in_period,
    count(*) FILTER (WHERE deleted_at IS NOT NULL)      AS decommissioned,
    count(*) FILTER (WHERE deleted_at IS NULL)          AS still_live_at_close,
    round(avg(extract(epoch FROM (
        coalesce(deleted_at, now()) - created_at
    )))::numeric, 1)                                    AS mean_lifetime_seconds,
    percentile_disc(0.95) WITHIN GROUP (
        ORDER BY extract(epoch FROM (coalesce(deleted_at, now()) - created_at))
    )                                                   AS p95_lifetime_seconds
FROM sandbox_lifecycle
WHERE created_at >= date_trunc('month', now() - interval '1 month')
  AND created_at <  date_trunc('month', now())
GROUP BY template
ORDER BY created_in_period DESC;

-- The row that matters to an assessor is not any of the counts. It is that
-- `template` only ever contains values from the approved image registry. If
-- it can contain anything else, your inventory boundary is not enforced --
-- it is merely described.
SELECT DISTINCT template
FROM sandbox_lifecycle
WHERE created_at >= now() - interval '90 days'
  AND template NOT IN ('base', 'code-interpreter', 'agent', 'browser', 'postgres-16');

Snapshot memory images are an asset class the baseline never anticipated

This is the part where I think 800-53 genuinely has no row for what you are holding. A Firecracker snapshot is two files: the device and vCPU state, and the memory image. The memory image is a byte-for-byte copy of the guest's RAM at the instant it was frozen. Whatever the workload had in memory is in it — decrypted database rows, an access token, a private key a process generated, the plaintext of a request that was in flight when the freeze happened.

Under the baseline that one object is simultaneously several things. It is data at rest under SC-28, so it needs encryption and key management. It is media under MP-6, so it needs a sanitisation-on-disposal story. It is a system component with a lifecycle under CM-8, so it belongs in the inventory. And because forking copies a parent's memory into its children, it is a replication path that spreads whatever it contains — a same-host fork lands in 400 to 750 milliseconds and a cross-host one in 1.2 to 3.5 seconds, and the cross-host case means the image crossed a network to get there.

None of the controls forbid any of this. They simply were not drafted with the possibility that a durable, transportable, restorable copy of a running machine's RAM would be a routine operational artefact. What you have to write down is therefore not templated for you: where the images live, who can read them, how they are encrypted and with whose keys, how long they persist, how destruction is evidenced, and what crosses a boundary when a fork lands on a different host. None of that is technically hard. All of it is easy to omit until somebody asks what the vm.mem file is.

FIPS 140, all the way into the guest

SC-13 surprises engineers more than any other control, because it does not ask whether you use strong cryptography. It asks whether the cryptographic module performing the operation is CMVP-validated and being operated in an approved mode. AES-256 from an unvalidated library does not satisfy it; the identical algorithm from a validated module in approved mode does. The distinction is bureaucratic and it is also the entire control.

In an architecture like this the module question surfaces in more places than a first pass suggests: TLS termination at the edge, the object-storage client fetching seeds and memory chunks, control-plane database connections, whatever transport the host uses to reach inside a guest, and — because the guest is where federal data is actually processed — the guest image's own userspace and kernel crypto.

Concretely from our side, and stated as a limitation rather than a capability: the agent generates an ed25519 keypair using Go's standard library and injects the public half into guests for the SSH bridge. Ed25519 became an approvable algorithm when FIPS 186-5 landed, but algorithm approvability and module validation are separate questions, and Go's default crypto is not a validated module operating in approved mode. That is a perfectly reasonable engineering choice for a commercial platform and it is not a FedRAMP-ready one, and I would rather say so plainly than imply a posture we do not have.

Getting there is a build-and-configuration exercise rather than a feature: compile against a validated module everywhere, configure approved mode, disable everything outside it, and re-bake every template so the guest is compliant too. That last clause is the expensive one, because it means the image you hand customers has to be part of the crypto story, and customers will install things into it.

Continuous monitoring, or: it never becomes "done"

A SOC 2 Type II observes a window and produces a report. FedRAMP produces an authorization and then never stops. Monthly you owe scan results across infrastructure, web and database; an updated Plan of Action and Milestones with every finding tracked to a remediation date; an updated inventory; and deviation requests for anything you want treated as a false positive or as operationally risky to fix. Annually you owe a full assessment and a penetration test. Miss the cadence and the authorization is at risk independently of whether anything is actually wrong with the system.

The engineering consequence is that evidence has to be produced by a machine on a schedule rather than by a human before a deadline. For an ephemeral fleet the monthly inventory delta is the awkward deliverable, and the honest way to produce it is a job that reconciles the control plane against the hosts, records the class population and its lifetime distribution, and files the artefact whether or not anyone reads it that month.

from pandastack import Sandbox

# The CM-8 question an assessor asks about an ephemeral fleet is not "list
# every VM" -- by the time you answer, the answer is wrong. It is "prove that
# nothing outside the approved image set can be running."
#
# So the evidence artefact is a population snapshot plus a provenance check,
# taken on a schedule and filed whether or not anyone reads it that month.
APPROVED = {"base", "code-interpreter", "agent", "browser", "postgres-16"}

live = Sandbox.list()

population = {}
unapproved = []
forked = []

for sbx in live:
    population[sbx.template] = population.get(sbx.template, 0) + 1
    if sbx.template not in APPROVED:
        unapproved.append((sbx.id, sbx.template))
    # A sandbox restored from a user snapshot inherits that snapshot's memory
    # image. Provenance therefore runs through the snapshot, not the template,
    # and the snapshot is its own inventoried artefact.
    if sbx.from_snapshot:
        forked.append((sbx.id, sbx.from_snapshot))

for template, count in sorted(population.items(), key=lambda kv: -kv[1]):
    print(f"{template:20s} {count:5d} live")

if unapproved:
    # This is a finding, not a log line. Something created a guest from an
    # image that is not in the authorisation boundary.
    for sandbox_id, template in unapproved:
        print(f"UNAPPROVED IMAGE: {sandbox_id} running {template}")

print(f"{len(forked)} guests restored from user snapshots (provenance via snapshot id)")

Two things about that script are the point. The first is that it produces a population and a provenance check rather than an enumeration, which is the only version of the question that stays true for longer than a second. The second is the from_snapshot branch: a guest restored from a user snapshot inherits that snapshot's memory image, so its provenance runs through the snapshot rather than the template, and the snapshot is its own inventoried artefact. Miss that and your image-provenance argument has a hole in it exactly where the interesting data is.

"We ship on a git tag" is a governance problem

CM-3 change control is unremarkable. What is unusual outside FedRAMP is the significant change request: certain categories of change require you to notify the authorizing agency and the programme office, with a security impact analysis, and in practice to get agreement, before you deploy. A new external service, a change to the cryptographic implementation, a change to the boundary, a new component type, a material architectural change — all candidates.

Now look at how we actually ship. A push to main that touches the API directory builds a binary, uploads it, and hot-swaps it onto every running edge VM in about fifteen seconds, with the load balancer keeping traffic flowing across the restarts. The host agent ships on a version tag. That velocity is a deliberate product property and it is precisely the thing a significant change process exists to interrupt.

The reconciliation is not to slow everything down. It is a taxonomy, enforced in the pipeline rather than described in a document. The overwhelming majority of changes are routine: they flow through normal change control, get logged, and the log becomes evidence rather than an obstruction. A small, precisely defined set — anything that alters the boundary diagram, introduces an external service, changes crypto, or changes the isolation model — has to stop at a gate before deployment. The failure mode is putting that gate in a wiki, where an engineer with merge rights steps over it without ever knowing it existed. It belongs in CI as a required check on the specific paths that define the boundary.

Boundary protection, and the awkward truth about egress

SC-7 and its enhancements treat the boundary as a network object: mediated, monitored, and denying outbound traffic by default with explicit documented exceptions. For a platform whose entire product is running a stranger's code and letting it reach the internet, this is where compliance and product collide hardest, and it is worth describing our actual posture rather than an aspirational one.

Each sandbox lives in its own network namespace with a /30 out of a pre-allocated pool of 16,384, and egresses through NAT out the host's WAN interface. The host's forward chain carries a set of first-match drops ahead of the accept. Pool-to-pool traffic is dropped, so no guest can scan the /16 and reach a neighbour's SSH or Postgres port. The entire 169.254.0.0/16 link-local range is dropped, so no guest can curl the cloud metadata service and walk away with the host's service-account token — on GCP that token has broad scope and would read every customer's seeds and snapshots, which makes it the single most important rule in the chain. The well-known Stratum mining ports are dropped, because free-tier crypto-mining is a recurring abuse pattern and it trips the cloud provider's own abuse detection, which is a worse day than it sounds.

Then everything else is accepted. That is default-allow with a denylist. The comment in our own source says as much: a denylist, not a panacea, since a miner can use a custom port or tunnel over 443. It stops the abuse we have actually observed and it does not satisfy a control that asks for deny-by-default at the boundary.

#!/bin/sh
# The shape of a default-deny egress policy for a guest pool, contrasted with
# the default-allow-plus-denylist most sandbox platforms actually ship.
#
# WHAT WE RUN TODAY (honest): first-match DROPs, then a blanket ACCEPT.
#   iptables -I FORWARD 1 -s "$POOL" -d "$POOL"            -j DROP  # no guest reaches a neighbour
#   iptables -I FORWARD 1 -s "$POOL" -d 169.254.0.0/16     -j DROP  # no guest reaches cloud metadata
#   iptables -I FORWARD 1 -s "$POOL" -o "$WAN" -p tcp \
#            --dport 3333 -j DROP                                   # stratum / mining pools
#   iptables -A FORWARD    -s "$POOL" -o "$WAN"            -j ACCEPT  # <-- everything else
#
# That is a denylist. It stops the abuse we have actually seen. It does not
# satisfy a control that asks for deny-by-default at the boundary.

set -eu
POOL="10.200.0.0/16"
WAN="ens4"

# WHAT SC-7 WANTS: policy DROP, plus an enumerated, documented allow list.
iptables -N PS_EGRESS 2>/dev/null || iptables -F PS_EGRESS
iptables -C FORWARD -s "$POOL" -o "$WAN" -j PS_EGRESS 2>/dev/null \
  || iptables -A FORWARD -s "$POOL" -o "$WAN" -j PS_EGRESS

# Established return traffic. Without this nothing works at all.
iptables -A PS_EGRESS -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Every allowed destination is an entry someone signed off on and that appears
# in the SSP. An ipset keeps the rule count sane when the list is long.
iptables -A PS_EGRESS -p tcp --dport 443 -m set --match-set ps_allowed_egress dst -j ACCEPT

# Log then drop. The log is the evidence that the boundary is mediated; a
# silent drop proves nothing to an assessor and debugs badly for your users.
iptables -A PS_EGRESS -m limit --limit 20/min -j LOG --log-prefix "ps-egress-deny "
iptables -A PS_EGRESS -j DROP

# The reason this is a product decision and not a firewall change: with the
# policy flipped, `pip install`, `npm install`, `go mod download` and every
# outbound API call a customer's code makes stop working until an egress proxy
# exists and its allow list is maintained. Flip it in staging first and count
# what breaks -- it is always more than the list you wrote down beforehand.

Mechanically the flip is the same chain with the policy inverted and an allow list attached. The hard part is not iptables. It is that flipping it changes what customer code is able to do, and every package install, model API call and git clone becomes a question of whether the destination is on a list somebody maintains. That is a product decision wearing a compliance costume, and it is the reason most sandbox platforms have not made it.

IL4 and IL5 are a different building

If the buyer is the Department of Defense, FedRAMP is the entry fee rather than the destination. The Cloud Computing SRG defines impact levels: IL2 for public and non-critical information, IL4 for controlled unclassified information, IL5 for higher-sensitivity CUI and mission-critical unclassified national-security systems, IL6 for classified. A DoD provisional authorization is granted through DISA and generally builds on a FedRAMP Moderate or High baseline plus DoD-specific requirements on top.

What those add is mostly not architecture. Infrastructure and data located in the United States. Personnel screened to specific citizenship and background requirements. Connectivity through DoD-controlled boundary services rather than the open internet. And at IL5, stronger separation between DoD workloads and everything else on your platform.

That last requirement is the one place where an architecture like this has a genuinely strong argument. IL5 separation guidance is fundamentally a question about how much of the stack a DoD workload shares with your other tenants, and "every tenant workload runs on its own kernel behind a hypervisor with a deliberately minimal device model" is a materially better sentence than "namespaces, cgroups and seccomp". It is the same argument that caused Firecracker to be built in the first place. It does not get you an authorization, but it means the isolation section of the SSP writes itself, which is more than most platforms can say. The personnel, location and physical requirements remain organisational, and no hypervisor has ever fixed one of those.

Where this architecture argues well, and where it argues badly

• Argues well: tenant separation. A per-workload VM with its own kernel, its own memory and its own network namespace is a structural boundary rather than a configurational one, which means the argument that it still holds next quarter is much easier to make than for a runtime flag that one privileged container undoes.
• Argues well: least functionality. CM-7 wants a minimal, enumerable software set. A guest that is a restore of a baked image and is destroyed after one execution has exactly that by construction, rather than as the result of hardening a machine that has been accumulating packages since 2023.
• Argues well: media protection on the ephemeral disk. The rootfs is a copy-on-write clone destroyed with the VM, so the sanitisation story for the working disk is short and true. The snapshot store is where the difficult version of that conversation lives.
• Argues badly: inventory and scanning. The control set assumes a stable, enumerable set of machines. Ours is a population with a birth rate, and every honest answer requires moving the unit of accounting up a level and then defending that move.
• Argues badly: egress. Default-allow with a denylist is defensible engineering for a commercial sandbox and is not deny-by-default. Changing it changes the product.
• Argues badly: FIPS inside the guest. Customers install their own dependencies into an environment you handed them, so a validated crypto posture in the guest is a shared-responsibility line you have to draw explicitly and then police, which is uncomfortable in both directions.
• Argues badly: velocity. Every property that makes the platform pleasant to operate — hot-swap deploys, auto-baking a template on first spawn, an agent that autonomously reaps idle guests — makes "describe the exact state of the authorized system" harder to answer. Autonomy and auditability pull in opposite directions and you will be choosing between them repeatedly.

The order I would do it in

1. Confirm you have a buyer. Authorizations run through an agency sponsor, and a compliance programme without one is an extremely expensive way to learn NIST vocabulary. The programme has also been reworking its authorization process in recent years, so check current FedRAMP guidance for the mechanics rather than any blog post, including this one.
2. Draw the boundary before writing a single control narrative. Then apply one test to every component: does federal information touch it. Include the surprising ones — your object storage is guest memory if you stream restore, and your event pipeline holds audit records.
3. Fix the inventory before promising anything to anyone. Reconcile from the hosts, not from intent. If your control plane has ever believed a destroyed machine was alive, find out why before an assessor does.
4. Make the image the unit of scanning, and make it structurally true that nothing outside the approved image set can run. A statement in a document is a description; a check that fails a create is a control.
5. Write the snapshot paragraph early. Where memory images live, who reads them, how they are encrypted, when they die, and what crosses a host boundary on a fork. It is the section no template gives you and the one most likely to generate follow-up questions.
6. Cost the FIPS work honestly. It is a build change in every binary, a configuration change in every service, and a re-bake of every template. It is not a toggle and it is not a quarter.
7. Put the change gate in CI, scoped to the paths that define the boundary. A gate an engineer can merge past is not a gate.
8. Flip egress to default-deny in a staging fleet and count what breaks. It will be more than the list you wrote beforehand, and the gap between those two numbers is the real size of the project.

The interesting finding, having worked through it, is that the mismatch is not about security. The isolation story a microVM platform tells is stronger than what the baseline assumes it is going to be told, and the sections on separation, least functionality and disposal are easier to write here than for almost any other architecture. What breaks is the bookkeeping — specifically the assumption running quietly underneath half the control set that a system is a stable set of machines you can list, scan and describe.

If you are evaluating whether you can ever sell into government, the architecture is not your blocker. The blocker is that FedRAMP is an operating model: monthly deliverables, a change gate with teeth, a sponsor, an assessor, and a POA&M that somebody owns as a job. It costs seven figures and a couple of years, and it is worth it precisely when you have the buyer and not a moment before. If you do not, the useful subset is free and is good engineering regardless: draw the boundary properly, make the inventory real, and write down what is actually inside your snapshots.