Data Residency for Sandboxes: Every Byte, Including RAM

There is a sentence that gets typed into security questionnaires far more casually than it deserves: "customer data stays in the EU." It sounds like a configuration setting. It is actually a claim about every byte your platform writes down, including the ones nobody decided to write. On a platform built out of microVM snapshots, the sneakiest of those live in a file called `vm.mem` — a byte-for-byte image of whatever the guest was holding in RAM at the moment you froze it. Nobody puts "a copy of the customer's decrypted payload, in memory, as a file, in object storage" on their data-flow diagram. That does not stop it from being true.

I'm Ajay; I build PandaStack, an open-source Firecracker microVM platform, and this comes up every time a customer with a regulated workload gets past the demo. This post is the honest version of the answer: why latency-driven placement and residency-driven placement are different problems that happen to use the same word "region", the full list of surfaces where sandbox data actually lands, how to express residency as a hard scheduling filter that fails closed instead of a preference that silently spills, why cross-host fork is a residency event, and what all of this costs you operationally. Because it does cost you, and anyone who tells you multi-region residency is free is selling something.

Two reasons to care about regions, and only one is optional

Latency-driven placement is an optimisation. Your user is in Frankfurt, so you'd rather their sandbox ran in Frankfurt than in Oregon, because a fat round-trip on every agent step adds up to a product that feels sluggish. Get it wrong and the experience degrades; nobody gets a letter from a regulator. This is the constraint schedulers are traditionally built for: a weight, a bonus term, a soft affinity that gets outvoted when the preferred region is full. Spilling to a further region is exactly correct there, because a slow sandbox beats no sandbox.

Residency-driven placement is a constraint, and the difference is that spilling is never the correct behaviour. If a workload is marked EU-only and the EU is full, the right answer is a `503` with a capacity error, not a sandbox in `us-east`. This is a genuinely uncomfortable design position for anyone who has spent a career making systems degrade gracefully, because here graceful degradation is the incident. The failure mode you're engineering for is not "the user waited", it is "we told an auditor something that was false, in writing, twice a year."

Enumerating the surfaces: where sandbox data actually lands

Before you can enforce residency you have to know what you're enforcing it over. The instinct is to treat a sandbox as a compute event — code runs, output comes back, nothing persists. That's true of the sandbox and false of the platform underneath it. The honest inventory, roughly in order of how often each one gets missed:

• The guest rootfs — the obvious one. Whatever the workload wrote to disk lives on a host in some region, usually as a copy-on-write clone of a template image. Ephemeral, but "ephemeral" only means it exists until it doesn't, and residency applies to that window.
• Memory snapshots — the one everyone forgets. Pausing, hibernating, or baking a VM writes guest RAM to a file: decrypted secrets, in-flight PII, the model's context window, that CSV somebody loaded into a DataFrame, all of it verbatim and now at rest. The most sensitive artifact the platform produces, disguised as an implementation detail.
• Forks and their parents — a fork inherits its parent's memory image. If the fork lands on another host, the memory image travelled. Cross-region fork is not a performance decision, it is a data transfer.
• Object storage buckets — snapshots, seeds, backups, and function bundles live in a bucket, and buckets have replication settings that default to multi-region in more places than you'd like. A bucket labelled "eu" that really is an EU multi-region is fine; one quietly set to a global class is a residency breach with a green dashboard.
• Logs and metrics pipelines — guest stdout, build logs, exec output, and event streams get shipped somewhere for search and retention, and that pipeline is almost always one global cluster because it was built when you had one region. Log lines contain user data; often more of it than the database does.
• Control-plane database rows — sandbox IDs, template names, metadata tags, org and user identifiers, timestamps, and whatever customers typed into a `label` field. Usually personal data under GDPR even with no payload in it, and usually living in exactly one Postgres in exactly one region.
• Backups and their retention — the backup of an EU database in a non-EU bucket is the classic finding. So is the point-in-time archive, the WAL shipping target, and the disaster-recovery copy that exists specifically to be somewhere else.
• Support tooling — the screenshot in the ticket, the exec session an engineer opened to debug a stuck VM, the log excerpt pasted into Slack, the heap dump attached to a bug. This surface has no schema, no encryption policy, and full access. It is where residency programs go to die.

Why memory snapshots deserve their own paragraph

A Firecracker snapshot is two files: the device and CPU state, and the guest's memory. The memory file is not a summary or a diff — it's the machine's RAM, page for page. Everything the guest had decrypted, deserialised, or downloaded is in there in plaintext, because RAM is where things go to be plaintext. TLS gave you encryption in transit and the disk gave you encryption at rest; in between sits the one copy that has neither, which you just wrote to a file and possibly uploaded to a bucket.

This is not an argument against snapshots — snapshot-restore is what makes a fresh, hardware-isolated VM per request viable at all, and on PandaStack it's the difference between a 3-second cold boot and a p50 179ms create. It's an argument for treating the memory artifact as the most sensitive object in the system: same region as the workload, same encryption policy as the customer's payload, shortest retention you can tolerate, and a hard rule that it never crosses a residency boundary for any reason including "the other region had spare capacity." If you inherit one rule from this post, that's the one.

Region pinning is a filter, not a preference

Most schedulers, ours included, are a scoring function: rank hosts by free CPU and free memory, add bonuses for locality, pick the best. That shape is right for latency and wrong for residency, because a score can always be outvoted. The fix is structural — an eligibility filter that runs first and produces a candidate set, then scoring over only that set. Residency lives in the filter, nothing downstream may add a host back in, and an empty candidate list means a capacity error rather than a fallback to the global pool.

The second half is failing closed on missing information. A host whose region label is empty, a heartbeat that's stale enough that you can't confirm where it is, a request whose residency tag didn't get resolved — every one of those is a reject, not a shrug. Unknown is not permitted; unknown is denied.

package scheduler

import "errors"

// ErrNoCapacity is a legitimate, expected outcome. Returning it is the
// system working correctly. Placing outside the residency zone to avoid
// returning it is the system failing.
var (
	ErrNoCapacity   = errors.New("no eligible host in residency zone")
	ErrUnknownScope = errors.New("residency scope unresolved: refusing to place")
)

type Request struct {
	Template  string
	Residency string // "eu", "us", "in" -- a CONSTRAINT, not a hint
	PreferZone string // "eu-west-1b" -- a PREFERENCE, may be ignored
}

type Agent struct {
	ID            string
	Residency     string // stamped at provisioning, never inferred at runtime
	Zone          string
	FreeCPU       float64
	FreeMemGB     float64
	HeartbeatAge  float64 // seconds
	SeedsPresent  map[string]bool
}

// Pick runs eligibility BEFORE scoring. The two phases are deliberately
// separate functions so that no future "just add a small bonus for..."
// patch can accidentally reintroduce a host the filter rejected.
func Pick(req Request, agents []Agent) (Agent, error) {
	if req.Residency == "" {
		// Fail closed. An unlabelled request is a bug, and the safe
		// behaviour for a bug is to refuse, not to pick the fastest host.
		return Agent{}, ErrUnknownScope
	}

	eligible := make([]Agent, 0, len(agents))
	for _, a := range agents {
		switch {
		case a.Residency == "":
			continue // unknown location == not eligible, ever
		case a.Residency != req.Residency:
			continue // THE constraint. No score can overturn this.
		case a.HeartbeatAge > 30:
			continue // can't confirm it's alive, or where it is
		case !a.SeedsPresent[req.Template]:
			continue // pulling a seed cross-region is a data transfer
		}
		eligible = append(eligible, a)
	}

	if len(eligible) == 0 {
		// Do NOT widen the search. Do NOT retry globally. Surface a 503
		// and let capacity planning be a capacity problem.
		return Agent{}, ErrNoCapacity
	}

	return bestScore(req, eligible), nil // preferences apply only in here
}

func bestScore(req Request, eligible []Agent) Agent {
	best, bestScore := eligible[0], -1.0
	for _, a := range eligible {
		s := 0.6*a.FreeCPU + 0.3*a.FreeMemGB
		if a.Zone == req.PreferZone {
			s += 1.0 // a preference: nudges, never decides
		}
		if s > bestScore {
			best, bestScore = a, s
		}
	}
	return best
}

The bit worth stealing isn't the scoring maths, it's the shape: two functions, one of which can't see the other's inputs. When someone adds "prefer hosts with a warm cache" in six months they'll add it to `bestScore`, because that's the only place a bonus fits. The filter stays boring, which is what you want in the code that answers the auditor's question.

Cross-host fork is a data transfer wearing a scheduling costume

Forking is the feature people fall in love with on a snapshot platform: take a running VM with its environment set up and its context loaded, then branch it into N copies to explore N paths. Same-host fork on PandaStack runs 400-750ms, because the memory image and rootfs are already local and the copy is copy-on-write. Cross-host fork runs 1.2-3.5s, and the gap is the entire point of this section: it is a multi-gigabyte memory image moving over the network.

That transfer is a data event. If the destination host is in another region, you just replicated the customer's decrypted RAM across a border as a side effect of a latency optimisation, and the API call that did it was named `fork`. So the policy has to be explicit: same-residency by default, cross-host-within-region as an ordinary placement decision, and cross-region fork as a separate opt-in operation rather than a fallback. Migration and evacuation need the same rule. "Host is unhealthy, move the workload" is a fine reflex that must still respect the filter; on a dying host with no in-region capacity, the right move is to fail the workload loudly rather than rescue it into the wrong jurisdiction.

Template and seed artifacts have to be regional too

Here's a subtle one that catches people who otherwise did everything right. Snapshot platforms boot from baked artifacts — a template rootfs plus a pre-baked snapshot, published to object storage and pulled by hosts. Those are usually your content rather than customer data, so the temptation is one global bucket every region pulls from. Two problems. A host pulling a seed across an ocean has a miserable cold start, which pushes people straight back toward global caching. And custom templates are customer content: a tenant's own code, credentials baked into a layer, a seeded dataset — that artifact inherits their residency scope. Publish seeds per region, make "seed present in this region" part of the eligibility filter (as in the Go above), and never let a missing seed trigger a cross-region pull.

What this looks like from the client

On the caller's side, region pinning should be an explicit input rather than something inferred from where the call came from — client-IP inference is how a support engineer on a VPN puts a German tenant's workload in Virginia. Pass the constraint, tag the workload so it stays attributable, and verify what you got:

from pandastack import Sandbox

RESIDENCY = "eu"  # comes from the tenant record, never from client IP


def run_for_tenant(tenant_id: str, case_id: str, src: str) -> str:
    """Create a residency-pinned sandbox and refuse to use a mis-placed one."""
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=900,
        metadata={
            # These tags are the audit trail: they travel with the sandbox,
            # land in the lifecycle table, and answer "whose data was this
            # and where were we told to keep it?" months later.
            "residency": RESIDENCY,
            "tenant": tenant_id,
            "case": case_id,
            "classification": "personal-data",
        },
    )

    # Belt and braces. The load-bearing enforcement is the scheduler's hard
    # filter, server-side; this is the client refusing to proceed if the
    # placement it got back disagrees with what it asked for. A residency
    # check that only exists in the client is theatre -- but a client that
    # doesn't check has no way to notice the day the server-side rule breaks.
    if sbx.metadata.get("residency") != RESIDENCY:
        sbx.kill()
        raise RuntimeError(f"placement outside {RESIDENCY}: refusing to run")

    try:
        sbx.filesystem.write("/work/main.py", src)
        r = sbx.exec("python /work/main.py", timeout_seconds=600)
        return r.stdout
    finally:
        # Destroy it. Every extra second of lifetime is another second of
        # guest RAM and disk sitting on a host inside your audit scope.
        sbx.kill()

Note what the metadata is for. It isn't access control — anyone who can create a sandbox can write any label they like. It's attribution: when someone asks in November which workloads processed personal data in September and where they ran, the answer has to come from a query, not an engineer's memory of a deploy.

Encryption and key locality: a real mitigation, a partial one

The standard mitigation for artifacts that might travel is to encrypt them with keys held in the residency region — customer-managed keys, a regional KMS, and a rule that key material never leaves. If ciphertext lands somewhere it shouldn't and the key is unreachable from there, the practical exposure is far smaller than a plaintext copy. Have that control regardless. It still isn't a substitute for placement. Legally, several regimes care about where data is processed and who can be compelled to produce it, not only whether a third party could read bytes at rest — "we hold the key" is an argument you want in addition to "the data never left", not instead of it. Mechanically, a running VM holds its memory in plaintext on the host by definition. Encryption protects the artifact; it does not protect the execution.

Proving after the fact where a workload ran

A control you can't evidence isn't a control, it's an intention. The auditor's question is never "is your scheduler configured correctly today" — it's "show me, for this six-month window, every workload belonging to this tenant and the region it executed in." So the placement decision has to be written down when it's made, alongside the workload's identity, and it has to survive the sandbox being destroyed. Lifecycle rows outliving their sandboxes is the whole point.

-- The report an auditor actually asks for: every sandbox belonging to a
-- residency-scoped tenant, and the region of the host it was placed on.
-- Rows persist after the sandbox is destroyed -- that is the requirement.
SELECT
    l.org_slug,
    l.residency_requested,
    a.region                       AS placed_region,
    count(*)                       AS sandboxes,
    min(l.created_at)              AS first_seen,
    max(coalesce(l.deleted_at, now())) AS last_seen
FROM sandbox_lifecycle l
JOIN agents a ON a.id = l.agent_id
WHERE l.created_at >= now() - interval '180 days'
  AND l.residency_requested IS NOT NULL
GROUP BY 1, 2, 3
ORDER BY 1, 2;

-- The alert that matters more than the report: anything placed outside the
-- region it asked for. In a correct system this returns zero rows forever,
-- which is exactly why it should page someone the first time it doesn't.
SELECT l.id, l.org_slug, l.residency_requested, a.region, l.created_at
FROM sandbox_lifecycle l
JOIN agents a ON a.id = l.agent_id
WHERE l.residency_requested IS NOT NULL
  AND a.region NOT LIKE l.residency_requested || '%'
ORDER BY l.created_at DESC;

Run the second query on a schedule and treat a non-empty result as a page, not a dashboard tile. A residency violation is a compliance incident with a clock on it — most regimes expect notification in days, and the days start when it happened, not when you noticed. The gap between those two dates is the only variable you control, and continuous evidence is how you keep it small.

Latency-driven vs residency-driven placement

Same word, same API field, two completely different engineering contracts:

• Nature of the rule — Latency-driven: a preference, expressed as a weight in a scoring function. Residency-driven: a constraint, expressed as a filter that runs before scoring.
• Behaviour under capacity pressure — Latency-driven: spill to the next-best region, since a slow sandbox beats none. Residency-driven: return a capacity error, since no sandbox beats one in the wrong jurisdiction.
• Behaviour on unknown inputs — Latency-driven: guess, using client IP or a default region, and be roughly right. Residency-driven: fail closed — an unlabelled request or an unlabelled host is a reject, not a default.
• Scope of enforcement — Latency-driven: the compute placement, and that's it. Residency-driven: compute plus rootfs, memory snapshots, forks, buckets and their replication class, logs, metrics, control-plane rows, backups, and support access.
• Fork policy — Latency-driven: fork wherever there's room, and cross-host costing 1.2-3.5s instead of 400-750ms is the whole trade-off. Residency-driven: same-region only, because moving a memory image is a data transfer, not a scheduling detail.
• Failure evidence — Latency-driven: a p99 graph gets worse and someone opens a ticket. Residency-driven: a query that must return zero rows returns some, and a regulatory clock starts.
• Cost of being right — Latency-driven: essentially free; it's one term in a formula. Residency-driven: fragmented capacity, per-region seed replication, slower cold starts in small regions, and a materially more complicated on-call.

The honest bill: what residency actually costs you

Capacity fragmentation is the big one, and it's just arithmetic. One global pool absorbs bursts beautifully because the peaks of unrelated tenants don't line up. Split it into five residency zones and you run five smaller pools, each needing its own headroom for its own peak, none able to lend to the others. You buy more total capacity for the same workload, and the smallest zone has the worst ratio. That headroom is what you're really selling when you sell a residency guarantee, and it should be priced accordingly.

Cold starts get worse in small regions, structurally: snapshot-restore is fast because the artifact is already on the host. A region with three hosts and low traffic has cold caches more often, so a create is likelier to be the slow path — a first-ever template boot is about 3s versus a p50 179ms restore. Volume hides that in a big region; a small one has nowhere to hide it. Pre-warming seeds per region helps and costs storage in every region: replication multiplies artifact storage by the number of zones, and every template rebuild fans out to all of them, turning a republish into a distributed operation with partial-failure modes. Read that twice before promising eight regions.

Then there's on-call, which is the cost nobody budgets. Every runbook grows a region dimension. "Drain the host" needs a caveat about where workloads may be drained to. Debugging gets harder, because the engineer on call at 3am may not be permitted to open an exec session into the region that's broken, and the log line they need may live in a cluster they can't query from where they are. That last one is not a technical constraint you can engineer away — it's the policy working as designed, and it will feel exactly like the policy fighting you. Build the tooling for it up front, because discovering it during an incident is a genuinely bad evening.

When this is overkill

If you have no regulated tenants and no contractual residency commitments, don't build this. Pin sandboxes near your users for latency, run one region until traffic justifies a second, and spend the effort on your product. Residency machinery built for a hypothetical enterprise deal is pure carrying cost, and the speculative version will be the wrong shape when the real requirement lands — the actual clause is always more specific and weirder than the one you imagined.

It's also premature if your platform produces no durable artifacts. If nothing is ever snapshotted, logs are dropped rather than shipped, and no customer content persists, then "where does it run" collapses back into a latency question with a much smaller compliance shadow. The moment you add snapshots, forks, or persistent volumes — which is to say the moment the platform gets genuinely useful — the surface reappears. Know which side of that line you're on rather than assuming.

Where it earns its keep is the intersection everyone eventually hits: regulated customer data, an execution platform that snapshots memory, and a contract with a jurisdiction named in it. There the sequence is clear — enumerate every surface including the ones that feel like implementation details, make residency a hard filter that fails closed, keep forks and seeds in-region, treat regional keys as defence in depth rather than as the guarantee, and write the placement down so you can prove it later. Less glamorous than boot-time numbers, and it's the part that decides whether "we're EU-only" is a fact or a sentence.