Per-Tenant Backup and Restore, Isolated by a MicroVM

Backup and restore is the feature nobody puts on the roadmap and everybody ends up owning. It starts as a cron job that pg_dumps a schema to a bucket, and within a year it's a product surface: customers export their data on demand, import it back, migrate between workspaces, restore a deleted project from last Tuesday. And somewhere in that year it quietly became one of the most dangerous pieces of code you run — because a backup job is, by design, a process with broad read access to a tenant's data, and a restore job is, by construction, a process that writes attacker-influenced bytes into your storage layer. Those are the two worst permission shapes in your entire system, and most teams run both of them in the same long-lived worker pool, with the same credential, filtered by a `tenant_id` in a Python variable.

I'm Ajay; I built PandaStack, a Firecracker microVM platform, and the shape I keep coming back to for this class of job is one machine per job, holding one tenant's credential, for the length of one job. Not because microVMs are magic, but because backup and restore is a workload where the blast radius of a single logic bug is "tenant A's data ends up in tenant B's account," and that is not a class of bug you fix with more code review. This post is about why the shared-worker pattern fails in ways that are hard to see, what a restore of an untrusted archive is actually doing on your machine, and how to build the thing so a bug stays inside a box you can throw away.

Two directions, one nightmare

Treat backup and restore as two separate threat models, because they fail in opposite directions and people usually only defend one of them.

A backup job is an over-privileged reader. Its whole purpose is to touch everything a tenant owns — every row, every object, every blob — and produce a single artifact containing all of it. There's no principle-of-least-privilege story that makes a backup worker narrow; it's broad by definition. The only question is whether it's broad within one tenant or broad across all of them. In the shared-pool design, it's the second one: the worker holds a credential that can read every bucket, because it has to serve whichever tenant is next in the queue. A prefix bug, an off-by-one in a path join, a cache key that forgot to include the tenant — any of those turns "broad within a tenant" into "broad across the estate," and the artifact you hand a customer contains somebody else's data.

A restore job is an under-scrutinized writer. It takes bytes the customer supplied — bytes that may have originated from your system, or may have been edited in a hex editor at 2am by someone curious — and writes them into your storage. It creates rows, files, object keys, sometimes whole schemas. Every one of those writes is a place where a namespace can be chosen. If the archive gets to influence where things land, then the archive gets to choose the tenant, and you have built a cross-tenant write primitive with a nice progress bar on it.

The anti-pattern: one worker pool, filter by tenant_id

Here's the design almost everyone ships first, and to be clear, I've shipped it too. You have a queue. Jobs arrive with a `tenant_id`. A pool of long-lived workers pops jobs and runs them. The worker holds a credential broad enough to serve any tenant, and correctness comes entirely from the worker remembering to scope every single operation by the tenant on the job it happens to be running right now.

That's a system where isolation is an invariant maintained by discipline, and discipline is not a security boundary. The failure modes are boringly specific:

• Ambient credentials. The worker's storage client is a module-level singleton scoped to the whole estate. Every code path that forgets to pass a prefix inherits access to everything, and it fails open — the wrong-tenant read succeeds silently instead of throwing a 403 you'd notice in staging.
• Process-global state between jobs. A cached connection, a temp directory, an env var, a `chdir` a previous job did and never undid. Job N leaves something behind; job N+1 for a different tenant picks it up. This is the bug class that reproduces once a month and never in a test.
• Leftover files on disk. A restore that crashed halfway leaves an extracted tree in /tmp. The next job's cleanup glob is slightly too generous, or its build step reads a config file it didn't write. You now have a data path between two tenants made entirely of filesystem leftovers.
• Resource contention as a side channel. A decompression bomb from one tenant fills the disk or eats the memory, and every other tenant's job on that box fails or degrades. Denial of service against your other customers, delivered by a zip file.
• The retry that re-targets. A job fails, gets requeued, and the retry path reconstructs context from partially-mutated state — pointing the second attempt at the wrong destination. Restores are exactly the workload where an idempotency bug becomes a data-corruption bug.
• Blast radius of one RCE. If a malicious archive achieves code execution in the worker — and later I'll argue that's much easier than you think — it inherits the estate-wide credential, the queue connection, and network reach to every internal service the worker can see.

None of these require a sophisticated attacker. Most of them are ordinary bugs whose only unusual property is that the code they live in happens to hold the keys to everything. That's the thing to fix — not the bugs, the keys.

One machine, one tenant, one job

The alternative is to make the isolation structural instead of logical: each backup or restore job runs in its own microVM, created for that job, holding a credential scoped to exactly that tenant and exactly that job's needs, and destroyed when the job ends. The tenant boundary stops being a variable you have to remember to check and becomes a hypervisor boundary that holds whether or not your code is correct.

Two properties do the real work here. The first is that the credential is injected at create time rather than baked into a shared worker image. The VM is built fresh, so there is no ambient identity for it to inherit — whatever you write into it is the total set of capabilities it has. Scope the token to one tenant's prefix, give it a TTL slightly longer than the job's timeout, and the worst case of a total compromise is that an attacker gets the access they already had to their own data. The second property is that the machine is disposable. There is no "next job" that could inherit leftover state, because there is no next job on this machine — there's no machine at all a few seconds from now.

The objection to this used to be cost. A VM per job sounds absurd if a VM means a multi-second boot and gigabytes reserved up front. Firecracker plus snapshot-restore changes that arithmetic: on PandaStack a sandbox is created by restoring a baked snapshot rather than cold-booting, the restore step lands around 49ms, end-to-end create is p50 179ms and p99 around 203ms, and only the first-ever cold boot of a template is around 3 seconds. Networking is pre-allocated too — 16,384 /30 subnets per agent, built ahead of time so you're not paying namespace setup on the hot path. At those numbers, a fresh hardware-isolated machine per job costs less than the queue hop that scheduled it.

• Tenant boundary — Shared worker pool: a `tenant_id` variable checked by every code path that remembers to. MicroVM per job: a guest kernel and a credential that only names one tenant.
• Credential scope — Shared worker pool: estate-wide, long-lived, ambient in the process env. MicroVM per job: minted per job, scoped to one prefix, expires shortly after the job's timeout.
• State between jobs — Shared worker pool: temp files, cached clients, chdir, env, page cache — all shared. MicroVM per job: none, because the machine is deleted whole after one job.
• Blast radius of RCE in an archive — Shared worker pool: the estate credential, the queue, and every internal service the worker can reach. MicroVM per job: one tenant's own data and an egress allowlist of two hosts.
• Resource abuse (bombs, fork storms) — Shared worker pool: degrades or kills every other tenant's job on that box. MicroVM per job: hits the guest's own fixed vCPU and RAM and stops there.
• Forensics after an incident — Shared worker pool: correlate interleaved logs from a process that served hundreds of tenants. MicroVM per job: one machine, one tenant, one job, one lifecycle record.
• Cost of a fresh environment — Shared worker pool: high, which is exactly why the pool is long-lived and shared. MicroVM per job: a snapshot restore of roughly 49ms, so per-job is the cheap option.

Restore is running untrusted code, whether you admit it or not

This is the part I want to be loud about, because it's the part that gets rationalized away. Teams tell themselves that restore isn't code execution — it's "just extracting an archive" or "just loading a dump." Then you look at what the tools actually do with the bytes.

Archive extraction: the file decides where it lands

An archive is a list of paths chosen by whoever built the archive, and naive extractors will happily honor them. Zip-slip is the classic: an entry named `../../../../etc/cron.d/oops` extracts exactly where it says, because the extractor joined the archive-supplied path onto your destination without normalizing it. Tar adds symlinks and hardlinks — write a symlink entry pointing at a directory outside the extraction root, then write a regular file entry "through" it, and you've written outside the root in two perfectly legal steps. Absolute paths, device nodes, setuid bits, and names that normalize differently on different filesystems all belong on the same list.

Then there's the size question. A decompression bomb is a few kilobytes on the wire that expands to terabytes on disk, and nested formats multiply. If your extraction runs on a shared worker, the bomb's actual victim is every other tenant whose job was on that box. If it runs in a microVM with a fixed disk and fixed RAM, the bomb fills a machine you were going to delete anyway.

Database dumps: the format is a program

A PostgreSQL custom-format dump is not data — it's a sequence of commands, and `psql` and `pg_restore` are interpreters. My favorite example, because it's so helpfully documented, is `COPY ... FROM PROGRAM`: a legitimate feature that runs a shell command on the database server as the postgres OS user. A dump file containing one is a dump file that executes a program when you restore it. There is also `CREATE EXTENSION`, `DO $$ ... $$` blocks, functions in untrusted procedural languages, event triggers that fire on later DDL, and — my other favorite — a dump that drops or replaces objects the restoring role can reach outside the intended schema.

So "restore a customer-supplied dump" is a sentence that means "execute a customer-supplied program with database superuser adjacency." Which is fine! Lots of systems execute untrusted programs. They just do it inside a sandbox, on a machine with nothing else on it, with credentials that name one database, and they don't call it something soothing like "import."

Building it: a restore worker per job

The shape is straightforward. Create a sandbox with a TTL, write in a credential scoped to exactly one tenant, do the work, read out a result, kill the machine. Notice what's absent: no credential is baked into the template, nothing is mounted from the host, and the `finally` branch kills the VM on every path — success, failure, and the exception you didn't anticipate.

from pandastack import Sandbox
import json

def restore_tenant_archive(tenant_id: str, archive_url: str) -> dict:
    """Restore one tenant's archive inside a machine that exists only
    for this job and holds only this tenant's credentials."""

    # Mint the narrowest credential that can do this job, scoped to one
    # tenant's prefix and one database, expiring just after our timeout.
    creds = mint_scoped_credentials(
        tenant_id=tenant_id,
        storage_prefix=f"tenants/{tenant_id}/",
        database=f"tenant_{tenant_id}",
        ttl_seconds=1200,          # job budget is 900s; small margin, no more
    )

    sbx = Sandbox.create(template="base", ttl_seconds=1800)
    try:
        # Credentials are INJECTED, never baked into the template image.
        # A leaked template gives an attacker nothing; a leaked VM gives
        # them access the tenant already had to their own data.
        sbx.filesystem.write("/run/job/creds.json", json.dumps(creds))
        sbx.filesystem.write("/run/job/tenant", tenant_id)

        # Fetch into a scratch dir. Bounded: --max-filesize stops the
        # 4KB-on-the-wire / 6TB-on-disk genre of archive at the door.
        fetch = sbx.exec(
            "mkdir -p /scratch && "
            f"curl -fsSL --max-filesize 2000000000 -o /scratch/in.tar.gz "
            f"'{archive_url}'",
            timeout_seconds=300,
        )
        if fetch.exit_code != 0:
            raise RuntimeError(f"fetch failed: {fetch.stderr[-2000:]}")

        # Extract with the paranoid flags on. See the next block for why
        # every one of these is load-bearing.
        ex = sbx.exec(
            "mkdir -p /scratch/out && cd /scratch/out && "
            "tar --extract --gzip --file=/scratch/in.tar.gz "
            "    --no-same-owner --no-same-permissions "
            "    --no-absolute-names --no-overwrite-dir "
            "    --one-top-level=payload --exclude='*..*'",
            timeout_seconds=600,
        )
        if ex.exit_code != 0:
            raise RuntimeError(f"extract rejected: {ex.stderr[-2000:]}")

        # Load. This step executes the dump's commands -- which is exactly
        # why it happens here, in a machine with one tenant's DSN and an
        # egress allowlist, rather than in a pooled worker.
        load = sbx.exec(
            "cd /scratch/out/payload && python3 /opt/restore/load.py",
            timeout_seconds=900,
        )
        return {
            "tenant_id": tenant_id,
            "ok": load.exit_code == 0,
            "log": load.stdout[-8000:],
            "err": load.stderr[-8000:],
        }
    finally:
        # Deleted whole, on every path. No temp files, no cached client,
        # no leftover credential for the next tenant's job to inherit --
        # because there is no next job on this machine.
        sbx.kill()

`exec` returns `stdout`, `stderr`, and `exit_code`, and the exit code is what you branch on — an extractor that prints warnings and exits 0 succeeded, and one that prints nothing and exits 1 did not. Every call gets a `timeout_seconds`, because an archive designed to make your decompressor spin forever is a real thing and "still working" and "never finishing" look identical from outside. And `ttl_seconds` on create means a job orphaned by your own orchestrator crashing still reaps itself instead of sitting there holding a credential.

Validate inside the box, not instead of it

The VM boundary is not permission to skip validation; it's what makes validation survivable when it's imperfect. Do both. Normalize and reject archive paths before writing them, cap the expanded size and the entry count, refuse symlinks and hardlinks and device nodes outright unless your format genuinely needs them, and restore dumps with a role that has no superuser and cannot reach `COPY FROM PROGRAM`.

import os, tarfile

MAX_TOTAL   = 20 * 1024**3    # 20 GiB expanded, hard stop
MAX_ENTRIES = 500_000

def safe_extract(archive: str, dest: str) -> None:
    """Runs INSIDE the microVM. Rejects the archive-decides-where-it-lands
    genre. The VM is the backstop; this is the front stop."""
    dest = os.path.realpath(dest)
    total = entries = 0

    with tarfile.open(archive, "r:*") as tf:
        for m in tf:
            entries += 1
            if entries > MAX_ENTRIES:
                raise ValueError("entry count exceeded")

            # Nothing but plain files and directories. A symlink entry plus
            # a later write 'through' it is a two-step escape that each
            # step looks legal on its own.
            if not (m.isfile() or m.isdir()):
                raise ValueError(f"rejected non-regular entry: {m.name!r}")

            # Zip-slip: the archive names the path, so normalize and verify
            # the RESULT is under dest -- don't pattern-match for '..'.
            target = os.path.realpath(os.path.join(dest, m.name))
            if target != dest and not target.startswith(dest + os.sep):
                raise ValueError(f"path escapes destination: {m.name!r}")

            # Decompression bomb: check the declared size, then verify the
            # real one on disk afterwards -- headers are attacker-supplied.
            total += m.size
            if total > MAX_TOTAL:
                raise ValueError("expanded size exceeded")

            m.mode = 0o750 if m.isdir() else 0o640   # no setuid, ever
            m.uid = m.gid = 0
            tf.extract(m, dest, set_attrs=False)

Egress: two hosts and nothing else

A restore worker needs to reach object storage and the target database. That's the entire list. Everything else it could reach is pure downside: your internal admin API, the cloud metadata endpoint that will happily hand out instance credentials, another tenant's database host, an attacker's collection server. Default-deny egress with a two-entry allowlist turns a successful code-execution bug into a process that can compute things and then tell nobody about them.

On PandaStack each sandbox gets its own network namespace with its own veth pair and TAP device, which is the property that makes per-job egress policy tractable — the filtering attaches to one machine's interface rather than to a shared pool's outbound path, so you can write a different policy per tenant without them interfering. The metadata endpoint deserves a special mention: it's the single most valuable thing reachable from a compromised worker, and blocking that one address is the highest-leverage line in the whole ruleset.

# Default-deny egress for one restore job's namespace. Two destinations,
# both required, nothing else. Applied per-sandbox on the host side.

NS=ns-$SANDBOX_ID

# Cloud metadata first -- the highest-value target reachable from a
# compromised worker, and the one people forget because it's link-local.
ip netns exec $NS iptables -A OUTPUT -d 169.254.169.254 -j REJECT

# Allow only object storage and this tenant's database host.
ip netns exec $NS iptables -A OUTPUT -d $OBJECT_STORE_CIDR -p tcp --dport 443 -j ACCEPT
ip netns exec $NS iptables -A OUTPUT -d $TENANT_DB_IP     -p tcp --dport 5432 -j ACCEPT

# DNS to the resolver we control, so lookups can't be a covert channel.
ip netns exec $NS iptables -A OUTPUT -d $RESOLVER_IP -p udp --dport 53 -j ACCEPT

# Established replies, loopback, then deny the rest.
ip netns exec $NS iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
ip netns exec $NS iptables -A OUTPUT -o lo -j ACCEPT
ip netns exec $NS iptables -P OUTPUT DROP

# Sanity-check the deny actually denies. Do this in CI, not in prod at 3am.
ip netns exec $NS timeout 3 curl -sS http://169.254.169.254/ && echo "LEAK" || echo "blocked"

Your backup is a hypothesis until you've restored it

Every team I've talked to has backups. A meaningfully smaller number have ever restored one. In between sits Schrödinger's backup: an artifact whose validity is genuinely unknown until the moment you open it, which is also, by tradition, the worst possible moment to find out. A backup that exists, has a plausible file size, and shows a green checkmark in your job dashboard is a hypothesis. The experiment is the restore.

The reason people don't run the experiment is that it's historically been expensive and scary — you needed somewhere to put the restored data, and "somewhere" meant provisioning a database, which meant a ticket, which meant it happened once a year during an audit. Disposable infrastructure removes the excuse. Create a throwaway managed Postgres (on PandaStack that's 30–90s, since it blocks until the server is genuinely accepting connections), restore last night's backup into it, run assertions that would actually catch a corrupt dump, then delete the whole thing. Do it nightly, on a schedule, for a rotating sample of tenants.

The assertions are where the value is, and "the restore command exited 0" is not one of them. `pg_restore` can exit successfully having created an empty schema. Check row counts against what you expect, check that the newest row is recent enough that you didn't back up a stale replica, check referential integrity on the joins your app depends on, and check that a couple of representative application-level queries return sane results. Then check the thing everyone forgets: that the tenant labels inside the restored data all name the tenant you think you backed up.

#!/usr/bin/env bash
# Nightly restore drill. Prove the backup is real by restoring it into a
# database that exists for four minutes and then ceases to.
set -euo pipefail

API=https://api.pandastack.ai
AUTH="Authorization: Bearer $PANDASTACK_API_KEY"

# 1. A real, throwaway Postgres. Blocks until it's actually accepting
#    connections (30-90s), so there's no readiness guessing game.
DB=$(curl -sS -X POST "$API/v1/databases" -H "$AUTH" \
       -H 'Content-Type: application/json' \
       -d '{"label":"restore-drill"}')
DB_ID=$(jq -r .id <<<"$DB")
DSN=$(jq -r .connection_url <<<"$DB")
trap 'curl -sS -X DELETE "$API/v1/databases/$DB_ID" -H "$AUTH" >/dev/null' EXIT

# 2. Restore last night's backup for one sampled tenant.
pg_restore --no-owner --no-privileges --dbname "$DSN" \
           "backups/$TENANT_ID/$(date -u -d yesterday +%F).dump"

# 3. Assertions that would actually catch a bad backup. "exit 0" is not
#    one of them -- pg_restore will cheerfully create an empty schema.
psql "$DSN" -v ON_ERROR_STOP=1 <<SQL
  \\set QUIET on
  -- Not empty.
  SELECT CASE WHEN count(*) < 1000
    THEN raise_error('row count collapsed: ' || count(*)) END FROM invoices;
  -- Not stale: did we back up a replica that stopped replicating in March?
  SELECT CASE WHEN max(created_at) < now() - interval '36 hours'
    THEN raise_error('newest row is too old: ' || max(created_at)) END FROM invoices;
  -- Not corrupt: the joins the app depends on still resolve.
  SELECT CASE WHEN count(*) > 0
    THEN raise_error('orphaned invoice rows: ' || count(*)) END
    FROM invoices i LEFT JOIN accounts a ON a.id = i.account_id
    WHERE a.id IS NULL;
  -- Not somebody else's: every row names the tenant we backed up.
  SELECT CASE WHEN count(DISTINCT tenant_id) <> 1
    THEN raise_error('multiple tenants in one backup') END FROM invoices;
SQL

echo "restore drill passed for $TENANT_ID"
# trap deletes the database. Total lifetime: about four minutes.

That last assertion is the multi-tenant one, and it's cheap insurance against the exact failure this whole post is about. If a backup job ever leaked across the tenant boundary, the evidence is sitting right there in the artifact — more than one `tenant_id` in a per-tenant export. Checking it nightly means you find out from a failing cron job on a Wednesday, instead of from a customer who opened their export and found a competitor's invoices in it.

A backup you have never restored is not a backup. It's a file with a hopeful name, and you will learn what it actually contains on the worst day of your quarter.

The ephemeral-VM audit story

There's a benefit here that isn't security in the strict sense but is the one that gets you through a customer's security questionnaire: a job that owns a whole machine produces a legible record. One VM, one tenant, one job, with a create and a destroy event bracketing everything that happened in between. "Which tenant's data did this process touch" is answered by the VM's identity rather than by reconstructing it from interleaved log lines emitted by a worker that served four hundred tenants that hour.

• Every job has a machine-lifecycle record: created at T, destroyed at T+n, one tenant, one credential, one scope. That's an audit trail with a shape, not a log grep.
• Credential blast radius is written down in advance. The scoped token you minted at create time IS the documented answer to "what could this job have reached," and it expires on its own.
• Cross-contamination is answerable structurally. "Could job A have seen tenant B's data" has a mechanical answer — different machines, different kernels, different credentials — rather than an argument about code paths.
• Egress policy is per-job and inspectable. The allowlist for a given restore is a concrete artifact you can show someone, not a shared network policy you hope covers this case.
• Data residency and retention get simpler. The plaintext copy of a tenant's data lives on a machine that is deleted when the job ends; there's no worker-node disk quietly accumulating everyone's decrypted exports.

Other platforms have their own isolation stories for background jobs, and you should verify the specifics against their docs rather than my summary of them — the honest comparison is qualitative, not a benchmark table. But the structural question is vendor-neutral and worth asking of whatever you're running today: when a restore job runs, what credential does the process hold, and how many tenants can that credential reach?

Where to start

You don't have to rewrite the pipeline this quarter. There's a natural order, and the first two steps deliver most of the value.

1. Move restore first. It's the direction that writes attacker-influenced bytes and the one that runs untrusted code, so it's where a machine boundary buys the most per unit of work.
2. Stop baking credentials into the worker image. Mint per-job, per-tenant, short-lived credentials and inject them at create time — this alone converts "read every bucket" into "read one prefix," independent of what runs them.
3. Add default-deny egress with an allowlist of the two hosts the job actually needs, and block the metadata endpoint explicitly. Then test that the deny denies, in CI.
4. Move backup jobs next, for the audit story and to kill the estate-wide read credential.
5. Schedule the restore drill. Nightly, sampled, into a throwaway database, with assertions that would catch a corrupt or cross-contaminated backup — including the count of distinct tenant IDs.
6. Delete the shared worker pool. It's the thing that was holding the keys; the whole point was never to have a long-lived process that can reach everything.

The mental model I'd leave you with is this: backup and restore is not data plumbing, it's the execution of a customer-supplied program against your storage layer with credentials you chose. Treated that way, every decision falls out naturally — narrow the credential, isolate the machine, allowlist the egress, delete it afterward, and prove the whole thing works by actually doing it on a random Tuesday when nothing is on fire. The alternative is finding out how it all behaves on the day you most need it to behave, which is a research method with an extremely poor track record.