One microVM per DSAR: exporting a person's data without leaking someone else's

Somewhere in your estate there is a job that queries the primary database, then the warehouse, then object storage, then the log store, then four third-party processors, joins everything it finds by a single person's identifiers, and writes the result into one archive. Then it emails a link to that archive to whoever asked. You built this on purpose. You built, deliberately and with a ticket number, exactly the artefact an attacker would spend six months trying to assemble, and then you set up a delivery mechanism for it. It is called a data-subject access request, it is a legal obligation, and it is the single most sensitive process you run.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, and DSAR pipelines keep showing up in conversations that started out being about something else. The shape is always the same: a perfectly reasonable worker on a queue, handling requests one after another, because that is how you write a worker. This post is about why that specific job wants a machine of its own — one that is created for one subject, holds one person's data, and ceases to exist before the next request is scheduled.

What a DSAR export actually is, mechanically

Strip away the compliance vocabulary and an Article 15 export is a distributed join with a very awkward join key. Somebody claims to be a data subject, you verify that claim, and then you have to find everything you hold about them. "Everything" is not a table. It is spread across systems that were designed independently, key the same human differently, and in several cases were never designed to be queried by person at all.

1. Resolve identity. One email becomes a user id, a customer id, three device ids, a Stripe customer, a support-desk contact, and a hashed identifier in the analytics warehouse. Getting this wrong in either direction is the whole game: too narrow and the export is incomplete, too broad and you have just disclosed a stranger's data to the requester.
2. Query the primary store. Rows across dozens of tables, plus the foreign keys that make those rows meaningful, plus soft-deleted records that are still personal data because you still hold them.
3. Query the warehouse. Event streams, sessions, derived tables, and whatever your analytics team materialised at 3am in 2023 and never documented.
4. Walk object storage. Uploaded avatars, attachments, exported reports the user generated themselves, and the CSV somebody dropped in a bucket during a migration.
5. Scrape logs. Application logs contain personal data whether or not you intended them to. So do error traces, so do webhook payload archives.
6. Ask third-party processors. Your CRM, your email vendor, your support tool, your session-replay product. Each has its own API, its own latency, and its own opinion about what "all data for this person" means.
7. Join, normalise, and serialise into something a human can read and a machine can parse, because portability expects a structured, commonly used, machine-readable format rather than a screenshot of a dashboard.
8. Package, encrypt, deliver, and record that you did all of the above.

The output is a single file that contains, in one place, a person's identity, their behaviour, their purchases, their support history, their IP addresses, and often their location trail. No production system holds that dossier in assembled form. Your export job creates it from nothing, holds it in memory and on disk for a few minutes, and then hands it over. That is the job. Everything below is about the fact that this file exists at all, and about the machine it exists on.

A DSAR pipeline is a supervised, audited, legally mandated data-exfiltration tool that you operate against yourself. The only question is whether it exfiltrates to the right person.

The failure mode: subject A's rows inside subject B's archive

Most threat modelling on DSAR pipelines fixates on identity verification, which is correct but incomplete. The failure that actually shows up in incident writeups is duller and more mechanical: a long-lived worker processes subject A, then subject B, and something belonging to A survives into B's output.

There is nothing exotic about how this happens. It happens because the process is a normal Unix process doing normal things, and normal things persist:

• Module-level state. A cached connection, a memoised identity map, a `_LAST_RESULT` somebody added for debugging, a dataframe held by a global. Python does not forget between queue messages; only you do.
• The working directory. `/tmp/export/` from the previous run, half-written because the run before crashed. A glob that says `*.jsonl` does not care which subject wrote those lines.
• Partially written archives. If run A died after opening a zip and writing three members, and run B appends to a file it found at the same path, the resulting archive is a very specific kind of catastrophe.
• Pandas and friends. Frames, indexes, and pyarrow buffers survive as long as a reference does, and library-level caches are not cleared by your `finally` block because you did not write them.
• The page cache. You deleted the file. The kernel still holds the contents until it needs the memory. Anything on that machine that can read memory can, in principle, read them.
• Error reporting. A crash mid-export sends a stack trace to your error tracker, and stack traces carry local variables. Subject A's rows now live in a SaaS product that is not on your processor list.
• Log interleaving. Two concurrent exports in one process write to the same log stream. Which line belonged to which subject is now an inference, and inference is not an audit trail.

The per-request microVM answer is not clever. It is that the process which touched subject A's data does not exist when subject B's job starts. Not "was cleaned", not "was reset", not "had its globals reinitialised by a fixture". The guest kernel, its page cache, its filesystem, its heap and its process table are gone, because the virtual machine they lived in was destroyed. Cross-contamination is not prevented by a code path you have to keep correct; it is prevented by the absence of a shared substrate.

"We clean up /tmp" is a different claim from "the machine is gone"

Every team that has thought about this at all has a cleanup step. The cleanup step is usually correct for the case it was written for, and the case it was written for is "the job finished normally". Here is the gap between deleting files and not having a machine:

• Deletion is a directory operation, not an erasure. `unlink` removes a name. The blocks keep their contents until something else claims them, and on a copy-on-write or log-structured filesystem "something else" may be a long time coming.
• Crashed runs skip the cleanup. `finally` does not run when the process is OOM-killed, when the node is drained, or when a library calls `os._exit`. Those are exactly the runs that leave the largest and least expected residue.
• Swap. If the box is under memory pressure, some of subject A's dataframe is now on a disk you were not thinking about, in a file your cleanup script has never heard of.
• Core dumps. A segfault in a native extension writes the process's memory to disk, by design, in a location set by a sysctl nobody on your team has read.
• Freed memory is not zeroed memory. `free()` returns pages to an allocator, not to a shredder. A later allocation in the same process can be handed pages that still contain the previous subject's rows, and a use-of-uninitialised-memory bug turns that into a disclosure.
• Shared caches by design. Connection pools, HTTP client caches, DNS caches, and any local disk cache you added to make the warehouse queries cheaper. All of them are per-process or per-node, none of them are per-subject.
• Your cleanup enumerates. Which means it can be incomplete, and you will only discover the omission by finding the thing it did not delete.

None of these individually is likely to produce an incident. Collectively they are why "the machine no longer exists" is a categorically stronger statement than "we removed the files we knew about". A microVM per request converts a housekeeping discipline into a lifecycle fact.

The shape: one microVM per request

The unit of isolation should match the unit of sensitivity, and here the unit of sensitivity is one human being. So: one VM per DSAR, created when the request is scheduled, destroyed when the archive is sealed.

That only works if creating a machine is cheap enough to stop being an architectural decision. On PandaStack a sandbox is created by restoring a pre-baked Firecracker snapshot rather than cold-booting — p50 179ms, p99 203ms, with the restore step itself around 49ms — and there is no warm pool of idle VMs sitting between requests holding anything. Only the very first boot of a template costs about three seconds. Per-request isolation at that price is cheaper than the code review you would otherwise spend arguing about global state.

1. Verify identity first, outside the VM, in your normal control plane. The export job should never be the thing deciding whether the requester is who they say they are.
2. Create the VM with the request id and a pseudonymous subject handle in metadata. Do not put the subject's email in the sandbox name; you will read it in a dashboard six months from now for no reason.
3. Assert the egress shape before any collector runs, and fail closed.
4. Mint read-only, per-request, short-lived credentials for each source system and pass them on the exec call. Never in the template, never in a dotfile, never in a snapshot.
5. Collect per source, one process per system, one output file per system, so failures are attributable and the access log has a shape.
6. Assemble and encrypt inside the guest, to a key the requester controls. The plaintext archive should never exist outside this machine.
7. Copy ciphertext out over the host-side filesystem channel. The guest gets no bucket credential it could use to push data anywhere.
8. Kill the VM. Set a TTL as the backstop for the runs nobody closed.

# dsar_export.py -- one Firecracker microVM per data-subject access request.
# Subject A's rows and subject B's rows never share a machine, because the
# machine that held A's rows is destroyed before B's request is scheduled.
import json
import shlex
from pandastack import Sandbox

COLLECT = """#!/bin/bash
set -uo pipefail
mkdir -p /work/out
cd /work

# Assert the network before anything touches subject data. A collector
# that cannot reach an arbitrary destination cannot accidentally become
# one, and a policy gap found here costs nothing.
bash /work/preflight.sh > /work/out/preflight.log 2>&1 || exit 78

# One collector per system, one output file per system. Attribution is
# the point: when a regulator asks which stores were searched, you want
# a per-source record rather than one interleaved application log.
python3 collectors/primary_db.py   > /work/out/primary.jsonl
python3 collectors/warehouse.py    > /work/out/warehouse.jsonl
python3 collectors/object_store.py > /work/out/objects.jsonl
python3 collectors/app_logs.py     > /work/out/logs.jsonl
python3 collectors/processors.py   > /work/out/processors.jsonl

# Join + normalise. assemble.py also records the systems that returned
# NOTHING -- "we searched and found no rows" is an answer a regulator
# will accept; a missing section is a question you have to answer later.
python3 assemble.py /work/out > /work/out/manifest.json

# Encrypt to the requester's public key, inside the guest. The plaintext
# archive exists only in this VM's filesystem, for the length of one job.
tar -C /work/out -cf - . | age -R /work/recipient.pub > /work/export.age
sha256sum /work/export.age > /work/export.sha256
"""


def run_dsar(req_id: str, subject_handle: str, recipient_pub: str) -> dict:
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=3600,              # backstop, not the plan
        metadata={
            "job": "dsar-export",
            "request_id": req_id,
            "subject": subject_handle,  # pseudonymous handle, not an email
            "article": "15",
        },
    )
    try:
        sbx.filesystem.write("/work/recipient.pub", recipient_pub)
        sbx.filesystem.write("/work/preflight.sh", open("preflight.sh").read())
        sbx.filesystem.write("/work/collect.sh", COLLECT)

        # Credentials are minted for THIS request, read-only, expiring on
        # the order of this job's timeout, and passed on the exec call.
        # Not baked into the template, not written to a file a collector
        # bug could echo into the archive, not present before this line.
        creds = mint_scoped_readonly_credentials(req_id, ttl_seconds=1800)
        env = " ".join(f"{k}={shlex.quote(v)}" for k, v in creds.items())
        r = sbx.exec(
            f"env {env} SUBJECT={shlex.quote(subject_handle)} "
            f"bash /work/collect.sh",
            timeout_seconds=1500,
        )

        if r.exit_code != 0:
            # A half-written archive is not a partial success. Nothing is
            # copied out, and the machine holding the fragments is killed
            # in the finally block below -- fragments included.
            raise ExportFailed(req_id, r.exit_code, r.stderr[-4000:])

        # Ciphertext leaves over the host-side filesystem channel. The
        # guest never held a credential that could push data anywhere.
        blob = sbx.filesystem.read("/work/export.age")
        digest = sbx.filesystem.read("/work/export.sha256").decode()
        manifest = json.loads(sbx.filesystem.read("/work/out/manifest.json"))
        return {
            "request_id": req_id,
            "bytes": len(blob),
            "sha256": digest.split()[0],
            "sources_searched": manifest["sources"],
            "ciphertext": blob,
        }
    finally:
        # End of job, end of machine. Not a cleanup script: the guest
        # kernel, its page cache, its /tmp and its heap stop existing.
        sbx.kill()

Two details carry most of the weight. The plaintext archive is assembled and encrypted inside the guest, so the unencrypted dossier never crosses a boundary you would have to audit. And the credentials arrive on one exec call, which means the window in which they exist is bounded by a command rather than by a machine's uptime.

Rules for the credentials

• Read-only for the export phase. An Article 15 job has no business holding a credential that can write, let alone delete. Enforce that in the grant, not in the code.
• Scoped to the subject where the source system allows it. Row-level security, a per-request database role, or a view parameterised by subject id turns "the collector had a bug" into "the collector got zero rows" rather than "the collector exported the table".
• Short-lived. Mint on schedule, expire on the order of the job timeout. A leaked credential measured in minutes is a different conversation from one measured in months.
• Never in the template or the snapshot. Anything baked into an image is available to every future run of that image, which is precisely what you are trying to avoid.
• No platform identity in the guest at all. No object storage key, no internal service token, no instance metadata reachable. Results come out over the host-side filesystem API, which the guest cannot authenticate to and therefore cannot abuse.
• Separate credentials for erasure. The delete-capable grant is minted at a different phase, for a different command, and should never coexist with the collection phase.

Narrow egress, asserted before the first row is read

Isolation decides what a compromised job can reach on your host. Egress control decides whether it can reach anything worth reaching at all. A DSAR job's network needs are unusually enumerable — you know exactly which systems it must read, because you wrote the collector list — so default-deny with a per-job allowlist is realistic here in a way it rarely is elsewhere.

On PandaStack each sandbox gets its own network namespace, out of 16,384 pre-allocated /30 subnets per agent, so "this job may reach the warehouse, the primary replica, and three processor APIs, and nothing else" is a property of one machine rather than a firewall rule somebody has to remember to remove afterwards. Assert it from inside the guest before the collectors run, and fail closed if the shape is wrong.

#!/bin/bash
# preflight.sh -- runs inside the DSAR VM before any collector does.
# Asserts the network this job is allowed to see. Finding a policy gap
# here is free; finding it in a stack trace containing subject data is
# a notification exercise.
set -uo pipefail
fail=0

# 1. The systems this export MUST read. Generate this list from the
#    collector manifest for this request -- not from a wildcard somebody
#    added during an incident in March and never took out again.
for host in db-replica.internal warehouse.internal objects.internal \
            api.crm.example api.support.example; do
  if curl -sS -o /dev/null --max-time 5 "https://$host/healthz" 2>/dev/null; then
    echo "allow   $host"
  else
    echo "BLOCKED $host  <-- allowlist wrong; export would be INCOMPLETE"
    fail=1
  fi
done

# 2. What must be unreachable. The metadata endpoint is the one that
#    turns "a bug in a collector" into "holds the host's own identity".
#    The error tracker is on this list on purpose: a crashed export must
#    not ship a stack trace full of subject rows to a third party.
for target in 169.254.169.254:80 metadata.google.internal:80 \
              ingest.errortracker.example:443 pastebin.example:443; do
  host="${target%%:*}"; port="${target##*:}"
  if timeout 3 bash -c "exec 3<>/dev/tcp/$host/$port" 2>/dev/null; then
    echo "REACHABLE $target  <-- aborting before any subject data is read"
    fail=1
  else
    echo "denied  $target"
  fi
done

# 3. Fail closed. Exiting here aborts the run while the machine still
#    contains nothing more sensitive than a shell script.
exit "$fail"

Three places to run a DSAR export

Same job, three topologies. Treat any characterisation of a specific product's isolation or pricing behaviour as something to verify against that vendor's current documentation, since those details vary by configuration and change over time.

• Residual data between subjects — Shared worker pool: subject A's rows persist in process memory, module globals, library caches, the working directory, swap and the page cache, and the cleanup step is skipped exactly when a run crashes. Container per job: fresh filesystem and process namespace, which handles the obvious residue, but the host kernel, its page cache, the node's swap and any mounted shared volume are still common ground. microVM per job: own guest kernel, own memory, own filesystem, all destroyed with the machine; residue is not cleaned up, it stops existing.
• Credential blast radius — Shared worker pool: one long-lived process environment holding whatever every source system needs, readable by any code that ends up running in it. Container per job: env is per-container, but the node's instance metadata and mounted service-account tokens are usually one hop away. microVM per job: credentials arrive on a single exec call, scoped read-only to one request, expiring near the job timeout, with no platform identity present in the guest at all.
• Audit granularity — Shared worker pool: concurrent requests interleave in one log stream, so "what did we access for this person, and when" is reconstructed by filtering rather than read off. Container per job: better, if you remembered to label everything and ship logs per container. microVM per job: the request is an object with a lifecycle — created, exec'd, read, killed — with the request id in metadata, so the answer is a lookup rather than an inference.
• Failure containment — Shared worker pool: a crash mid-export leaves a partial archive on shared disk and may ship locals to an error tracker. Container per job: partial archive dies with the container, but shared volumes and node-level dumps do not. microVM per job: a failed run copies nothing out and the fragments go with the machine, while egress policy keeps the crash report from leaving in the first place.
• Cost — Shared worker pool: cheapest in raw compute, most expensive in the tail, because the failure mode is a notifiable breach rather than a retry. Container per job: cheap, with orchestration overhead and a kernel you share whether you wanted to or not. microVM per job: you pay for a machine per request — on PandaStack, $0.054 per active vCPU-hour and $0.0162 per GiB-hour, billed per second, with idle costing effectively nothing — which for a job that runs for minutes and is measured in requests per day is not the line item anyone will question.

The audit trail: one request, one machine, one record

When a regulator or a data subject asks what you accessed on their behalf and when, the quality of your answer is decided months earlier by your architecture. A shared worker gives you a log stream you have to filter, from a process that was doing several people's requests at once, with retries interleaved. A per-request VM gives you a bounded object with a lifecycle, and "which machine handled this request" stops being an inference from timestamps.

• The request id, the pseudonymous subject handle, and the sandbox id, joined. That triple is the spine of the record.
• Lifecycle events for the VM itself: created at, first exec, last exec, killed at. This is your access window, stated precisely, without having to trust an application log.
• The collector manifest, including the systems that returned nothing. Completeness is a claim you have to be able to evidence, and empty results are evidence.
• The egress allowlist that was in force, plus every denial. Both what the job was permitted to reach and what it tried to.
• The archive's SHA-256 and its size, so "the file we delivered" is identified rather than described.
• Delivery: the recipient key fingerprint, the download timestamp, and the expiry of the link. A dossier behind a link that never expires is a dossier you have published on a slow timer.
• For erasure: the plan, the journal, and the checkpoint snapshot id — see below.

Erasure is the same shape, and it is worse

An Article 17 erasure job is architecturally the twin of the export job: it resolves the same identities, enumerates the same systems, and touches the same rows. The difference is that it is destructive, it runs against production, and there is no undo. Every property that makes cross-contamination bad in an export makes it catastrophic in an erasure. A stale identity map that leaks subject A's rows into B's export is a breach; the same stale map in an erasure run deletes a different customer's account.

So the erasure job gets the same per-request VM, plus three things the export does not need: a dry run that resolves without deleting, a checkpoint before the destructive phase, and idempotency that survives being interrupted halfway.

# erasure.py -- Article 17 has the same shape as Article 15, plus the
# part where it is destructive and there is no undo button anywhere.
import json
from pandastack import Sandbox


def erase(req_id: str, subject_handle: str) -> dict:
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=7200,
        metadata={"job": "erasure", "request_id": req_id, "article": "17"},
    )
    try:
        sbx.filesystem.write("/work/preflight.sh", open("preflight.sh").read())
        sbx.filesystem.write("/work/erase.py", open("erase.py").read())

        # PHASE 1 -- DRY RUN. Resolves every row, object and log line that
        # WOULD be touched and writes an explicit plan. The credential for
        # this phase has no delete permission at all, so "the dry run was
        # accidentally not dry" is not a state this program can reach.
        read_env = env_string(mint_scoped_readonly_credentials(req_id, 1800))
        dry = sbx.exec(
            f"env {read_env} python3 /work/erase.py --plan --out /work/out",
            timeout_seconds=1200,
        )
        plan = json.loads(sbx.filesystem.read("/work/out/plan.json"))
        if dry.exit_code != 0 or plan["blast_radius"] > plan["expected_max"]:
            # Resolving more rows than a single person should have is the
            # signature of a bad identity join. Hold it for a human.
            return {"request_id": req_id, "status": "held", "plan": plan}

        # PHASE 2 -- CHECKPOINT. Snapshot the machine holding the resolved
        # plan. If the apply dies at target 400 of 900, you restore this
        # and resume from a known list instead of re-resolving a world you
        # have already half-deleted. Same-host restore lands in 400-750ms.
        checkpoint = sbx.snapshot()

        # PHASE 3 -- APPLY. A second, delete-capable credential, minted
        # now, for this command only. Each target carries an idempotency
        # key, and the journal line is written BEFORE the delete, not
        # after -- so a crash leaves "maybe done", never "silently missed".
        write_env = env_string(mint_delete_credentials(req_id, 3600))
        run = sbx.exec(
            f"env {write_env} python3 /work/erase.py --apply "
            f"--plan /work/out/plan.json --journal /work/out/journal.jsonl "
            f"--idempotency-key {req_id}",
            timeout_seconds=5400,
        )

        return {
            "request_id": req_id,
            "status": "done" if run.exit_code == 0 else "partial",
            "checkpoint": checkpoint.id,
            "journal": sbx.filesystem.read("/work/out/journal.jsonl").decode(),
        }
    finally:
        sbx.kill()

The checkpoint deserves a note, because it is the part people skip. The expensive, fragile, non-deterministic step in an erasure run is resolution: working out precisely which rows, objects and log entries belong to this person, across systems whose contents are changing while you look at them. Once you have that plan, it is the most valuable artefact in the job. Snapshotting the VM that holds it means an interrupted apply resumes against the same list rather than re-deriving it from a dataset you have already partly destroyed — a re-derivation that is, by construction, no longer reproducible.

• Dry run, always, and diff the plan against the previous request's shape. A sudden order-of-magnitude change in resolved targets is an identity-join bug announcing itself.
• Journal before you delete, not after. An entry that says "about to delete X" survives a crash; one written after the fact does not exist for the delete that killed the process.
• Idempotency keys per target, so a resumed run is safe to re-issue against a system that already processed it.
• Never wire the plan and the apply to the same credential. Read and destroy are different powers and should be different grants.
• Erasure of a record is not erasure of its derivatives. Aggregates, backups, search indexes, caches and replicas each need their own decision, and "the backup restores it in six weeks" is a real failure mode.
• Notify downstream recipients. If you shared the data, telling the people you shared it with is part of the job, and that is another set of API calls with another set of credentials.

What this does not buy you

This is an engineering post, not legal advice, and the microVM is not a compliance control panel. It reduces exactly one class of risk: data from one subject persisting on a machine into another subject's job, and credentials from one job persisting into the next. That is a real class and it is the one most likely to produce a self-inflicted breach. It is not the whole problem, and it would be dishonest to imply otherwise.

• Identity verification is still yours. Sending a complete, perfectly isolated dossier to an impostor is a worse outcome than any contamination bug, and the VM has no opinion about it.
• Third-party processors are still yours. Your isolation stops at your perimeter; what your CRM vendor does with a subject-data API call is governed by a contract, not by a hypervisor.
• Backups and replicas are still yours. An erasure that does not account for point-in-time backups, read replicas, and search indexes is an erasure with a scheduled reversal.
• The statutory clock is still yours. The response window for an access request is one month from receipt, extendable by two further months where the request is complex or numerous, provided you tell the person within that first month and explain why. Your pipeline's throughput is an engineering input to a legal deadline, not a substitute for tracking it.
• Deletion of the archive after delivery is still yours. The export you generated is now a file somewhere. Give the link an expiry, encrypt to the requester's key, and delete the ciphertext on a schedule you can evidence.
• The logs of the job itself are still yours. It is entirely possible to build a beautifully isolated export pipeline whose own debug logging quietly reconstructs the dossier in your log store.

What you get in exchange is narrow and worth having. The sentence "we assemble a complete profile of a named individual and write it to a file" stops being an uncomfortable admission in a design review and becomes a description of a bounded process: one subject, one machine, one short-lived read-only credential, one enumerated network, one encrypted artefact, and a machine that no longer exists. The dossier still gets built, because the law says it must. It just has nowhere to linger and nobody to leak into.