Running Code Over PHI Without Expanding Your HIPAA Blast Radius

The feature is never written up as a compliance change. A clinician uploads a spreadsheet of patients and asks a question in English, and an agent writes some pandas and runs it. A customer wants their own transform applied to an HL7 feed before it lands. A product manager wants an OCR pass over scanned referral faxes, or an NLP model to pull medications out of discharge summaries, or a notebook the data science team can point at the warehouse. Every one of those is the same request underneath: take protected health information out of the store you designed very carefully, and hand it to a process you did not.

I'm Ajay; I build PandaStack, an open-source Firecracker microVM platform, and healthcare-adjacent teams keep arriving with a version of this question. They have usually done the obvious things — the database is encrypted, the bucket has a policy, access is behind SSO — and then noticed that none of it describes the ninety seconds their code is actually holding the records. This post is about that window: where PHI physically ends up when you execute code over it, and an architecture that makes the answer short enough to write down.

What the Security Rule actually asks of an execution platform

The Security Rule organizes into administrative, physical and technical safeguards, and most engineers only ever read the third bucket. That is worth naming early — the administrative half holds the risk analysis, the workforce training and the business associate agreements, which are the artifacts an investigator asks for first. But the technical safeguards are the ones that bite an execution platform, and four of them map onto design decisions you are making anyway: access control, audit controls, integrity, and transmission security.

Then there is the one everybody under-thinks, and it is not a technical safeguard at all. The Privacy Rule's minimum necessary principle limits uses and disclosures to what is reasonably needed for the purpose. Engineers read that as an authorization concern and move on. Applied to code execution it is far sharper: does this job need the whole table, or one patient? The full record, or three fields? Most PHI jobs I look at are handed vastly more than they consume, because handing over a whole dataset is one line of code and scoping it is a conversation.

One honest note on the standard's vocabulary. Some implementation specifications are marked required and some are marked addressable, and encryption of ePHI at rest is addressable. Addressable does not mean optional. It means you assess whether it is reasonable and appropriate, and if you conclude it is not, you document why and implement an equivalent alternative measure. In practice, encrypt it. Nobody wants to be the person explaining a documented decision not to encrypt patient data to an investigator who is already reading about it.

Where PHI actually goes when you run code over it

Here is the part that almost nobody writes about, and it is the technical heart of this. Your data flow diagram shows PHI moving from a store to a compute node and back. That diagram is wrong — not misleading, actually wrong — because loading a record into a process creates copies of it in at least five places, most of which outlive the request, and several of which leave the machine entirely.

Process memory, and therefore your snapshot store

The first copy is the obvious one: the record is in the process's memory. Ordinarily that is fine and nobody thinks about it, because the process exits and the pages get recycled. It stops being fine the moment your platform grows a snapshot feature — and every serious sandbox platform grows one, because snapshots are how you get fast starts, and forks are how you branch an agent's state.

A snapshot of guest memory is a byte-exact copy of everything the process was holding at the instant you took it. Not a reference to it, not a summary of it: the bytes. If a patient's chart was in a parser buffer, a decrypted request body, a Python object, or a not-yet-freed allocation, those bytes are in the file. It is a photograph of everything the process was thinking, stored in a bucket, forever. And unlike a database row, nothing about it looks like PHI — it looks like a several-gigabyte blob named vm.mem, which is exactly why it gets replicated to another region for faster restores, copied into staging so someone can reproduce a crash, and retained under whatever the default lifecycle policy on that bucket happens to be.

So say it plainly: if you snapshot or fork machines that have loaded PHI, your snapshot store is a PHI store. It needs encryption at rest, access control, an entry in your data inventory, a retention schedule, deletion you can evidence, and a line in your risk analysis. The much easier path is the one the architecture should push you toward — snapshot the clean template before any patient data exists, restore per job, and never capture a machine that has seen a record. Note that hibernation is a snapshot with a friendlier name, and so is live migration.

Two quieter members of the same family: swap, which writes anonymous memory to a device that outlives the process that owned it, and core dumps, where a segfault in a DICOM or PDF parser writes the whole address space to disk and a crash handler helpfully forwards it somewhere. Turn both off in any guest that holds PHI — two lines of configuration that close a category.

Temp files, and whatever the library wrote while parsing

The second copy is on disk, and you did not write the code that put it there. Imaging libraries decompress pixel data to a temp file. OCR pipelines rasterize pages before reading a single character. Dataframe libraries spill under memory pressure. Archive extraction, font caches and every wrapper that shells out to a command-line tool use the filesystem as scratch, because that is what the filesystem is for.

On a long-lived worker that residue survives the request by however long the machine lives, and unlinking a file removes neither the page cache copy nor the underlying blocks. This is the unglamorous way PHI ends up on a node your data flow diagram says never stores anything.

Logs: the most common accidental disclosure in the industry

The third copy is the one that actually causes incidents. A parser hits a malformed record and raises, the framework's exception handler serializes the offending object into the message, and the record is now a log line. The stack trace is a HIPAA incident with syntax highlighting. Somewhere in a healthcare codebase right now there is a line that reads logger.info(patient) — added during a debugging session, never removed — and it is the most expensive line of code in that repository.

Three things compound. Logs are designed to leave the machine, so this is the one leak with delivery built in. They usually leave to a third-party SaaS with its own retention and replication, and possibly no business associate agreement, which turns a logging config into an impermissible disclosure. And they are indexed and searchable by your whole engineering organization — a minimum-necessary problem even when the vendor is fully papered.

The fix is structural rather than a redaction regex bolted on at the end. Denylists work beautifully until a free-text note contains something shaped like an address. An allowlist fails toward missing data; a denylist fails toward a breach notification.

# WRONG. Every one of these is a real pattern from a real healthcare codebase,
# and every one of them ships PHI to your log vendor.
logger.info(f"failed for {patient}")            # the whole record, stringified
logger.warning("bad dob: %s", row["birth_date"])
logger.debug("payload=%r", request.json)        # the entire submitted chart
try:
    parse_hl7(msg)
except Exception:
    logger.exception("parse failed")            # traceback may embed the message


# RIGHT. Stable event keys plus identifiers that are meaningless outside your
# own database. The log says what happened and which record it happened to.
# It never says what was IN the record.
log.event(
    "phi.parse.failed",
    record_id=record.id,          # your surrogate key, not an MRN, not an SSN
    patient_ref=patient.surrogate_id,
    field="birth_date",           # WHICH field was bad, never its value
    reason="unparseable_date",    # a stable enum, not the parser's message
    execution_id=execution.id,    # ties back to the audit trail below
)

# When you need the parser's own message, keep it inside the boundary and
# hand the operator a pointer instead of the contents.
except HL7ParseError as e:
    detail_id = phi_detail_store.put(str(e))    # inside the PHI boundary
    log.event("phi.parse.failed", record_id=record.id, detail_ref=detail_id)

# And make the wrong thing hard rather than forbidden by code review: give
# your PHI model types a __repr__ that refuses to render, so an f-string or a
# traceback that reaches for one gets a placeholder instead of a chart.
class Patient:
    def __repr__(self) -> str:
        return f"<Patient {self.surrogate_id} redacted>"
    __str__ = __repr__

And if the code calls a model, the prompt is a disclosure

If the job passes any part of a record to an inference API, that is a disclosure to a subprocessor, and the questions are the same ones you would ask about any vendor: is there a business associate agreement available and executed, what are the retention terms for prompts and completions, is request logging on by default, does abuse monitoring retain content, and where geographically does any of it sit. Terms differ by provider, by plan and by API surface, and they change — confirm the current terms with the vendor rather than trusting a blog post, including this one.

Agent architectures make this worse in a way that is easy to miss. The scratchpad accumulates and tool outputs get echoed back into the next prompt, so a record that was legitimately in context for one step is still in the transcript six steps later when the agent calls a different tool with a different retention policy. Trace where the context object goes, not just where you first put the data.

• Process memory — the record lives in the heap for the life of the process, and any snapshot, fork, hibernation or live migration copies it verbatim into a file — snapshot only clean templates, never a machine that has loaded PHI, and treat any existing snapshot store as a PHI store with full encryption, access control, retention and deletion evidence.
• Swap and crash dumps — the kernel writes anonymous memory to a device, or a segfault writes the whole address space to disk and a handler forwards it — disable swap in the guest and set core_pattern to discard, in the image, not in a runbook.
• Temp files and page cache — libraries you did not write spill decompressed images, rasterized pages and intermediates to disk, and unlink does not scrub blocks — make scratch a small tmpfs that dies with the machine, so there is no persistent device for residue to land on.
• Application logs — exception handlers serialize the record they choked on, and the pipeline ships it to a third-party index with different retention — emit structured events with an allowlist of fields, never format a record into a message string, and confirm a BAA with the telemetry vendor before anything leaves.
• Inference API calls — prompts, tool outputs and accumulated agent context are disclosures to a subprocessor whose retention you do not control — send the minimum span of text needed, prefer de-identified input, and confirm BAA availability and zero-retention terms in writing.
• The result path — outputs, caches, queues, error-reporting payloads and the metadata you attach to the job all cross the boundary — return only derived values, and keep job metadata to surrogate identifiers because metadata shows up in list APIs and operational dashboards.

The design: a machine that exists for exactly one execution

Every leak above has the same root: the machine outlives the job. So make it not. The shape is a microVM created for one execution, with no persistent disk, no egress except an allow-listed internal endpoint, PHI passed in over an audited channel, results passed back out over the same channel, and the machine destroyed at the end. Residency is then measured in seconds and enforced by the platform's TTL rather than by a cleanup routine.

That last distinction is worth sitting with, because it is the whole compliance argument. "The data was deleted afterwards" is a claim about a process that has to run correctly — including on the unhappy path, which is precisely the path where cleanup routines do not run, because the orchestrator crashed or the pod got evicted or somebody killed the job. Evidencing it means showing the deletion job, its logs, its failure handling, and its coverage. "The data was never at rest" is a claim about the architecture: there was no persistent device attached, so there is nothing to have failed to delete. The first is a control you operate. The second is a property you can point at, and pointing at properties is a much shorter meeting.

The historical objection was that a VM per job is absurdly expensive, and when a VM took thirty seconds to appear it was. Snapshot-restore removes that. On PandaStack there is no warm pool of idle machines: every create restores a pre-baked template snapshot on demand, around 179ms p50 and 203ms p99, with the roughly three-second cold boot paid once at bake time rather than per request. Forking a warmed machine on the same host is 400-750ms, and 1.2-3.5s across hosts. Each agent also pre-allocates 16,384 network slots, so the ceiling on concurrent executions is host memory and CPU rather than plumbing.

from pandastack import Sandbox

# One execution, one machine. Note what is deliberately absent from this
# guest: your warehouse DSN, your object-store credentials, your KMS key,
# any other patient's data, and any route to the internet. A bug in the
# parsing library owns a throwaway machine holding one patient's records,
# with nowhere to send them, for three minutes.

ANALYSIS_TIMEOUT_S = 120


def run_over_phi(execution_id: str, patient_ref: str, records: bytes,
                 user_code: str) -> dict:
    sbx = Sandbox.create(
        template="code-interpreter",
        # A retention control the platform enforces, not just a safety net.
        # If this process dies mid-execution, the machine still dies.
        ttl_seconds=180,
        metadata={
            # SURROGATE IDENTIFIERS ONLY. Metadata is queryable, comes back
            # from list APIs, and is printed in operational dashboards that
            # sit outside the PHI boundary. A name, an MRN, a date of birth
            # or a diagnosis code does not go here -- not once, not "just
            # for debugging", not truncated.
            "execution": execution_id,
            "patient_ref": patient_ref,
            "class": "phi",
        },
    )
    try:
        # Scratch is a tmpfs that dies with the guest; there is no persistent
        # volume in this machine. Minimum necessary applies here too -- pass
        # the columns the job consumes, not the whole export.
        sbx.filesystem.write("/work/records.ndjson", records)
        sbx.filesystem.write("/work/job.py", user_code)

        run = sbx.exec(
            "cd /work && python job.py --in records.ndjson --out result.json",
            timeout_seconds=ANALYSIS_TIMEOUT_S,
        )

        # Fail closed, and do not put the guest's stderr in a log line -- it
        # is written by code that was holding patient data and it will
        # eventually contain some.
        if run.exit_code != 0:
            raise ExecutionFailed(execution_id, run.exit_code)

        # Only the derived result crosses the boundary.
        return json.loads(sbx.filesystem.read("/work/result.json"))
    finally:
        # Guest memory, tmpfs scratch, temp files the parsing library wrote,
        # and any half-written buffer stop existing as a single event. No
        # snapshot is taken of this machine, ever -- see the section above.
        sbx.kill()

The egress fence deserves emphasis because it is the control that holds when everything else has failed: default-deny at the guest's network namespace, enforced by the host, allowing only the internal endpoint the job legitimately needs. That removes exfiltration, the cloud metadata endpoint and the pivot into your VPC — in the world where the job already has a root shell, which is the world worth designing for.

Isolation between patients and tenants

Now the boundary between one execution and the next. Containers are not non-compliant; anyone telling you otherwise is selling something, and plenty of healthcare products run on them appropriately. But be honest about what the sentence looks like in a risk assessment. A container boundary is namespaces, cgroups, seccomp filters and a mandatory access control profile, all layered over one shared kernel — so the paragraph you write says that the job processing one health system's records and the job processing another's issue syscalls into the same kernel, and that separation is maintained by policy inside a shared runtime.

That is defensible. It is just a longer conversation, because the effective boundary is a composite: node pools and taints, network policy plus whether your CNI enforces it in the mode you actually run, admission control, token mounting, the log agent reading every pod's stdout, the kubelet. Each part is reasonable; together they are an argument a well-meaning platform engineer can silently invalidate on a Thursday by adding a host mount to debug something unrelated.

A guest kernel per execution under KVM is simply an easier boundary to describe. The interface is small enough to enumerate in a sentence — a handful of virtio devices, one control channel, one network device with a host-side allowlist — so a penetration tester can be pointed at it and an assessor can read it on one page. That legibility, more than any abstract security ranking, is what makes it cheaper to defend when the workload is PHI. Side channels remain a category the hypervisor reduces rather than eliminates; those mitigations are host-level and belong in your hardening standard.

The controls that have to exist around it

Architecture shrinks the surface. It does not produce the artifacts, and the artifacts are what somebody eventually asks for. The practical list, in roughly the order teams get it wrong:

1. Business associate agreements with every subprocessor that could touch PHI — and enumerate that list honestly. It includes your cloud provider, your log and APM vendor, your error tracker, your inference provider, your email and support tooling if a record can end up in a ticket, and any hosting platform your execution runs on. Confirm availability and scope directly with each vendor; do not infer it from a trust page.
2. Encryption in transit and at rest, everywhere the data lands — including snapshots, volumes, backups and the intermediate buckets nobody drew on the diagram. Addressable is not optional in practice.
3. A per-execution audit record: who initiated it, what code ran, which patient scope it was authorized for, when it started and ended, and the outcome. One job to one machine makes this exact instead of an exercise in correlating interleaved worker logs.
4. Structured logging with a field allowlist, plus repr guards on your PHI types so a stray f-string or traceback renders a placeholder rather than a chart. Assume every log line will eventually be read by someone with no clinical relationship to the patient.
5. Retention and deletion you can evidence — for the outputs, the audit trail, and anything the pipeline cached. A TTL enforced by the platform is evidence; a cron job someone wrote is a control you have to prove ran.
6. Access control and authentication on the control plane that creates these machines, because whoever can start an execution can choose its patient scope. That service is now in scope for everything the data is.
7. De-identification wherever the job does not genuinely need identified data — the strongest move on this list, and the subject of the next section.

The audit record is the artifact that turns all of this from a description into something checkable, and its most important property is that it can be complete without containing any PHI at all.

{
  "event": "phi.execution.completed",
  "execution_id": "5f3c1a2e-9d47-4b21-9a6c-0f8e2b7d1c34",
  "sandbox_id": "b71e0c94-2a35-4d88-bf10-6c9a3e5d7f21",
  "template": "code-interpreter",
  "template_generation": "2026-08-19T11:04:02Z",
  "initiated_by": "user:1841",
  "on_behalf_of_org": "org:northside-clinical",
  "purpose": "treatment_analytics",
  "patient_scope": { "kind": "single", "patient_ref": "pt_9f2c11" },
  "field_scope": ["birth_date", "encounter_date", "medication_code"],
  "record_count": 1,
  "egress_policy": "allow:fhir-internal-only",
  "persistent_disk": false,
  "snapshot_taken": false,
  "started_at": "2026-09-05T14:22:07.118Z",
  "destroyed_at": "2026-09-05T14:22:49.902Z",
  "outcome": "success"
}

Nothing in that record is PHI, and the cheapest way to keep it that way is to have no field that could hold any. When somebody proposes adding the patient's name "just for the support dashboard", that column puts the audit store, its replicas, its backups and every analytics copy inside the boundary. The two booleans near the bottom are the ones I would alert on: an execution that ran with a persistent disk, or that got snapshotted, is an exception you want to see the same day rather than during an assessment.

The strongest move is not having PHI in the job at all

If you can de-identify before the code runs, do that instead of everything above. De-identified data is not PHI, and the entire apparatus — safeguards, BAAs, breach analysis — falls away for that dataset. HIPAA gives two routes: Safe Harbor, the removal of a specified list of identifiers, and Expert Determination, where a qualified statistician documents that re-identification risk is very small. Read the current text of both with counsel rather than from memory.

Three practical caveats. Free-text clinical notes resist Safe Harbor badly — names, dates and facility names appear mid-sentence, and no tool is perfect on prose. Structured data can be re-identifiable in combination even when every listed identifier is gone, which is what Expert Determination exists to assess. And the de-identification pipeline itself processes identified data, so that step lives inside the boundary this post describes — you have moved the PHI surface to one narrow job rather than eliminating it. Still a very good trade.

What this architecture does not do

It does not encrypt anything, decide who may access what, or write your risk analysis. It does not cover the administrative safeguards, which is where the majority of enforcement activity actually concentrates. It leaves the host, the hypervisor and the control plane that schedules these machines fully in scope — a narrow, stable, well-described boundary instead of a wide shifting one is a good trade, not a vanishing act. It does not make your logging vendor a business associate, or your inference provider one. And it produces no certification, attestation or audit report, because those come from an audit, not from an architecture diagram.

The summary

PHI does not stay where your data flow diagram says it stays. Running code over it puts copies in process memory, in temp files a dependency created, in logs that leave the building by design, in prompts sent to a subprocessor, and — the one that should worry you most — in any memory snapshot taken while the process was holding a record, which is a full-fidelity copy of patient data sitting in a bucket under a lifecycle policy nobody chose deliberately. Give each execution its own microVM with no persistent disk, no egress beyond one allow-listed endpoint, an audit record that names machines and surrogate identifiers instead of patients, structured logs with a field allowlist, and a TTL the platform enforces. Snapshot clean templates, never loaded machines. De-identify first wherever the job allows it. The honest sentence in your risk analysis then gets short, specific and testable, and the paperwork around it gets tractable. That is what architecture buys. Compliance is still a program, and it is still yours.