Hostile and Precious: Per-Case microVMs for Forensics and E-Discovery

The evidence arrives on an encrypted drive with a chain-of-custody form taped to the outside: mail archives, a couple of disk images, a folder of mobile extractions, four hundred gigabytes of a custodian's documents. Your job is to open all of it and produce findings that survive somebody else's expert picking at them.

The material you are legally obligated to preserve byte-for-byte is also the material most likely to be actively hostile to whatever machine you open it on. Forensics is the only discipline where "this file is trying to attack me" and "this file is an exhibit I must not alter" are routinely true about the same file at the same time.

Most shops solve this by hoping — open the PST with the network unplugged, or remote into the shared forensics box and agree to be careful. Both are load-bearing conventions with no enforcement behind them. This post is about the shape that enforces it: one microVM per matter, evidence attached read-only, tools inside, results out through a narrow channel, and a teardown that destroys a whole machine rather than a folder.

The double bind: evidence is hostile and precious

Start with hostile. A ransomware-incident mailbox contains the phishing attachment that started it, still armed, in Deleted Items. A litigation collection includes whatever the custodian downloaded, and custodians download anything. Disk images from a compromised endpoint contain the implant by definition — that is why you imaged it. Running a parser across that material feeds attacker-authored bytes into libraries with long CVE histories: PST and OST readers, PDF renderers, archive extractors, thumbnailers, Office filters. Half of e-discovery processing is calling a decades-old C library on a file written to break it.

The exotic case is a memory-corruption exploit in the PDF viewer. The boring case, the one that happens on ordinary Tuesdays, is the tracking pixel: a remote image reference in an HTML email, a DOCX with an external template, a PDF that fetches a resource on open. None of these are exploits. They are documented features, and each is a callback saying "this document was opened, at this time, from this IP address" to whoever authored it. In an internal investigation that recipient is the subject of the investigation, and you have just told them you are looking. In a hostile-actor case you have told an adversary their implant reached an analyst environment, which is the signal that triggers cleanup. The beacon does not need to be clever. It needs to load once.

Now the other half: this same file is an exhibit. Its hash was recorded at acquisition and must reproduce later, in front of people whose job is to argue that it doesn't. "We processed the Reyes mailbox on the shared box sometime in March" is not a description of an environment. The controls that make evidence safe to open and the controls that make it defensible get treated as separate programs. They are the same program.

Why the laptop and the shared forensics VM are both wrong

The analyst workstation

The examiner's laptop is the worst possible place to open hostile evidence, and it is where most evidence gets opened. It holds credentials to the case management system, the collection tooling, the client's environment, and the VPN. It is domain-joined. It has every other matter the examiner touched this quarter sitting in a working directory, because nobody deletes working directories.

So a bad parse does not cost you one matter. It costs the examiner's entire caseload, plus a disclosure conversation with every client in it. The mitigation everyone reaches for — unplug the network — lasts exactly until someone needs to look something up, and it does nothing about cross-case contamination, which is happening quietly the whole time.

The shared forensics VM

The upgrade is a beefy dedicated box with the processing tools installed that everyone remotes into. It genuinely beats the laptop — evidence stops living on analyst endpoints — which is why it's the industry default. But it converts a per-analyst problem into a per-firm one. Every matter shares a filesystem, a network identity, and a temp directory. Processing tools cache aggressively (thumbnail databases, extracted attachment folders, index shards), and those caches are keyed by content, not by matter. When opposing counsel asks whether material from an unrelated client could have influenced your search results, the honest answer involves a lot of hedging about scratch space.

And when something goes wrong on the shared box, everything on it is in scope. One hostile disk image compromises a host holding evidence for a dozen clients, so your incident is now twelve incidents with twelve notification obligations. The shared forensics VM is a single point of failure carefully sized for maximum blast radius.

Containers, briefly

Containers are the obvious modernization and they solve the wrong half. Per-matter processing containers give you clean, reproducible tool environments, which is real value — but a container is a polite suggestion to the shared host kernel, and the thing you are feeding it is a file engineered to make a parser misbehave. Hardened runtimes like gVisor and Kata exist precisely because people noticed; evaluate those against their current documentation if you go that route. For evidence, the boundary you want is the one plain containers do not offer: a separate kernel.

Four options, five dimensions

• Cross-case contamination — Analyst workstation: every matter the examiner ever touched shares one filesystem. Shared forensics VM: all matters share temp dirs, caches, tool state. Container: fresh filesystem per run, shared host caches and volumes in practice. Per-case microVM: separate guest disk per matter, no path between them.
• Hostile-file blast radius — Analyst workstation: examiner credentials, VPN, and every open matter. Shared forensics VM: every client's evidence on the host, so one incident becomes N notifications. Container: the host kernel and every co-tenant on it. Per-case microVM: one guest kernel serving one matter, behind hardware virtualization.
• Egress control — Analyst workstation: corporate network by default; "unplug it" is a habit, not a control. Shared forensics VM: one network identity for all matters, usually with broad outbound access. Container: host networking policy, easy to widen, hard to prove per-run. Per-case microVM: dedicated netns and tap, default-deny enforced outside the guest, denials logged per matter.
• Reproducibility — Analyst workstation: whatever was installed that week. Shared forensics VM: shared tool state drifting under everyone. Container: the image digest pins tools, not examination state. Per-case microVM: a snapshot captures tools and examination state together.
• Provable teardown — Analyst workstation: a folder delete and a promise. Shared forensics VM: selective deletion from a live multi-tenant filesystem. Container: image and volume lifecycle reasoned about separately. Per-case microVM: destroy the VM and its disk — what you deleted is the whole environment, logged with the matter id.

The per-case microVM shape

One VM per matter. Not per firm, not per examiner, not per tool. The matter is the unit of legal obligation, so make it the unit of isolation and the two stop drifting apart.

The VM boots its own guest kernel under hardware virtualization, so a parser exploit that gets code execution gets it on a kernel serving exactly one matter, with no other client's data reachable. Evidence is attached read-only, so the guest is structurally incapable of altering the exhibit — not "the tools are configured not to write," but no writable handle exists. Processing tools live inside the image, and results leave through one narrow channel: you read structured output back out, and nothing else crosses.

On PandaStack each sandbox is a Firecracker microVM created by restoring a baked snapshot, roughly 179ms at p50, which matters more than it sounds like it should. When spinning up an isolated environment is a coffee break, examiners reuse the one that's already open. When it's a fifth of a second, per-matter isolation stops being a policy people comply with and becomes the path of least resistance. In practice: create the VM carrying the case metadata, push an extraction script in, run the parser under a hard timeout, pull structured results back, destroy the VM.

import json
from pandastack import Sandbox

MATTER = "2026-CV-00417"
CUSTODIAN = "reyes-j"
EXHIBIT = "reyes-mailbox.pst"
ACQUIRED_SHA256 = "9f2c1e..."  # recorded at acquisition, from the custody record

# The parser runs entirely inside the guest. It reads the exhibit read-only and
# writes one JSON file; that file is the only thing that ever leaves.
PARSER = """
import hashlib, json, sys
from pathlib import Path
import pypff  # libpff bindings, baked into the examination image

src = Path("/evidence/reyes-mailbox.pst")
pff = pypff.file()
pff.open(str(src))

rows = []
def walk(folder, path=""):
    for i in range(folder.get_number_of_sub_messages()):
        m = folder.get_sub_message(i)
        rows.append({
            "folder": path,
            "subject": m.get_subject(),
            "sender": m.get_sender_name(),
            "delivered": str(m.get_delivery_time()),
            "attachments": m.get_number_of_attachments(),
        })
    for i in range(folder.get_number_of_sub_folders()):
        sub = folder.get_sub_folder(i)
        walk(sub, path + "/" + sub.get_name())

walk(pff.get_root_folder())

h = hashlib.sha256()
with src.open("rb") as f:
    for chunk in iter(lambda: f.read(1 << 20), b""):
        h.update(chunk)

Path("/workspace/index.json").write_text(json.dumps({
    "source_sha256": h.hexdigest(),
    "message_count": len(rows),
    "messages": rows,
}))
print("indexed", len(rows), "messages", file=sys.stderr)
"""

sbx = Sandbox.create(
    template="base",
    ttl_seconds=8 * 3600,          # backstop: the VM cannot outlive the shift
    metadata={
        "matter_id": MATTER,
        "custodian": CUSTODIAN,
        "exhibit": EXHIBIT,
        "exhibit_sha256": ACQUIRED_SHA256,
        "examiner": "a.kumar",
        "egress": "deny",
    },
)
try:
    sbx.filesystem.write("/workspace/index_pst.py", PARSER)

    r = sbx.exec("python3 /workspace/index_pst.py", timeout_seconds=900)
    if r.exit_code != 0:
        # A crashed parser on hostile input is a finding, not just a failure.
        raise RuntimeError(f"parser exited {r.exit_code}: {r.stderr[-2000:]}")

    index = json.loads(sbx.filesystem.read("/workspace/index.json"))
    assert index["source_sha256"] == ACQUIRED_SHA256, "exhibit hash changed in-guest"

    print(f"{MATTER}: {index['message_count']} messages in {r.duration_ms}ms")
finally:
    sbx.kill()   # the environment, its disk, and every temp file go with it

The metadata is not decoration. It is the join key between a running VM and a matter, which is what lets you answer "what is currently processing client X's data" and "prove this environment was destroyed" from the same query. Retention questions are unanswerable when the answer depends on an examiner remembering something.

Egress: the difference between a quiet investigation and a tipped-off subject

In most sandboxing contexts, default-deny egress is hygiene. Here it is the primary control, because the loudest failure mode of evidence review is not compromise. It is notification.

It has to be enforced outside the guest, and this is the part teams get wrong. Guest-side firewall rules are configuration inside the thing you assume is compromised: if a parser exploit gets root in the guest, it edits your rules and then makes the connection. The rule must live on the far side of the isolation boundary. On PandaStack each sandbox gets its own network namespace and tap device — 16,384 pre-allocated /30 subnets per agent — so the policy is applied to the VM's network path by the host, not requested politely from the guest.

Default-deny for a forensics VM means the allowlist is usually empty. Not "small." Empty. Tools are baked into the image, exhibits are attached locally, results come out through the control channel. If you genuinely need something — an internal evidence store, a hash-lookup service inside your perimeter — it is one destination, named, logged, and reviewed.

Then treat denials as evidence. A blocked outbound connection during a PST index is a finding: something in that mailbox tried to phone home, and you have the destination, the timestamp, and the exhibit that triggered it. On a workstation with the cable unplugged, that same event is silence. Default-deny doesn't just prevent the beacon; it converts the beacon into a line in the report.

Reproducibility: snapshot the examination, fork the destructive parse

Reproducibility here is not a nice engineering property. It is what gets your findings taken seriously when someone else's expert disagrees with them. "Rebuild the environment from these install notes and hope the tool versions still resolve" is not reproduction, as anyone who has rebuilt a three-year-old processing stack for a matter that finally went to trial can confirm.

A snapshot captures the running machine — tools, versions, loaded state, intermediate artifacts — as a restorable object. Take one at each defensible checkpoint: after provisioning and before evidence is attached, after ingest and hashing, after the first processing pass. A second examiner restores the exact machine you worked in rather than a description of it.

Forks are the other half, and they make aggressive analysis safe. Carving a damaged image, decompiling an implant, letting a repair tool loose on a corrupt archive — all can wreck working state, and on a shared box the recovery is starting the matter over. A fork gives the destructive attempt its own copy-on-write machine; same-host forks land in the 400-750ms range, so branching is cheaper than being careful. If the parse eats itself, kill the fork. The baseline never moved.

# Checkpoint the examination, then branch for anything risky.
baseline = sbx.snapshot()          # tools + ingested evidence + hashes, frozen

# A repair-then-parse pass that has a real chance of destroying working state.
branch = sbx.fork()
try:
    branch.filesystem.write("/workspace/carve.py", CARVE_SCRIPT)
    r = branch.exec(
        "python3 /workspace/carve.py --repair --aggressive",
        timeout_seconds=1800,
    )
    if r.exit_code == 0:
        carved = branch.filesystem.read("/workspace/carved.json")
        open(f"out/{MATTER}-carved.json", "wb").write(carved)
    else:
        # Failed carve is a data point about the exhibit; keep stderr, drop the VM.
        open(f"out/{MATTER}-carve.err", "w").write(r.stderr)
finally:
    branch.kill()

# The pristine examination environment is untouched, and `baseline` can be
# restored months later by a second examiner or an opposing expert.
print("baseline snapshot:", baseline.id)

Pair the snapshot with an integrity record, because a snapshot proves what the environment was and a hash chain proves what the evidence was. Hash the exhibit before the VM ever sees it, hash it as the guest sees it, and hash it again after teardown.

import hashlib, json, datetime

def sha256_file(path: str, chunk: int = 1 << 20) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for block in iter(lambda: f.read(chunk), b""):
            h.update(block)
    return h.hexdigest()

src = f"/evidence/{MATTER}/{EXHIBIT}"

before = sha256_file(src)                       # host side, pre-attachment
inside = sbx.exec(f"sha256sum /evidence/{EXHIBIT}",
                  timeout_seconds=600).stdout.split()[0]   # as the guest saw it
sbx.kill()
after = sha256_file(src)                        # host side, post-teardown

record = {
    "matter_id": MATTER,
    "exhibit": EXHIBIT,
    "acquired_sha256": ACQUIRED_SHA256,
    "pre_examination": before,
    "as_examined": inside,
    "post_examination": after,
    "unchanged": len({ACQUIRED_SHA256, before, inside, after}) == 1,
    "examined_at": datetime.datetime.now(datetime.UTC).isoformat(),
    "environment_snapshot": baseline.id,
}
with open(f"custody/{MATTER}-{EXHIBIT}.json", "w") as f:
    json.dump(record, f, indent=2)

assert record["unchanged"], "HASH MISMATCH — stop and escalate before proceeding"

Three hashes and a snapshot id is a boring artifact, and boring is the goal. It says the exhibit that went in is the exhibit that came out, the guest saw the same bytes, and the machine that produced the findings can be stood back up. That is most of a defensibility argument, generated as a side effect of the pipeline rather than reconstructed from memory eighteen months later.

Contamination and retention: deletion you can actually substantiate

Cross-case contamination is the risk nobody markets against, because it is nobody's dramatic incident. It is a slow leak: a processing tool writes an extracted-attachment cache to shared scratch, an index shard from last quarter is still on disk, somebody copies a working file to /tmp and never comes back. None of these are breaches. All of them are things you would rather not describe under oath.

A per-matter VM removes the mechanism rather than the temptation. There is no shared scratch space because there is no sharing; the tool cache lives inside a guest that exists for one matter and dies with it. You don't have to trust that everyone cleaned up, because there is no shared place to fail to clean.

Retention gets the same upgrade. Engagements end with an obligation to return or destroy client material, and on a shared system that promise is a folder delete plus a hopeful email. Teardown of a per-matter VM destroys the guest disk and everything derived from the evidence that ever lived on it — caches, temp files, indexes, extracted attachments — as one logged event carrying the matter id, exhibit hashes, examiner, and timestamp. That is the difference between asserting deletion and substantiating it. Custody records and snapshots you deliberately keep stay under your retention policy; the point is that they become an enumerable list rather than whatever happened to survive on a shared filesystem.

Bursty matters: hundreds of VMs for a week, then zero

Forensics and e-discovery workloads are among the spikiest in existence. A matter opens and for ten days you need every core you can find to hit a production deadline. Then it goes quiet for two months while lawyers argue. Then a second production request arrives and you need the same capacity again on 48 hours' notice. Capacity provisioned for the peak sits idle most of the year and is still somehow too small during the peak — the on-premises forensics lab in one sentence.

Per-case VMs invert it: fan out one VM per custodian or evidence container, run them in parallel, collapse to nothing when the burst ends. Two mechanics make that safe rather than merely fast. TTLs mean a processing VM cannot outlive its job — if a parser wedges on a malformed archive at 2 a.m., the VM reaps itself instead of sitting there for a week holding client evidence. Scale-to-zero means a quiet matter costs nothing while it waits, so nobody keeps a shared box warm "because spinning up is a hassle."

Set the TTL to the job, not the matter. Bulk processing runs get hours; an interactive review session gets a shift with an explicit extend. The examiner who needs longer asks for longer, and that request is logged, which is a feature.

What this does not do

An honest scope note, because this is a domain where overclaiming causes real harm: per-case microVM isolation is an operational control. It is not legal advice and it is not an admissibility argument. Admissibility, preservation obligations, spoliation exposure, litigation-hold scope, cross-border restrictions on custodian data, and what your jurisdiction expects from an examination protocol are questions for counsel — before you change your process, not after.

What the architecture gives you is narrower and still worth having: it makes the technical claims you want to make actually true. The exhibit was read-only. The environment was isolated to this matter. Egress was denied by default and denials were logged. The environment is restorable. Destruction was one logged event. Those are facts about your infrastructure a technical witness can testify to, and arriving at the legal conversation with enforced controls beats arriving with a description of your team's habits.

And this does not make hostile files safe to open. It makes them safe to open in a specific place. If examiners can still double-click a PST on a domain-joined laptop, you have built an excellent isolated environment and left the front door open. The control that matters is the one where evidence physically cannot land anywhere else: collection writes to evidence storage, evidence storage attaches only to matter VMs, and analyst endpoints never get a readable path to raw exhibits at all.