Per-Tenant AI Voice Transcription in MicroVMs

A multi-tenant transcription service is one of the least glamorous and most legally exposed things you can run. The product is simple: a customer uploads audio, you return text. The audio is customer support calls with card numbers read aloud, clinicians dictating diagnoses, lawyers recording depositions, sales teams recording calls in states with two-party consent laws. Every one of those files is the kind of data that turns a bug into a notification obligation.

And the default architecture for it is a Python worker with a Whisper-class model loaded once at boot, pulling jobs off a queue. It's the obvious design — the model load is expensive, so you amortize it by keeping the process alive, and then you feed every tenant's audio through that same process forever. It works. It's fast. It is also a shared address space, a shared filesystem, a shared page cache, and a shared ffmpeg, holding data from customers who signed contracts saying their data is segregated from other customers' data.

I'm Ajay; I built PandaStack. This post is about why a shared transcription worker leaks in more ways than people count, why one Firecracker microVM per tenant is the boundary that actually holds, and — the part that usually kills the idea — how snapshot-restore of an already-warm model process means you don't pay a cold model load for the privilege.

Where the audio actually leaks

When people audit a shared transcription worker they check the obvious thing — does tenant A's job ever return tenant B's transcript — find that it doesn't, and declare it isolated. The leaks are not in the job routing. They're in the substrate underneath it, and there are more of them than the design accounts for.

• Scratch files in /tmp — decoded PCM, resampled WAVs, and chunked segments get written to disk because that's how audio tooling works. They're deleted on the happy path. On a crash, an OOM kill, a timeout, or a SIGKILL from your orchestrator, they persist — and the next tenant's job runs on a filesystem containing the previous tenant's audio.
• ffmpeg's own temporaries — ffmpeg writes its own intermediate artifacts, and a segmented or two-pass invocation leaves fragments behind. You didn't write that code and you don't control its cleanup path.
• Page cache — every file the worker reads stays resident in the host's page cache after deletion, indefinitely, until reclaim. Deleting the file removes the directory entry, not the bytes in RAM. Any process on that host that can read from memory can read what's cached.
• Model and framework state — decoder KV caches, the model's activation buffers, and framework-level scratch tensors hold fragments of the previous utterance until they're overwritten by the next one. Nothing clears them between jobs; they're just reused.
• The Python heap — audio arrays, decoded transcripts, and intermediate strings are freed but not zeroed. Freed memory in a long-lived process is a buffer holding the last tenant's words until an allocator hands it out again.
• Environment and credentials — a shared worker holds the storage credentials, the database connection, and the queue token for every tenant. Any code execution in that process reaches all of them at once.

None of those require an attacker. They're leaks by ordinary operation — a crashed job, a full disk, a debug dump, a core file written by a segfaulting native library with a few hundred megabytes of decoded audio in it. Multi-tenancy turns every one of those routine failures into a cross-tenant incident.

A malformed .wav is an unsolicited code-execution proposal

Then there's the part people genuinely underestimate. Before any model sees an utterance, something has to turn the uploaded file into 16 kHz mono PCM. In practice that something is ffmpeg, and ffmpeg is one of the largest, most battle-scarred parser surfaces in open source: hundreds of demuxers and decoders, most of them written in C, most of them parsing container and codec formats designed decades ago by people who assumed the input was well-formed.

Your users are uploading arbitrary bytes with a filename that ends in .mp3. You are handing those bytes to a C program and asking it to interpret embedded length fields, chunk offsets, and codec parameters. That is not "reading a file." That is running an untrusted format's instructions through a parser, and the entire history of media-processing CVEs says this class of bug does not go away. ffmpeg has a long, well-documented record of memory-safety issues found in demuxers and decoders — the project fixes them steadily, which is exactly the point: the surface is large enough that there is always another one.

• Container parsing is attacker-controlled — a crafted MP4/Matroska/WAV header can drive allocation sizes, seek offsets, and loop bounds inside the demuxer before a single audio sample is decoded.
• Codec decoders are worse — a malformed bitstream exercises arithmetic decoders and reference-frame handling written for speed, not for hostile input.
• Auto-detection widens it — if you let ffmpeg probe the format, an upload named `call.wav` can route into an entirely different demuxer than the one you thought you were exercising.
• Resource exhaustion needs no bug at all — a file declaring an enormous duration or absurd channel count can make decode allocate or spin until something dies. In a shared worker, that something is everyone's job.
• Feature reach — unless you restrict protocols and formats, media tools can be talked into touching things other than the file you handed them. Decode should not be a component with network reach.

The model: one microVM per tenant, per job

The structural fix is to stop running decode and transcription inside a process that holds anything belonging to anyone else. Your control plane — the queue consumer, the database credentials, the storage keys, the billing meter — stays on a trusted host. It creates a Firecracker microVM, writes exactly one tenant's audio into it, runs decode and transcription inside the guest, pulls the transcript back over the API, and destroys the VM.

Now walk the leak list again. /tmp is inside a guest that ceases to exist. ffmpeg's temporaries are in that same guest. The page cache holding decoded audio is the guest's page cache, in guest RAM, freed with the VM. The model's KV state and the Python heap go with it. A memory-safety bug in a demuxer gets the attacker arbitrary code execution inside a hardware-virtualized VM with its own kernel, one tenant's audio, and no credentials — which is a very different afternoon from the same bug in a worker holding forty customers' recordings and a storage key.

The reason this is practical rather than merely correct is create latency. A cold Firecracker boot is roughly 3 seconds, which would be fatal for a per-job model. But PandaStack doesn't cold-boot on create — every create restores a baked snapshot on demand. The snapshot-load step is around 49ms and end-to-end create lands at 179ms p50, roughly 203ms p99. A per-job hardware-isolated machine costs about a fifth of a second. Each agent also carries 16,384 pre-allocated /30 network subnets, so fanning out across many tenants concurrently is bounded by memory and CPU, not by network slot capacity.

from pandastack import Sandbox

def transcribe(tenant_id: str, job_id: str, audio: bytes) -> dict:
    """One disposable guest per job. Decode + transcribe never touch the host."""
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=900,                       # hard cap: a decode bomb can't run forever
        metadata={"tenant": tenant_id, "job": job_id},
    )
    try:
        # The ONLY tenant data in this VM is this one tenant's file.
        sbx.filesystem.write("/work/in.bin", audio)

        # Decode inside the guest. If a crafted container corrupts ffmpeg's
        # heap, it corrupts a heap in a VM that holds nothing else and dies
        # in fifteen minutes regardless.
        dec = sbx.exec(
            "ffmpeg -hide_banner -nostdin -y "
            "-protocol_whitelist file "          # no network reach from decode
            "-t 7200 "                           # cap declared duration
            "-i /work/in.bin -vn -ac 1 -ar 16000 -f wav /work/audio.wav",
            timeout_seconds=300,
        )
        if dec.exit_code != 0:
            return {"ok": False, "stage": "decode", "error": dec.stderr[-2000:]}

        out = sbx.exec(
            "python3 /opt/transcribe.py /work/audio.wav /work/out.json",
            timeout_seconds=600,
        )
        if out.exit_code != 0:
            return {"ok": False, "stage": "asr", "error": out.stderr[-2000:]}

        # Pull the transcript out; leave every byte of audio behind.
        return {"ok": True, "transcript": sbx.filesystem.read("/work/out.json")}
    finally:
        # /tmp, ffmpeg scratch, guest page cache, decoder KV state, the Python
        # heap -- all of it is inside this VM, and all of it goes now.
        sbx.kill()

The cold model load problem, and how snapshots kill it

Here's the objection that normally ends this conversation: a speech model is not free to load. Reading weights off disk, allocating tensors, initializing the runtime, and warming the first inference path can take tens of seconds. Doing that per job is absurd, and that cost is precisely why everyone builds the long-lived shared worker in the first place. The isolation argument is easy; the economics argument is what actually decides the architecture.

Snapshots dissolve the objection, because the expensive thing here isn't computation you need to repeat — it's a memory state you can freeze. Load the model once, into a guest, let it fully initialize and run one warm-up utterance so the lazy allocations and first-inference paths are done. Then snapshot the machine. That snapshot is guest RAM plus device and CPU state: an interpreter with the framework imported, weights resident, buffers allocated, and the first inference already paid for. Restoring it doesn't start a process — it resumes one that already finished starting.

The result is that per-job isolation stops competing with the shared worker on warm-start cost. You get the model-already-loaded property that made the shared worker attractive, without the shared part. For burst, fork the warm snapshot: a same-host fork lands in roughly 400–750ms and the children share the weights' memory pages copy-on-write, so ten concurrent transcriptions don't cost ten copies of the model until each one starts writing. A cross-host fork is 1.2–3.5s when the warm image lives on another agent.

from pandastack import Sandbox

# ---- Bake ONCE: load the model and warm it, then freeze that machine. ----
warm = Sandbox.create(template="code-interpreter", ttl_seconds=3600)
warm.exec("python3 /opt/load_model.py --model small.en", timeout_seconds=900)
# Run one throwaway utterance so lazy allocs and the first inference path
# are already paid for in the frozen image.
warm.exec("python3 /opt/warmup.py /opt/silence_2s.wav", timeout_seconds=300)
snap = warm.snapshot()   # guest RAM: framework imported, weights resident
warm.kill()

# ---- Serve: fork the warm image per job. Model is ALREADY loaded. ----
# Same-host fork ~400-750ms (cross-host 1.2-3.5s). The forks share the
# weights' pages copy-on-write until they diverge, so N concurrent jobs
# cost far less RAM than N cold model loads.
def transcribe_warm(tenant_id: str, audio: bytes) -> str:
    sbx = snap.fork(ttl_seconds=900, metadata={"tenant": tenant_id})
    try:
        sbx.filesystem.write("/work/in.bin", audio)
        sbx.exec("ffmpeg -nostdin -y -protocol_whitelist file -i /work/in.bin "
                 "-vn -ac 1 -ar 16000 -f wav /work/audio.wav",
                 timeout_seconds=300)
        r = sbx.exec("python3 /opt/transcribe.py /work/audio.wav",
                     timeout_seconds=600)
        return r.stdout
    finally:
        sbx.kill()   # this tenant's audio never outlives this fork

Long audio: chunking, long-running guests, and TTL

Voice workloads have an awkward duration distribution. A voicemail is thirty seconds. A support call is eight minutes. A deposition is six hours, and a day of clinical dictation is a stack of files that arrives all at once at 6pm. A per-job VM model has to handle both ends without either burning a VM per two seconds of audio or holding one open for half a day with no supervision.

The shape that works is: chunk inside the guest, not outside it. Segment the decoded audio into overlapping windows within the same VM that decoded it, transcribe them sequentially or in-process, and stitch the result there. Chunking outside means shipping decoded PCM back to your control plane, which puts raw audio on your trusted host — the exact thing you built the guest to avoid. Keep the audio in one place: the guest that will be destroyed.

• Set ttl_seconds from the declared duration, generously. Probe the file's duration before decode, multiply by a realistic real-time factor, add slack, and cap. The TTL is a dead-man's switch, not a scheduling estimate — it exists so a wedged job becomes a reclaimed guest instead of a pager.
• Chunk with overlap inside the guest. Fixed windows with a few seconds of overlap, stitched on the boundary, keeps memory bounded on multi-hour files without ever moving PCM off the guest.
• Stream progress out, not audio. Emit partial transcripts and a percentage as they're produced. The control plane learns the job is alive without receiving anything it doesn't need to hold.
• One guest per file, not per chunk. A six-hour deposition is one tenant's data start to finish; splitting it across VMs multiplies create cost and buys no additional isolation.
• Batch the 6pm dictation stack with forks. Many small files from one tenant fan out cleanly as concurrent forks off the warm snapshot, each destroyed on completion.
• Treat TTL expiry as a real outcome. A guest reclaimed mid-transcription must surface as a failed, retryable job with a reason — not as a job that silently never finishes.

Egress: a transcription VM has no business phoning home

Isolation that only runs inward is half a boundary. A transcription guest that can open arbitrary outbound connections can, if something inside it is compromised, simply POST the audio somewhere. Ask what this workload legitimately needs to reach and the answer is close to nothing: it received its input through the sandbox API and it returns its output the same way.

So default-deny egress and open only what the job genuinely requires. On PandaStack each sandbox lives in its own network namespace with a dedicated veth pair and TAP device out of that pre-allocated /30 pool, which means egress policy is per-sandbox filtering on a real interface rather than a shared bridge you hope is partitioned correctly. If the model weights are baked into the template — and for this design they should be — the guest needs no network at all for the transcription itself.

• Bake weights into the template. A guest that downloads a model at job time needs network access at job time, and "allowed to fetch from the model host" is a hole shaped exactly like exfiltration. Bake once, restore forever.
• Default-deny outbound. Start from zero and add specific destinations, rather than blocking known-bad and hoping the list is complete.
• Move data through the sandbox API, not the guest's network. Writes in, transcript out, over the control plane's authenticated channel — so the guest needs no credentials and no route.
• Deny decode any protocol but file. Media tools that can be pointed at URLs are a decode-time SSRF primitive; restrict the protocol whitelist explicitly.
• Log what the guest attempted. A transcription VM trying to open an outbound connection is, by construction, an incident signal — there is no legitimate reason for it.

The head-to-head

• Isolation boundary — Shared Python worker: none; every tenant's audio, decoder state, and freed heap share one address space with every tenant's credentials. Container per job: shares the host kernel, so an ffmpeg memory-safety bug plus a kernel bug reaches the host and neighbors. MicroVM per job: hardware-virtualized guest with its own kernel; a compromised demuxer is confined to a VM holding exactly one tenant's file.
• Residue after the job — Shared worker: /tmp scratch survives crashes, page cache holds decoded audio, KV state and freed heap carry fragments into the next tenant's job. Container: filesystem goes with the container if it exits cleanly, but the host page cache and any bind-mounted scratch do not. MicroVM: guest RAM, guest page cache, and guest disk are all destroyed with the VM — there is no next job for residue to reach.
• ffmpeg parser risk — Shared worker: a demuxer bug is arbitrary code execution next to every tenant's recordings and your storage keys. Container: same code execution, one namespace hop from the host kernel. MicroVM: same bug, contained by hardware virtualization, in a guest with no credentials and a TTL.
• Cold model load — Shared worker: paid once, which is exactly why the process is permanent and permanently exposed. Container per job: paid per container, which is why nobody does it. MicroVM: warm the model once, snapshot it, fork per job (~400-750ms same-host) — amortized without a shared process.
• Noisy neighbor — Shared worker: one pathological file's decode loop starves every other tenant's queue. Container: cgroup limits help, kernel is still shared. MicroVM: fixed vCPU and RAM baked into the guest, plus ttl_seconds; a decode bomb saturates its own VM and gets reclaimed.
• Audit story — Shared worker: you can prove your router segregated jobs, and nothing about the substrate underneath. Container: process-level boundary that a reviewer will correctly push back on for regulated audio. MicroVM: per-tenant VM lifecycle with create and destroy events, which is a claim about the substrate rather than the application.

The audit story for HIPAA/GDPR-shaped requirements

I'm not a lawyer and this isn't compliance advice — get your framework interpreted by someone qualified. But architecture determines which questions you can answer with evidence rather than assertion, and that distinction is most of what a review actually turns on.

With a shared worker, the honest answers get uncomfortable fast. "Where was the PHI processed?" In a process that also held eleven other customers' data. "What else had access to it?" Everything in that address space. "How do you know it's gone?" We deleted the temp file. "Was it in memory afterward?" Yes, until something else overwrote it. Every answer is a statement about your code being careful, and careful code is not a control — it's an intention.

With a VM per job, the answers become facts about infrastructure. The audio was processed in VM `abc-123`, created at a timestamp, holding one tenant's data, destroyed at a timestamp. Nothing else ran in it. Its memory and disk were reclaimed with it. It had no outbound network. Its lifecycle is in your event log. That's a durable record of a boundary, and it's the difference between "we're careful" and "here is the segregation, per record."

• Emit a lifecycle record per job — sandbox ID, tenant, create and destroy timestamps, template and its bake generation. That tuple answers most of the segregation questions directly.
• Never log the payload. Log job IDs, durations, and outcomes; never transcript text or audio bytes. Logs outlive VMs and aggregate across tenants — they are the one place your per-job boundary can quietly leak.
• Record the egress policy that was in force, not just that one existed. "Default-deny, zero destinations allowed" is a reviewable statement.
• Tie the template bake to your build pipeline. A snapshot is a frozen machine, so a vulnerable ffmpeg in a baked image keeps being restored until someone re-bakes. Patch cadence for templates needs the same discipline as base images — this is a real cost of the snapshot model, not a footnote.
• Handle the restore-side clock. A guest restored from a snapshot baked weeks ago wakes believing it's still bake day, which breaks TLS validation and corrupts your timestamps. Sync the clock on every restore and resume, deliberately.

The summary

A shared transcription worker leaks through more channels than its design accounts for — /tmp scratch that survives crashes, ffmpeg's own temporaries, host page cache holding decoded audio long after deletion, decoder KV state, a freed-but-not-zeroed Python heap, and one set of credentials reachable from every job. Layer on ffmpeg's parser surface, where the input is an untrusted binary format handed to C code, and the leak channels become an exploitation path. For customer support calls, clinical dictation, and depositions, that's the wrong substrate regardless of how careful the job router is.

One microVM per tenant per job moves every one of those channels inside a boundary that gets destroyed on completion. The cost that normally makes this impractical — loading a speech model per job — goes away when you bake the warm model into a snapshot and fork it: ~179ms p50 to create, ~400-750ms for a same-host fork off an image where the weights are already resident. Chunk long audio inside the guest, set a TTL as a dead-man's switch, default-deny egress so decode has nowhere to phone home, and log the VM lifecycle instead of the payload. Then the answer to "where was this recording processed and what else could reach it" is a row in a table, not a promise.

For the neighboring problems: the general per-tenant SaaS argument is in /blog/microvm-saas-multi-tenant-isolation, per-tenant model serving and warm-snapshot mechanics are in /blog/microvm-per-tenant-ml-inference-isolation, and the egress side gets its own treatment in /blog/controlling-network-egress-untrusted-code.