Per-Tenant LLM Fine-Tuning Jobs in Isolated microVMs

You built a feature customers love: they upload their own data, pick a base model, and get back a fine-tune. A LoRA adapter on a small language model, a fine-tuned embedding model for their private corpus, a reranker trained on their labeled pairs. It's the kind of thing that turns a generic AI product into one that feels like it was built for each customer. And it's the kind of thing that, running the way most teams first build it, quietly puts every customer's private dataset one bug away from every other customer.

Here's the shape of the problem. A fine-tuning job holds three dangerous things at once: a private training dataset (often the customer's most sensitive data — support transcripts, medical notes, proprietary documents), a credential to pull that dataset and push the resulting weights somewhere, and frequently a customer-supplied training script or config that you did not write. A training script is just untrusted code with a compute bill attached. The moment you run several tenants' jobs on a shared pool of workers, you've built a machine whose entire job is to hold one customer's data and one customer's key in a process that will, next hour, hold a different customer's data and key — and sometimes both at once.

I'm Ajay, and I build PandaStack, a Firecracker microVM platform, so I spend my time thinking about exactly where the boundary sits when you run other people's code next to other people's secrets. This post is about confining that blast radius: one microVM per fine-tuning job, dataset credentials injected per run and destroyed with the VM, egress locked to the dataset store and model registry, so a poisoned training script or a compromised dependency can compute a wrong adapter for its own tenant — and nothing else.

Why a shared training worker leaks

The reflexive architecture is a pool of beefy workers that pull jobs off a queue. Job arrives, worker downloads the dataset, runs the training loop, uploads the adapter, picks up the next job. It's efficient and it's a data-exfiltration surface with a schedule. The trouble isn't that your engineers are careless; it's that a shared, long-lived process running arbitrary training code has too many ways to leak.

• The dataset is on disk while untrusted code runs. Training needs the data local, so the customer's private corpus sits in the worker's filesystem exactly while a customer-supplied script runs in the same process. A hostile or buggy script reads /data as easily as it reads its own config. If two tenants' jobs ever overlap, or one job's temp files aren't scrubbed before the next lands, tenant A's transcripts are readable by tenant B's script.
• The credential is right there. The worker holds a key to pull datasets and push weights. Untrusted training code in that process reads it out of the environment or a mounted secret and now owns your raw credential — not just its effects — to use off your infrastructure, on its own time.
• The metadata endpoint is one SSRF away. On a cloud host, 169.254.169.254 hands the node's role to anything that can make an HTTP request. A training script that fetches a URL — or that you tricked into fetching one — pulls the host's identity, which is almost always broader than the tenant's scoped credential.
• A poisoned script is code execution, not just bad training. 'Customer-supplied training config' shades into 'customer-supplied code' the instant you support custom loss functions, data-loading hooks, or a requirements file. pip-installing a training dep is a supply-chain event; a malicious build step runs before a single gradient does.
• Noisy neighbors and runaway jobs. A training loop that spins forever, forks a mining process, or OOMs the box takes the shared worker — and every other tenant's queued job — down with it.

One microVM per fine-tuning job

Stop sharing the machine. Run each tenant's fine-tuning job in its own Firecracker microVM. Each VM boots its own guest kernel and is confined by KVM hardware virtualization — the only way out is a tiny set of emulated virtio devices, not the full Linux syscall surface a container shares with its host. This is the same isolation model the large clouds use to run untrusted customer code on shared fleets. The dataset you download and the credential you inject live inside one VM, are used by one tenant's job, and are gone the instant that VM is destroyed. There is no next job on the same machine to inherit them.

Two properties make this a real wall rather than a nicer-looking leak. First, the dataset and credential are per-run: they enter one guest for one job and die with it, so there's no residue for the next tenant's job to read. Second, the guest has its own network namespace, so you decide exactly what it can reach: the dataset store's endpoint and the model registry, and nothing else. No route to the metadata service, no route to your internal APIs, no route to another tenant's storage. A poisoned training script inside that VM can waste its own tenant's compute budget, and that's the ceiling on the damage.

The usual objection is startup cost, and for training it barely applies — a fine-tune runs for minutes to hours, so a few hundred milliseconds of VM create is rounding error. But it's worth knowing it's cheap anyway: PandaStack creates by restoring a baked snapshot of an already-booted machine rather than cold-booting. The restore step is around 49ms; an end-to-end create is p50 179ms and p99 about 203ms. A true cold boot happens only on the first spawn of a template (around 3s). So even short embedding-training jobs get a fresh, disposable machine without a boot tax.

A worked example: a per-tenant LoRA job

Here's the concrete shape. A tenant kicked off a fine-tune. We create a fresh microVM, inject a short-lived credential scoped to exactly this tenant's dataset and this tenant's model-registry path, write in their (untrusted) training script, run it, ship the adapter, and destroy everything. The dataset never touches a shared worker; the VM never had a route to anything but the two endpoints it needs.

from pandastack import Sandbox

def run_finetune_job(tenant_id: str, job_id: str, train_script: str,
                     creds: dict) -> dict:
    """Run ONE tenant's fine-tuning job in its own microVM.
    `creds` are SHORT-LIVED, minted for THIS job: read-only on this tenant's
    dataset prefix, write-only on this tenant's model-registry path. They
    enter the guest for one run and die with the VM."""
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=7200,                       # backstop: a wedged job reaps itself
        metadata={"tenant_id": tenant_id, "job_id": job_id},  # audit + billing
    ) as sbx:
        # The tenant's untrusted training script. We never reviewed it and
        # never trust it. It runs behind a hardware boundary, not next to it.
        sbx.filesystem.write("/workspace/train.py", train_script)

        # Creds as a file a specific process reads, NOT a shared env. Scoped
        # to THIS tenant's dataset (read) + registry path (write), short-lived.
        sbx.filesystem.write("/workspace/creds.json", _to_json(creds))

        # Runner: point the data + registry SDKs at the injected creds and a
        # tenant-scoped prefix, then hand off to the tenant's train.py.
        runner = (
            "import json, os\n"
            "c = json.load(open('/workspace/creds.json'))\n"
            "os.environ['DATASET_URI']    = c['dataset_uri']\n"
            "os.environ['DATASET_TOKEN']  = c['dataset_token']\n"
            "os.environ['REGISTRY_URI']   = c['registry_uri']\n"
            "os.environ['REGISTRY_TOKEN'] = c['registry_token']\n"
            "exec(open('/workspace/train.py').read())\n"
        )
        sbx.filesystem.write("/workspace/run.py", runner)

        # timeout_seconds is the circuit breaker; it kills THIS vm only.
        result = sbx.exec("python3 /workspace/run.py", timeout_seconds=7000)
        if result.exit_code != 0:
            raise RuntimeError(result.stderr)

        # Pull the metrics/manifest the job wrote; the adapter itself went
        # straight to the registry via the write-scoped token.
        manifest = sbx.filesystem.read("/workspace/adapter/manifest.json")
        return {"tenant": tenant_id, "job": job_id, "manifest": manifest}
    # VM gone here: the dataset it downloaded, the creds file, any cached
    # token, and every checkpoint on the guest disk vanish with it.

The load-bearing details: the dataset credential is read-only and scoped to this tenant's prefix, so even a hostile script can't reach another customer's corpus; the registry credential is write-only to this tenant's path, so a poisoned job can't overwrite another tenant's model; the creds live in a file a specific process reads, not a persistent environment; and `metadata` tags the run with tenant and job ids so a suspicious egress attempt or a billing line traces to exactly one customer. When the `with` block exits, the VM is destroyed — the downloaded dataset, any STS session the SDK cached, and every checkpoint on the guest disk go with it. There is no warm worker to inherit them.

The operational rules around the VM

Isolating the machine is necessary but not sufficient. A perfectly isolated VM handed a permanent, broadly-scoped key with open egress is a well-contained way to leak everything. The VM contains the code; it does not sanitize the powers you grant it or the network you let it reach.

• Mint short-lived, per-run credentials. Use STS / temporary tokens valid for the job's expected duration plus a margin, not forever. If a leak happens, the key is dead before an attacker can drain the dataset with it. Never inject a long-lived account-level key into a guest running tenant training code.
• Split read from write. The dataset credential should be read-only on exactly one tenant's prefix; the registry credential should be write-only on exactly one tenant's model path. A job that turns hostile then can't overwrite its own training data or clobber a neighbor's model — the powers are shaped like the task, not like the account.
• Allowlist egress to two endpoints. The guest should reach the dataset store and the model registry, and nothing else — no arbitrary internet, no internal services, no metadata endpoint (169.254.169.254). Dependency installs, if you allow them, go through a pinned internal mirror, not open PyPI. If a script reads the dataset and tries to POST it somewhere, the packet has nowhere to go.
• Cap it with ttl_seconds and a timeout. A training loop that spins forever, forks a miner, or blows its budget hits a wall and gets reaped — it kills one VM, not the fleet. The ttl is your backstop for the leaked-VM case; the exec timeout is your circuit breaker for the wedged-job case.
• Attribute every run. Tag the sandbox with tenant and job ids so a runaway job, a denied egress attempt, or an anomalous read pattern traces back to exactly one customer for audit, billing, and revocation.

Shared worker vs. microVM-per-job

Laid side by side, the difference isn't subtle.

• Data blast radius — Shared worker: one tenant's script runs in a process that holds another tenant's dataset and key. microVM-per-job: one tenant's data + key, one tenant's code, one disposable machine; nothing crosses.
• Credential lifetime — Shared worker: keys cached in a long-lived SDK, potentially reused across jobs. microVM-per-job: short-lived, split read/write, destroyed at teardown.
• Untrusted-script containment — Shared worker: a shared kernel means a kernel LPE or escape reads the whole host. microVM-per-job: a separate guest kernel under KVM; an escape must beat the hypervisor, not a namespace.
• Noisy neighbor — Shared worker: a runaway loop starves every queued job on the box. microVM-per-job: a runaway loop hits its own ttl and dies alone.
• Cleanup — Shared worker: scrub /tmp, invalidate caches, hope. microVM-per-job: teardown is total and free; there's nothing to scrub.

Wiring it into your training pipeline

In practice the flow is: a fine-tune request lands (an API call, a queue message, a scheduled retrain), your control plane mints a short-lived read-scoped dataset credential and a write-scoped registry credential for that tenant, and you dispatch the job into a fresh microVM with those credentials and an egress allowlist. Here's the dispatch shape from the outside — mint, run, revoke, and let the VM's teardown handle the rest.

def dispatch_finetune(tenant, job):
    # 1. Mint run-scoped creds: READ-only on this tenant's dataset prefix,
    #    WRITE-only on this tenant's registry path. Short-lived.
    creds = mint_scoped_training_creds(
        tenant_id=tenant.id,
        dataset_prefix=f"datasets/{tenant.id}/",   # read-only
        registry_path=f"models/{tenant.id}/{job.id}/",  # write-only
        ttl_seconds=7200,
    )

    # 2. Run the untrusted job in a throwaway VM (see run_finetune_job).
    #    The guest's egress is allowlisted to the dataset store + registry
    #    only -- no metadata service, no internal APIs, no open internet.
    out = run_finetune_job(
        tenant_id=tenant.id,
        job_id=job.id,
        train_script=job.train_script,
        creds=creds,
    )

    # 3. Nothing to scrub: the VM (and the dataset + creds inside it) are
    #    already gone. Revoke the STS session so a copied token is dead
    #    even off-box.
    revoke_scoped_creds(creds)
    return out

Notice what you don't have to do: there's no warm worker to wipe, no dataset directory to sweep, no SDK cache to invalidate, no lingering session to hope the next job doesn't inherit. The credential's lifetime is fenced twice — by its own short expiry and by the VM's teardown — and you revoke the session at the end so even a token copied out mid-run is dead. Belt, suspenders, and a machine you were going to throw away anyway.

Putting it together

Letting customers fine-tune on their own data is a genuinely valuable feature and a genuine liability, and the liability is entirely about where the dataset and the key live while untrusted training code runs. Don't hold them in a shared worker, where a poisoned script reads the raw key, an SSRF pulls the host's broader role, and a cached session bleeds into the next tenant's job. Run each fine-tune in its own Firecracker microVM: inject short-lived, read-scoped dataset creds and write-scoped registry creds per run, block the metadata endpoint, allowlist egress to the two endpoints the job actually needs, cap it with a ttl, and let teardown destroy the dataset and the keys the moment the job ends. The compromised training script still happens. It just gets one tenant's data, a dead key, and a locked door instead of your whole customer base.