Per-Tenant Object Storage Isolation with microVMs

You run a multi-tenant SaaS, and somewhere in it lives a feature you'd rather not think about: tenants hand you code or configuration that runs against object storage. An image-resize step, an ETL transform, a customer-supplied plugin that reads a CSV from a bucket, munges it, and writes the result back. Maybe it's a full script, maybe it's a declarative transform you interpret, maybe it's a container image the tenant pushed. Whatever the shape, it needs to touch storage — pull an object from S3, GCS, or your MinIO cluster, and put an object back. And to touch storage, something has to hold a credential.

That credential is the whole problem. The moment a tenant's transform runs in a process that holds storage keys, you've built a machine whose only job is to hand those keys to code you didn't write. If that code is malicious — or merely compromised by a dependency, a prompt injection, or a crafted input — it can read the credential straight out of its environment and go wandering. The question isn't whether your IAM policy is tight. It's what happens when the code running under that policy turns hostile.

I'm Ajay — I build PandaStack, a Firecracker microVM platform, so I spend my days thinking about exactly where the boundary sits when you run other people's code next to your secrets. This post is about confining that blast radius: one microVM per tenant job, storage credentials injected per run and destroyed with the VM, and egress controls so a compromised transform can't read another tenant's bucket, exfiltrate your keys, or scope-creep its way across your storage account.

Scoped IAM alone is a fragile boundary

The reflexive answer to "don't let tenant A read tenant B's bucket" is scoped IAM: give the worker a role that can only touch the current tenant's prefix, and trust the policy. Scoping is necessary — but on its own, in a shared process or container, it's a boundary with holes you can drive through. The trouble is that a scoped policy governs what a credential is allowed to do; it does nothing about who inside the box gets to hold the credential, or which box the code can reach out and talk to.

• Credentials in the environment are readable by the code. If the worker holds a key in an env var, a config file, or a mounted secret, tenant-supplied code running in that same process reads it as easily as it reads its own input. Now the tenant has your raw credential — not just its effects — and can use it anywhere, on their own time, off your infrastructure.
• Metadata endpoints are one SSRF away. On a cloud host, the instance metadata service (169.254.169.254) will hand out the host's role credentials to anything that can make an HTTP request from that box. A tenant transform that can open a socket — or that you tricked into fetching an attacker URL — can pull the node's identity, which is almost always broader than the tenant's scoped role. SSRF-to-metadata is one of the most reliable escalation paths in multi-tenant compute.
• Credential caching outlives the request. SDKs cache credentials, refresh tokens, and STS sessions in memory and on disk. In a warm, reused worker, a credential minted for tenant A's job can still be sitting in the SDK's cache when tenant B's job lands on the same worker — and a shared kernel means a container escape or kernel bug turns "tenant A's code" into "read everything on the host."
• Scope creep is a filter, not a wall. "The transform only writes to its own prefix" usually means "the code is supposed to only write to its own prefix." A path-traversal in the object key, a wildcard that's broader than you thought, or a role that's reused across tenants turns a supposed boundary into a suggestion.

One microVM per tenant job

The fix is to stop sharing the machine that holds the credential. Run each tenant's storage job in its own Firecracker microVM. Each VM boots its own guest kernel and is confined by KVM hardware virtualization — the only way out is a tiny set of emulated virtio devices, not the full Linux syscall surface a container shares with its host. This is the same isolation model AWS Lambda and Fargate use to run untrusted code from thousands of customers on shared fleets. The credential you inject lives inside one VM, is used by one tenant's job, and is gone the instant that VM is destroyed.

Two properties make this a real boundary rather than a nicer-looking version of the same leak. First, the credential is injected per run and never persisted to a shared place — it enters the guest for one job and dies with the guest, so there's no cache for the next tenant's job to inherit. Second, the guest has its own network namespace, so you decide exactly what it can reach: your object store's endpoint and nothing else. No route to the metadata service, no route to your internal APIs, no route to another tenant's database. A compromised transform inside that VM can compute a wrong answer about its own tenant's data, and that's the ceiling on the damage.

The classic objection is startup cost — nobody wants a three-second VM boot in front of every resize job. PandaStack sidesteps that with snapshot-restore: every create restores a baked snapshot of an already-booted machine rather than cold-booting. The restore step is around 49ms; an end-to-end create is p50 179ms and p99 about 203ms. A true cold boot happens only on the very first spawn of a template (around 3s), after which you're paying restore prices. That's the trade that makes a VM-per-job practical: hypervisor-grade isolation at roughly container-grade latency.

A worked example: a tenant image-resize / ETL job

Here's the concrete shape. A tenant registered a transform that pulls source images from object storage, resizes them, and writes the results back. We run it in a fresh microVM, inject a short-lived credential scoped to exactly this tenant's prefixes, and destroy everything when it's done. The credential never touches a shared worker, and the VM never had a route to anything but the object store.

from pandastack import Sandbox

def run_tenant_transform(tenant_id: str, job_id: str, transform_code: str,
                         creds: dict) -> dict:
    """Run ONE tenant's storage-touching transform in its own microVM.
    `creds` are SHORT-LIVED, minted for THIS job, scoped to THIS tenant's
    prefixes only. They enter the guest for one run and die with the VM."""
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=300,                        # backstop: a leaked VM reaps itself
        metadata={"tenant_id": tenant_id, "job_id": job_id},  # audit + billing
    ) as sbx:
        # Write the tenant's untrusted transform into the guest. We never
        # reviewed it and never trust it.
        sbx.filesystem.write("/workspace/transform.py", transform_code)

        # Inject creds as a file the transform reads, NOT into a shared env.
        # They are scoped (this tenant's prefixes), short-lived (minted for
        # this job), and confined to this VM (gone at kill()).
        sbx.filesystem.write("/workspace/creds.json", _to_json(creds))

        # The runner points the storage SDK at the injected creds + a
        # tenant-scoped prefix. The transform only ever sees its own objects.
        runner = (
            "import json, os\n"
            "c = json.load(open('/workspace/creds.json'))\n"
            "os.environ['AWS_ACCESS_KEY_ID'] = c['access_key']\n"
            "os.environ['AWS_SECRET_ACCESS_KEY'] = c['secret_key']\n"
            "os.environ['AWS_SESSION_TOKEN'] = c['session_token']\n"
            "os.environ['PS_BUCKET'] = c['bucket']\n"
            "os.environ['PS_PREFIX'] = c['prefix']\n"
            "exec(open('/workspace/transform.py').read())\n"
        )
        sbx.filesystem.write("/workspace/run.py", runner)

        # timeout_seconds is the circuit breaker; it kills THIS vm only.
        result = sbx.exec("python3 /workspace/run.py", timeout_seconds=120)
        if result.exit_code != 0:
            raise RuntimeError(result.stderr)

        return {"tenant": tenant_id, "job": job_id, "log": result.stdout}
    # VM gone here: the transform, the creds file, every cached token,
    # and any half-written object handle vanish with it.

The load-bearing details: the credentials are written as a file the runner loads, so you control exactly which process sees them and they never leak into a persistent environment; they're scoped to this tenant's bucket and prefix, so even a hostile transform is confined to objects it was already allowed to touch; and `metadata` tags the run with tenant and job ids, so a suspicious egress attempt or a billing line traces back to exactly one customer. Most importantly, when the `with` block exits, the VM is destroyed — the creds file, any STS session the SDK cached, and every byte the transform touched are gone. There is no warm worker to inherit them.

Short-lived creds, per-tenant buckets, and egress allowlists

Isolating the machine is necessary but not sufficient. A perfectly isolated VM that you hand a permanent root storage key is a well-contained way to leak everything. The VM contains the code; it does not sanitize the powers you grant it or the network you let it reach. So the operational rules are about minimizing what the credential can do, how long it lives, and where the guest can send data.

• Mint short-lived, per-run credentials. Use STS / signed-token / temporary-credential mechanisms so the key you inject is valid for minutes, not forever, and expires around the job's timeout. If it leaks, it's dead before an attacker can drain a bucket with it. Never inject a long-lived root or account-level key into a guest that runs tenant code.
• Scope to one tenant's prefix — ideally one tenant's bucket. A credential restricted to a per-tenant bucket (or a tightly-scoped prefix under a shared bucket) means a path-traversal or a forgotten wildcard in the object key hits an access-denied wall instead of another tenant's data. Per-tenant buckets make the boundary structural rather than a policy condition the code could dodge.
• Block the metadata endpoint. The guest's network namespace should have no route to 169.254.169.254 (or the GCP/Azure equivalents). This single rule kills the most common credential-escalation path — an SSRF that pulls the host's role — because the box the code runs on physically cannot reach the metadata service.
• Allowlist egress to the storage endpoint only. The guest should be able to reach your object store and nothing else — no arbitrary internet, no internal services, no other tenant's database. If a transform reads an object and tries to POST it to an attacker's server, the packet has nowhere to go. Egress control is what turns "read my own bucket" into a job that can't smuggle the data out.
• Attribute every run. Tag the sandbox with tenant and job ids so a runaway job, a denied egress attempt, or an anomalous read pattern traces back to exactly one customer for audit and revocation.

Fresh-per-job vs. long-lived-per-tenant

There are two ways to hold the VM, and for storage-credential isolation the default should lean hard toward fresh-per-job.

• Fresh microVM per job — create a VM for the transform, inject the run-scoped credential, run it, kill it. This is the strongest posture: the credential exists for exactly one job, nothing survives between runs, and idle tenants cost nothing because there's no VM sitting around. At p50 179ms to create, a VM-per-job is cheap enough to be the default for image-resize, ETL steps, and webhook-driven transforms. This is where most per-tenant storage workloads should live.
• Long-lived microVM per tenant — for a tenant running a steady stream of jobs where per-job create latency matters, keep one persistent VM per tenant and route that tenant's jobs to it. Mark it persistent so the idle reaper leaves it alone. The hard rule: one tenant per persistent VM, forever, and still mint short-lived credentials per job rather than baking a long-lived key into the VM. The instant you reuse one persistent VM across two tenants, you've erased the boundary and rebuilt the shared-worker leak by hand.
• Fork a seeded VM for per-run isolation with warmth — snapshot a VM that already has the transform runtime and libraries loaded, then fork it per job. A same-host fork lands in 400–750ms (cross-host 1.2–3.5s) with copy-on-write memory and disk, so each job gets a private, disposable machine that starts from a known-good state without paying a full create, and still dies — credentials and all — when the job finishes.

Wiring it into your job pipeline

In practice the flow is: a job arrives (a webhook, a queue message, a scheduled ETL trigger), your control plane mints a short-lived storage credential scoped to that tenant, and you dispatch the transform into a fresh microVM with that credential and an egress allowlist. Here's the dispatch shape from the outside — mint, run, and let the VM's teardown handle credential destruction for you.

# Control-plane side: mint a short-lived, tenant-scoped credential per job,
# then dispatch the transform into its own microVM. The credential's lifetime
# is bounded by BOTH its own expiry AND the VM's teardown.

def dispatch_storage_job(tenant, job):
    # 1. Mint run-scoped creds (STS / temporary token) valid ~a few minutes,
    #    limited to this tenant's bucket + prefix, read+write only.
    creds = mint_scoped_storage_creds(
        tenant_id=tenant.id,
        bucket=tenant.bucket,          # ideally a PER-TENANT bucket
        prefix=f"jobs/{job.id}/",      # narrow the blast radius further
        ttl_seconds=300,               # dies around the job timeout
        permissions=["get_object", "put_object"],
    )

    # 2. Run the untrusted transform in a throwaway VM (see run_tenant_transform).
    #    The guest's egress is allowlisted to the storage endpoint only — no
    #    metadata service, no internal APIs, no open internet.
    result = run_tenant_transform(
        tenant_id=tenant.id,
        job_id=job.id,
        transform_code=job.transform_code,
        creds=creds,
    )

    # 3. Nothing to clean up: the VM (and the creds inside it) are already gone.
    #    Revoke the STS session too, so a copied token is dead even off-box.
    revoke_scoped_creds(creds)
    return result

Notice what you don't have to do: there's no warm worker to scrub, no `/tmp` to sweep for the credential file, no SDK cache to invalidate, no lingering session to hope the next job doesn't inherit. The credential's lifetime is fenced twice — by its own short expiry and by the VM's teardown — and you revoke the STS session at the end so even a token that was somehow copied out is dead. Belt, suspenders, and a machine you were going to throw away anyway.

Teardown is the security control

The reason this architecture stays safe under load is that destruction is total and cheap. When a job finishes — or blows its timeout, or the tenant churns — you kill the VM, and the injected credential, any STS session the SDK cached, the transform code, and every object it buffered in memory go with it. There's no worker to reset, no cached token to expire, no chance the next tenant's job inherits a stale credential. Fresh-per-job VMs teardown automatically via `ttl_seconds` and the `with` block; long-lived per-tenant VMs you `kill()` explicitly when the account closes. Either way, the credential leak surface between one job and the next is zero, because there is no next job on the same machine holding the same key.

Putting it together

Running tenant-supplied transforms against object storage is a genuinely useful feature and a genuine liability, and the liability is entirely about where the credential lives while untrusted code runs. Don't hold it in a shared worker, where the tenant's code can read the raw key, an SSRF can pull the host's broader role from the metadata endpoint, and a cached STS session bleeds into the next tenant's job. Run each tenant's job in its own Firecracker microVM: inject a short-lived, tenant-scoped credential per run, block the metadata endpoint, allowlist egress to the storage endpoint only, and let the VM's teardown destroy the credential the moment the job ends. Prefer per-tenant buckets so the boundary is structural, choose fresh-per-job as the default, and let snapshot-restore keep it fast enough that isolation isn't a tax. The compromised transform still happens. It just gets a dead key and a locked door instead of your whole storage account.