Per-Tenant Scheduled Report Generation in microVMs

Somewhere in your product is a scheduler that runs every night, and for each customer it does roughly the same thing: pull their data, run a report, ship it. Maybe it renders a PDF dashboard, maybe it runs a customer-defined SQL query against their warehouse, maybe it fills a template with their numbers and emails it out. It's the unglamorous machinery of B2B SaaS, and it quietly holds two things that don't belong together: every tenant's database credential, and a pile of customer-supplied templates and queries that you never wrote and can't fully trust.

Run all of that in a shared cron worker — one process that loops over tenants, connecting to each one's database and rendering each one's template in turn — and you've built a machine that touches every customer's data with every customer's credential, in a process that also evaluates every customer's untrusted content. A SQL query that escapes into a shared connection, a report template that fetches a remote image (hello, SSRF), a Jinja or Handlebars template that turns out to be template-injection RCE, one tenant's runaway query starving the whole batch: any of these turns 'generate everyone's reports' into 'let anyone's report reach everyone's data.'

I'm Ajay, and I build PandaStack, a Firecracker microVM platform, so I spend my time on exactly this boundary: running content you didn't write next to credentials that matter. This post is about fanning out — one microVM per tenant per scheduled run, per-run database credentials scoped to that one tenant, egress locked to that tenant's data source and delivery endpoint, and teardown that destroys the credential and the report bytes the instant the run finishes. (If you've read the piece on per-customer cron jobs, this is its data-touching cousin: same fan-out instinct, but focused on the specific hazards of report generation — untrusted templates, live SQL, and delivery.)

Why the shared cron worker leaks

The shared batch worker is efficient and it's a cross-tenant incident with a crontab. The problem isn't that report generation is inherently unsafe; it's that a single long-lived process holding every tenant's credential while evaluating every tenant's content has too many seams.

• One process holds every tenant's DB credential. To generate everyone's report, the worker connects to everyone's database. A bug, an injection, or a compromised dependency in the rendering path now sits on a process with reach into every customer's data at once. The credential's power isn't scoped to the tenant whose report is currently running.
• Customer SQL runs against live connections. If tenants define their own report queries, that SQL executes against a database connection. Injection, an unscoped query that reads a shared table, or a query that runs for an hour and starves the batch — all of it lands because the query ran where the credential lives.
• Report templates are a code-execution surface. Server-side template engines (Jinja2, Handlebars, ERB, Liquid) are powerful enough that a hostile template string is server-side template injection — often a direct path to RCE. A template that renders a customer-controlled field without sandboxing the engine is running customer code in your worker.
• Templates fetch remote content. A report that embeds a logo by URL, pulls a chart from a service, or references an external stylesheet gives the renderer a reason to make outbound requests — including, via SSRF, to the cloud metadata endpoint (169.254.169.254) that hands out the host's role.
• Noisy neighbors ruin the whole batch. A single tenant's expensive query or infinite-loop template in a shared worker inflates latency for every tenant behind it, and an OOM or crash can take down reports that hadn't run yet.

One microVM per tenant per run

The fix is to stop running the batch in one process. When the schedule fires, fan out: one Firecracker microVM per tenant per run. Each VM boots its own guest kernel, confined by KVM hardware virtualization, and holds exactly one thing — the current tenant's report job, with a credential scoped to that one tenant. A template-injection RCE or a runaway query is now trapped in a disposable machine that can only see one customer's data and can only talk to that customer's endpoints. When the report is done, the VM is destroyed, and the credential and the rendered bytes go with it.

Two properties make it a wall. First, the credential is per-run and per-tenant: it's injected into one guest for one report and dies with it, so there's no shared connection pool spanning tenants and no cached credential for the next job to inherit. Second, the guest has its own network namespace, so you decide what it can reach — that tenant's database and the delivery endpoint (an email relay, a storage bucket, a webhook), and nothing else. No metadata service, no other tenant's database, no arbitrary internet. A hostile template inside that VM can render a wrong report about one tenant's data, and that's the ceiling.

Fan-out is practical because create is cheap. PandaStack restores a baked snapshot of an already-booted machine rather than cold-booting: the restore step is around 49ms and an end-to-end create is p50 179ms and p99 about 203ms (a true cold boot happens only on the first spawn of a template, around 3s). A nightly batch across thousands of tenants is thousands of ~180ms creates spread over a window — cheap enough that per-run isolation stops being a cost you argue about. If a report needs a live managed database, note that stands up in 30–90s and is a different, heavier resource than a report-render VM.

A worked example: render one tenant's nightly report

Here's the concrete shape for a single tenant's run. We create a fresh microVM, inject a read-scoped credential for exactly this tenant's data, write in the (untrusted) report template, render it, hand the output to delivery, and destroy the VM.

from pandastack import Sandbox

def render_tenant_report(tenant_id: str, run_id: str, template: str,
                        creds: dict) -> dict:
    """Render ONE tenant's scheduled report in its own microVM.
    `creds` are minted for THIS run, scoped to THIS tenant's data (read-only)
    and delivery endpoint. They enter the guest for one report and die with
    the VM."""
    with Sandbox.create(
        template="code-interpreter",
        ttl_seconds=900,                      # backstop: a runaway render reaps itself
        metadata={"tenant_id": tenant_id, "run_id": run_id},  # audit + billing
        # Create with egress allowlisted to this tenant's DB endpoint + the
        # delivery endpoint ONLY. No metadata service, no other tenant.
    ) as sbx:
        # The customer-supplied template. We render it in a SANDBOXED engine,
        # inside a VM, so template-injection has no host to inject into.
        sbx.filesystem.write("/report/template.j2", template)

        # Read-scoped creds as a file a specific process reads, not a shared env.
        sbx.filesystem.write("/report/creds.json", _to_json(creds))

        runner = (
            "import json, os\n"
            "c = json.load(open('/report/creds.json'))\n"
            "os.environ['DB_DSN']      = c['db_dsn']        # read-only role\n"
            "os.environ['DELIVER_URL'] = c['deliver_url']\n"
            "os.environ['DELIVER_KEY'] = c['deliver_key']\n"
            # render.py runs the tenant's SQL (read-only) + renders the
            # template with a SANDBOXED engine + posts to DELIVER_URL.
            "exec(open('/report/render.py').read())\n"
        )
        sbx.filesystem.write("/report/run.py", runner)

        # timeout_seconds is the circuit breaker for a runaway query/template;
        # it kills THIS vm only, not the rest of the batch.
        result = sbx.exec("python3 /report/run.py", timeout_seconds=600)
        if result.exit_code != 0:
            raise RuntimeError(result.stderr)

        return {"tenant": tenant_id, "run": run_id, "log": result.stdout}
    # VM gone here: the credential, the DB connection, the rendered PDF, and
    # any remote content the template pulled in all vanish with it.

The load-bearing details: the database role is read-only and scoped to this one tenant, so injection or an unscoped query hits an access-denied wall instead of a neighbor's data; the credential lives in a file a specific process reads, not a shared pool; the template renders in a sandboxed engine inside a VM, so template-injection has nowhere to escalate; egress is allowlisted to the tenant's data source and the delivery endpoint only, so a remote-image reference or an SSRF attempt goes nowhere; and `metadata` ties the run to one tenant for audit and billing. When the `with` block exits, the VM and the credential inside it are gone.

Fanning out across the whole tenant list

The batch itself is just the same call across every tenant, each isolated from the next. Tie it to your scheduler — PandaStack's cron schedules can trigger the fan-out — mint each tenant's scoped credential at dispatch time, and let each VM's teardown clean up after itself. Because each agent pre-allocates 16,384 /30 subnets, every VM gets its own network namespace, so per-tenant egress allowlisting is the default rather than a scaling problem.

def run_nightly_batch(tenants):
    """Cron fires -> fan out one isolated microVM per tenant. A failure or a
    runaway in one tenant's run is contained to that tenant's VM; the rest of
    the batch is unaffected."""
    for tenant in tenants:
        # Mint per-run creds: READ-only on this tenant's report data, plus a
        # delivery key. Short-lived, dies around the run's timeout.
        creds = mint_report_creds(
            tenant_id=tenant.id,
            db_role=f"report_ro_{tenant.id}",   # read-only, this tenant only
            deliver_url=tenant.delivery_endpoint,
            ttl_seconds=900,
        )
        try:
            render_tenant_report(
                tenant_id=tenant.id,
                run_id=new_run_id(),
                template=tenant.report_template,   # untrusted
                creds=creds,
            )
        except Exception as e:
            # One tenant's failure is logged and skipped, not a batch abort.
            log_report_failure(tenant.id, e)
        finally:
            revoke_creds(creds)   # dead even if copied out mid-run

For a steady stream of same-shaped reports, you can snapshot a VM that already has your rendering toolchain and fonts warmed and fork it per tenant (same-host fork 400–750ms, cross-host 1.2–3.5s) so each report starts from a known-good image without a full create — and still dies, credential and all, when its report is done.

Shared cron worker vs. microVM-per-run

• Credential reach — Shared worker: one process holds every tenant's credential; a bug touches everyone. microVM-per-run: one tenant's read-only credential per VM; a bug touches one tenant.
• Template injection — Shared worker: SSTI in the render path is RCE on a host with every credential. microVM-per-run: SSTI is trapped in a disposable guest with one scoped credential and no egress.
• Runaway query — Shared worker: an hour-long query starves the whole batch. microVM-per-run: it hits its own timeout and dies alone.
• SSRF via remote content — Shared worker: the renderer can reach the metadata endpoint and internal network. microVM-per-run: egress allowlisted to the data source + delivery only.
• Cleanup — Shared worker: reset connections, scrub temp files, hope. microVM-per-run: teardown is total and free.

Putting it together

Scheduled report generation is a feature every B2B product ships and a place where a lot of trust quietly accumulates: every tenant's credential and every tenant's untrusted template, all in one nightly process. Don't run the batch in a shared worker, where injected SQL touches shared connections, a template-injection is RCE on a host with everyone's keys, and one tenant's runaway query sinks the batch. Fan out one Firecracker microVM per tenant per run: inject a read-only, per-tenant credential, render the template in a sandboxed engine inside the guest, allowlist egress to the data source and delivery endpoint, cap it with a ttl, and let teardown destroy the credential and the report bytes when the run ends. The malicious template still gets uploaded. It just renders a wrong report about one tenant's data, in a locked room you were about to demolish.