Per-Tenant MicroVM Isolation for PDF and Invoice Generation

Every billing product eventually ships the same feature: let customers upload their own invoice template. Their logo, their fonts, their payment terms, their weirdly specific column order that the finance team in Düsseldorf insists on. It ships as a text box in the settings page, it gets a friendly label like "Customize your invoice," and from that day forward your billing service evaluates strings written by strangers. Your invoice template engine is a scripting language you accidentally exposed to the internet, and nobody wrote it down in the threat model because it lives under the word "template."

I'm Ajay, and I build PandaStack, a Firecracker microVM platform, so I spend a lot of time on this exact boundary: running content you didn't write next to data that matters. And invoice rendering is a uniquely bad place to be careless, because the machine that renders the template is also the machine that has every tenant's invoice — names, amounts, addresses, tax IDs, and the occasional payment detail. Render everyone's templates in one shared worker and you have built a very efficient system for turning one customer's template into every customer's billing history.

This post is about the fix: one microVM per render job, egress blocked (including the cloud metadata endpoint that hands out your host's IAM role), and a snapshot-baked image with the fonts and toolchain already warm so the isolation doesn't cost you a cold start on the last day of the month when every invoice in the system wants to render at once.

Customer templates are remote code execution in a trenchcoat

The comforting mental model is that a template is data: a document with holes, and you fill the holes with values. The uncomfortable truth is that every template engine worth using is Turing-adjacent, and the ones that aren't still run inside a host language whose objects the template can usually reach. "Customer-supplied template" is a polite way of saying "customer-supplied program that runs on your server, with your process's privileges, on demand."

• Server-side template injection (SSTI). Jinja2, Handlebars, Liquid, ERB, Twig, Velocity — the classic escalation is walking the object graph from a harmless-looking expression to the runtime's class hierarchy, then to a subprocess call. The payload is a one-liner. Sandboxed modes exist and they are worth enabling, but they are a moving target and their CVE histories are not short.
• LaTeX is a full programming language with a shell in it. If your "professional" invoice path is pdflatex, then \\write18 is command execution by design, \\input{/etc/passwd} reads whatever the process can read, and \\openin will happily slurp a file into a macro that renders into the PDF you then email to the attacker. Shell-escape is off by default in most distributions, which is a comfort until you meet the ticket where someone turned it on to make a diagram package work.
• Headless Chrome is a browser, and browsers fetch things. HTML-to-PDF via Puppeteer or Playwright means the template can include <img src>, <link>, <iframe>, fetch(), and <script>. Unless you've blocked it, that page can load file:///etc/shadow into an element and screenshot it, or hit http://169.254.169.254/ and put your instance's IAM credentials in the rendered document. The exfiltration channel is the PDF itself, which you then dutifully deliver.
• Font and image pipelines are C code with a long CVE tail. FreeType, HarfBuzz, libpng, Ghostscript, ImageMagick — a hostile font file or a crafted SVG referenced by a template lands in a parser that has historically been a reliable source of memory-corruption bugs.
• Even "safe" templates make outbound requests. A remote logo URL is an SSRF primitive pointed at your internal network by a customer who only had to paste a URL into a settings field.

Why a shared render worker is a cross-tenant incident waiting for a cron

Almost everyone builds the render pool the same way: a queue, a fleet of worker processes, and each worker pulls jobs from every tenant in turn. It's the obvious design and it's efficient, and it means a single worker process, over the course of an hour, will have loaded Tenant A's template, Tenant B's invoice line items, Tenant C's logo, and Tenant D's customer addresses — all into one address space, on one filesystem, behind one set of cloud credentials.

Now consider what a successful SSTI in Tenant A's template gets. Not just code execution — code execution on a box that has a database connection scoped to every tenant, a temp directory holding half-written PDFs from the last twenty jobs, an S3 credential that can read the invoice archive, and a network route to your internal services. The attacker doesn't need to break out of a container to have a very good day; they only need to read /tmp and wait.

• Residue on disk. Rendering is a file-heavy operation: temp HTML, intermediate PostScript, cached remote images, the output PDF itself. Cleanup is a best-effort script, and best-effort scripts rot. One tenant's job routinely runs on a filesystem holding another tenant's documents.
• One credential set, all tenants. The worker needs the database and the object store to do its job, so it has both, for everyone. Scoping per-job inside a shared process is a convention, not a boundary.
• Shared kernel, shared page cache. Container isolation is namespaces on one kernel. A kernel LPE or a side channel turns a render-job compromise into a host compromise with every co-tenant job on it.
• Noisy neighbors, which on the 1st of the month is not hypothetical. One tenant's accidentally-infinite template loop, or a 400MB image reference, pins CPU and RAM in a worker shared by everyone in the queue behind them.
• Blast radius equals queue depth. Whatever the compromised worker touched, it touched for every job it ever ran — so incident response starts with "which tenants shared that pod," which is a question you never want to have to answer to an auditor.

One microVM per render job

The structural fix is to stop sharing the machine. Each render job — one tenant, one invoice batch, one template — gets its own Firecracker microVM: its own guest kernel, its own filesystem, its own network namespace, confined by KVM hardware virtualization rather than by shared-kernel namespaces. It receives exactly the data it needs to render, it renders, it hands back bytes, and then it is destroyed. Not cleaned. Destroyed.

In that model, the SSTI still succeeds. \\write18 still runs. Headless Chrome still tries to fetch 169.254.169.254. The difference is what those primitives can reach: a disposable guest containing one tenant's data, with no credentials worth stealing, no route to your internal network, and a lifetime measured in seconds. The attacker has achieved remote code execution on a machine you were about to delete, holding data they already supplied most of.

The reason this is practical rather than merely correct is that create is cheap. PandaStack restores a baked snapshot of an already-booted machine instead of cold-booting: the restore step is around 49ms, and an end-to-end create is p50 179ms with p99 around 203ms. A true cold boot only happens on the first spawn of a template, at roughly 3 seconds. Per-job isolation stops being a line item you argue about in a design review when it costs less than the average database round trip your renderer already makes.

Block egress, and specifically block 169.254.169.254

A microVM with open internet access is a well-isolated way to exfiltrate one tenant's invoices. Isolation of compute without isolation of network is half a wall. For a PDF renderer the good news is that the correct egress policy is unusually simple: for most invoice templates it is nothing at all.

Every sandbox gets its own network namespace — each agent pre-allocates 16,384 /30 subnets, so per-job network isolation is the default rather than a scaling problem — which means "this guest can reach nothing" is a configuration, not an architecture project. Three rules cover the realistic cases:

1. Default deny on egress. Resolve and fetch every remote asset — logos, web fonts, embedded images — outside the sandbox, ahead of time, through a fetcher that validates the URL, enforces a size cap, and refuses private address ranges. Write the resulting bytes into the guest as files. The renderer then works entirely offline, which also makes it deterministic and much faster.
2. Block the link-local metadata range explicitly, not just as a side effect of default-deny. 169.254.169.254 is the single highest-value target for an SSRF in a renderer, because on most clouds it hands out the host's role credentials to anyone who asks nicely with an HTTP GET. Block 169.254.0.0/16 and every RFC1918 range, and verify it from inside the guest rather than trusting the config.
3. Run headless Chrome with the seatbelt on anyway. Disable file:// access, disable local file access from files, disable remote fonts if you've pre-baked them, and treat every flag as a second layer rather than the layer. Chrome flags are a hardening measure; the network namespace is the boundary.

# Run INSIDE the render microVM as a pre-flight assertion. If any of these
# succeed, the job should fail closed rather than render an invoice that
# might contain your instance role credentials.
set -uo pipefail

fail=0

# 1. Cloud metadata endpoint must be unreachable. This is the one that
#    turns 'a customer pasted a URL' into 'a customer has your IAM role'.
if curl -s --max-time 2 http://169.254.169.254/latest/meta-data/ >/dev/null; then
  echo 'FAIL: metadata endpoint reachable from render guest'; fail=1
fi

# 2. General egress must be closed. Templates render from pre-fetched,
#    validated assets on local disk -- never from the live internet.
if curl -s --max-time 2 https://example.com >/dev/null; then
  echo 'FAIL: outbound internet reachable from render guest'; fail=1
fi

# 3. If LaTeX is in the toolchain, shell-escape must be off. \write18 is
#    command execution as a documented feature, which is a sentence that
#    should make you check your config more often than you do.
if pdflatex --version >/dev/null 2>&1; then
  kpsewhich -var-value shell_escape | grep -qx 'f' \
    || { echo 'FAIL: LaTeX shell-escape is not disabled'; fail=1; }
fi

[ "$fail" -eq 0 ] && echo 'preflight ok: renderer is deaf, dumb, and offline'
exit "$fail"

A worked example: render one tenant's invoice batch

Here's the concrete shape. Create a fresh microVM from a snapshot that already has the renderer, fonts, and toolchain baked in; write in the untrusted template plus pre-fetched assets and the invoice data for exactly one tenant; render; read the PDF back out; destroy the VM.

from pandastack import Sandbox

def render_invoice_pdf(tenant_id: str, job_id: str, template_html: str,
                       invoice_json: str, logo_png: bytes) -> bytes:
    """Render ONE tenant's invoice in its own disposable microVM.

    template_html is UNTRUSTED customer input. logo_png was fetched OUTSIDE
    the sandbox by a validating fetcher (size-capped, private ranges refused)
    so the guest itself needs no network at all.
    """
    with Sandbox.create(
        template="browser",              # headless Chrome + fonts baked in
        ttl_seconds=180,                 # backstop: a runaway render reaps itself
        metadata={"tenant_id": tenant_id, "job_id": job_id},  # audit + billing
        # Created with egress DENIED: no internet, no 169.254.169.254,
        # no route to internal services. The renderer works offline.
    ) as sbx:
        # Untrusted template + this tenant's data ONLY. No other tenant's
        # invoices exist on this filesystem, so none can leak from it.
        sbx.filesystem.write("/job/template.html", template_html)
        sbx.filesystem.write("/job/invoice.json", invoice_json)
        sbx.filesystem.write("/job/assets/logo.png", logo_png)

        # Render with the seatbelt on. These flags are layer two; the VM
        # boundary and the empty network namespace are layer one.
        result = sbx.exec(
            "chromium --headless --no-sandbox --disable-gpu "
            "--disable-remote-fonts --allow-file-access-from-files=false "
            "--print-to-pdf=/job/out.pdf file:///job/render.html",
            timeout_seconds=120,         # circuit breaker for an infinite loop
        )
        if result.exit_code != 0:
            raise RuntimeError(f"render failed: {result.stderr[-2000:]}")

        return sbx.filesystem.read("/job/out.pdf")
    # VM destroyed here. Temp HTML, intermediate PostScript, cached images,
    # the PDF, and anything a hostile template managed to write: all gone.
    # Not cleaned up. Gone.

The load-bearing details: assets are fetched and validated outside the guest so the renderer needs no network; the guest holds exactly one tenant's data, so cross-tenant leakage has nothing to leak; `timeout_seconds` is the circuit breaker for the infinite template loop, and it kills this VM only, not the queue behind it; `ttl_seconds` reaps the job even if your orchestrator falls over mid-render; and `metadata` ties the VM to one tenant for audit. When the `with` block exits, the entire machine is destroyed — which is the only cleanup routine that has never had a bug.

Bake the fonts and toolchain into a snapshot

A PDF toolchain is heavy. Chromium, a full TeX distribution, Ghostscript, and a font set that covers your customers' scripts is gigabytes of install and tens of seconds of warm-up. Doing that per job would drown the millisecond create under minutes of apt and fontconfig, which is exactly the objection people raise when you propose per-job VMs.

So don't do it per job. Install once into a sandbox, warm the font cache, launch the renderer once so its caches are populated, then snapshot the machine. Every subsequent render restores that exact state — fonts already indexed, binaries already in page cache — and starts rendering immediately. For a steady stream of same-shaped invoices you can go a step further and fork the snapshot per job instead of creating fresh: same-host fork is 400–750ms and cross-host is 1.2–3.5s, and each fork gets its own copy-on-write memory and disk, so one job's writes are invisible to every other.

# Run ONCE in a prep sandbox, then snapshot it. Every render job restores
# this state instead of paying for it, so isolation costs milliseconds
# rather than an apt-get.
set -euo pipefail

# 1. Renderer + the font set your customers actually use. Missing CJK glyphs
#    is the #1 support ticket for invoice PDFs, so bake them now.
apt-get install -y --no-install-recommends \
  chromium fonts-dejavu fonts-liberation fonts-noto-core fonts-noto-cjk

# 2. LaTeX path, if you offer one. Shell-escape OFF, permanently, in the
#    config -- not just as a CLI flag someone can forget to pass.
apt-get install -y --no-install-recommends texlive-latex-recommended
printf 'shell_escape = f\n' >> /usr/share/texmf/web2c/texmf.cnf

# 3. Warm the caches so the first render after restore is not the slow one.
fc-cache -f >/dev/null
chromium --headless --no-sandbox --disable-gpu \
  --print-to-pdf=/tmp/warm.pdf about:blank
rm -f /tmp/warm.pdf

# 4. Strip anything a hostile template could use to phone home. The network
#    namespace already blocks egress; this removes the tooling too.
apt-get purge -y curl wget netcat-openbsd >/dev/null 2>&1 || true

echo 'ready to snapshot: fonts indexed, chromium warm, no network tools'

One deliberate detail in there: removing curl and wget from the image after the toolchain is installed. The network namespace is what actually stops exfiltration — this is belt-and-braces, and it costs nothing. If a template achieves code execution and reaches for the obvious tool, it finds an empty holster in a room with no doors.

Shared render pool vs. microVM per job

• SSTI outcome — Shared pool: code execution on a host with every tenant's DB and object-store credentials. Per-job microVM: code execution in a disposable guest holding one tenant's data and no credentials.
• LaTeX \write18 / \input — Shared pool: reads any file the worker process can read, including other jobs' temp files. Per-job microVM: reads a filesystem containing this tenant's invoice and a font cache.
• Chrome hitting 169.254.169.254 — Shared pool: instance role credentials rendered into a PDF you then email out. Per-job microVM: connection refused; egress denied at the namespace.
• Temp-file residue — Shared pool: half-written PDFs from other tenants sitting in /tmp between jobs. Per-job microVM: the filesystem is destroyed with the VM; there is no between.
• Noisy neighbor on the 1st of the month — Shared pool: one runaway template pins a worker shared by the whole queue. Per-job microVM: it hits its own timeout and dies alone.
• Isolation boundary — Shared pool: namespaces on one shared kernel. Per-job microVM: own guest kernel under KVM hardware virtualization.
• Cost of isolation — Shared pool: free, until the incident. Per-job microVM: ~49ms restore, p50 179ms end-to-end create.
• Incident scope — Shared pool: 'which tenants shared that pod?' Per-job microVM: one tenant, one job, one destroyed VM.

Worth noting where competitors sit: container-based PDF services and serverless render functions generally give you process and namespace isolation on a shared kernel, and many managed HTML-to-PDF APIs run your template on multi-tenant infrastructure with outbound network enabled by default because customers ask for remote assets. That's a qualitative statement about a fast-moving market — verify the current isolation and egress model against each vendor's own docs before you decide it's good enough for the thing that renders your customers' billing data.

The end-of-month spike is an isolation problem too

Invoicing load is not smooth. It is flat for twenty-eight days and then everyone's billing cycle closes within the same few hours, and the render queue goes from idle to a five-figure backlog because that's what "monthly" means. In a shared pool this is where the security argument and the performance argument merge: the tenant with the 900-page consolidated invoice and the tenant with the badly-nested template are now competing for the same workers as everyone else, and the worst template in the system sets the p99 for the whole customer base.

One VM per job makes throughput per-tenant by construction. Each job has its own CPU and memory allocation, its own timeout, and its own failure. A tenant who writes a template with an accidental O(n²) loop over line items gets slow invoices; nobody else notices. And because there's no warm pool to keep heated, the idle twenty-eight days cost you nothing — you scale from zero to thousands of concurrent renders in the time it takes to restore that many snapshots, and back to zero when the month rolls over.

Putting it together

Invoice rendering looks like the most boring service in your architecture, and it's the one that combines customer-supplied code with every customer's financial data. The template engine is a scripting language, LaTeX has a shell in it, and headless Chrome is a browser that will read file:// and call the metadata endpoint if you let it. Sandbox the engine, sanitize the HTML, pass --no-shell-escape — do all of it. Then put the whole thing in a room that survives all of it failing at once.

One Firecracker microVM per render job: a baked snapshot with fonts and toolchain warm so create is a ~49ms restore rather than an apt-get; one tenant's data and nothing else on the filesystem; egress denied with 169.254.169.254 explicitly blocked and verified from inside the guest; a timeout and a TTL so a runaway template dies alone; and teardown that destroys the machine rather than cleaning it. The hostile template still gets uploaded. It still executes. It just produces an ugly PDF about one tenant's own invoice, in a locked room with no windows that you were about to demolish anyway.