It is 9:04 on a Saturday. Two hundred people have registered, forty of them arrived with laptops whose Python installs are archaeological sites — a Homebrew 3.9, a pyenv 3.11 nobody remembers enabling, an Anaconda that has quietly taken over PATH — and the venue wifi has just discovered what happens when sixty people run npm install on a starter repo at the same moment. The organiser is walking between tables saying "have you tried turning off your VPN" with the voice of someone who has said it eleven times.
By noon somebody has a crypto miner running "for the demo". By 3:58 a team is pushing a fix to production on the same machine the judges are about to use. At 4:00 the judges want to run thirty projects, in order, on whatever state each one was in when the bell rang, and at least one of those projects only works on its author's laptop, which has gone to lunch.
I build PandaStack, an open-source Firecracker microVM platform, and this post is the event-day architecture I would actually run: one isolated machine per team, baked the night before with the event's toolchain, reached through a browser, frozen at the deadline, judged from copies, and torn down when the prizes are handed out. It is a sibling to two posts about classrooms — Building an autograder that runs student code safely in microVMs is about running submitted code against hidden tests, and The Best Code Execution Sandboxes for Education Platforms in 2026 is a buyer's guide for education platforms. A hackathon is a different animal. The code is not a submission, it is a living development environment with a dev server, a database and a team of four typing into it at once, and it has to survive for eight hours rather than eight seconds.
What actually goes wrong on the day
Every organiser has their own list. Mine, from running and attending these, sorts into four buckets, and it is worth being precise about them because each one wants a different fix:
- Environment drift. Forty laptops, forty Python installs, three operating systems, one person on a corporate machine that forbids installing anything. The first two hours of a one-day event go to setup instead of building. This is the most expensive failure, and it is the one nobody budgets for.
- Bandwidth. The venue network was sized for email. A starter repo with a 400 MB
node_modules, multiplied by sixty, is a denial-of-service attack that the attendees commit against themselves at 9:05, and then again every time someone adds a dependency. - Strangers with root. You do not know these people. Most are delightful. One will run a miner, one will port-scan the venue subnet to see what happens, and one will
curl | sudo bashsomething from a forum thread at 2 a.m. during an overnight event, which is how a hackathon machine ends up in someone else's botnet. - Judging. Demos break between 3:58 and 4:05 with remarkable reliability. Teams keep committing after the deadline "just to fix the README". Judges need to run thirty projects in sequence, and the project that only works on one laptop is the project with the dead battery.
The architecture below fixes the first two by moving the environment off the laptop, the third with a hardware isolation boundary plus a few limits, and the fourth by freezing state instead of trusting people.
The shape: one machine per team, reached through a browser
The unit is a sandbox per team (or per attendee, for a workshop where everybody follows the same exercise). Each sandbox is a Firecracker microVM with its own guest kernel, its own network namespace and TAP device, and a copy-on-write root filesystem cloned from the event template. Every team starts from byte-identical state, and nothing a team does can reach another team's machine or the host underneath.
The options people actually reach for, compared on the axes that matter at an event:
- Attendees' own laptops — Setup: two hours of yours, per event. Isolation: perfect, nobody shares anything. Judging: whatever survives lunch. Cost: zero, paid in the first two hours.
- One big shared VM with user accounts — Setup: one machine to prepare. Isolation: Unix permissions between strangers, plus a shared kernel, shared CPU and a shared
/tmp; the miner slows everybody. Judging: everyone's state is live and mutable until you take the machine away from them. - Containers on a shared host — Setup: a good image fixes drift. Isolation: better than user accounts, but every container shares the host kernel, and "give each team root so they can apt-get" is exactly the configuration that kernel bugs get exploited from. A container is a polite suggestion to the kernel. Judging: snapshotting a running container's memory is possible but rarely what anyone has wired up.
- Hosted cloud dev environments (Codespaces, Gitpod and the like) — Setup: great, if every attendee has or makes an account with that provider and the event can absorb per-seat billing; check each provider's current isolation model, quotas and pricing in their docs. Judging: you are relying on each team to stop pushing.
- Per-team microVM from a baked template — Setup: one Dockerfile, built the night before. Isolation: a guest kernel per team behind KVM, so root inside is root of a small machine that belongs only to them. Judging: memory-and-disk snapshots, restored into fresh machines for judges. Cost: you pay for every hour every machine runs, which is the honest trade.
Bake the toolchain so the wifi only carries keystrokes
The single most effective thing you can do for an event is resolve every dependency before anyone arrives. The template is an ordinary Dockerfile that the platform builds into a microVM image and snapshots: start from a Debian-family image (here node:22-bookworm), add Python and a browser IDE, clone the starter repo and install its dependencies, and pull in the dataset the sponsor challenge needs. You do not add an init or sshd; the build installs both if the image lacks them. Everything that would have been a 400 MB download at 9:05 is now part of the image. (The stock base template, with mise and pre-warmed Node 22, Python 3.12, Go and Bun, is a fine choice if you do not need anything baked in beyond the runtimes.)
# templates/hack-2026/Dockerfile
# Everything an attendee would otherwise download over venue wifi at 9:05.
FROM node:22-bookworm
RUN apt-get update && apt-get install -y --no-install-recommends \
python3 python3-pip python3-venv git ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# A browser IDE, so a laptop only needs a browser.
RUN curl -fsSL https://code-server.dev/install.sh | sh
# The event's starter, with its dependencies resolved now, not at 9:05.
RUN git clone --depth 1 https://github.com/your-event/starter /opt/starter \
&& cd /opt/starter && npm ci \
&& pip3 install --break-system-packages -r requirements.txt
# The sponsor dataset everyone will ask for.
RUN mkdir -p /opt/data \
&& curl -fsSL https://data.example.org/challenge.parquet -o /opt/data/challenge.parquet
Fetch big things inside the Dockerfile, as above, rather than COPYing them in: the uploaded build context is capped at 50 MB and 50 files, and --size-mb sets the root filesystem size, so leave headroom for what teams will install during the day. Pick the guest RAM deliberately with --memory-mb, because it is baked into the template snapshot and is what every team's machine will have; you cannot change it per team at create time. CPU is 8 burstable vCPUs on every template. 2 GiB is comfortable for a Node or Python web app with a dev server; a team that wants to fine-tune anything wants a different event. Then build the night before and create one throwaway sandbox from it, so the first create of the template — the one that cold-boots in about 3 s and bakes the snapshot that every later create restores — happens at 11 p.m. in your kitchen instead of at 9:00 in front of team one.
# The night before. Build once, then spawn one sandbox so the first
# create pays the ~3 s cold boot + auto-bake instead of team 1 at 9:00.
pandastack template build -f templates/hack-2026/Dockerfile --name hack-2026 \\
--memory-mb 2048 --size-mb 8192
python3 -c "from pandastack import Sandbox; s = Sandbox.create(template='hack-2026'); print(s.boot_mode, s.boot_ms); s.kill()"
The point that is easy to miss: once the environment lives in a sandbox, npm install during the event runs in a datacenter, on a datacenter's network. The venue wifi carries a browser tab — an IDE's websocket and a preview page — which is a few kilobytes a second per person instead of a few hundred megabytes per dependency change. That is the actual bandwidth fix. It is not a faster network; it is not using the venue network for the heavy bits at all.
Provisioning N teams
Provisioning is a loop. Read a list of team names, create a persistent sandbox per team tagged with the event and team in metadata, copy the starter into a working directory, start the IDE with a per-team password, and print a row for the sheet you hand out at registration. Creates run in a small thread pool; each one is a snapshot restore, so a few dozen teams provision in a handful of seconds plus the time to start code-server.
# provision.py -- one persistent sandbox per team, prints a sheet to hand out.
# usage: PANDASTACK_API_KEY=pds_... python3 provision.py teams.txt > sheet.csv
import csv, secrets, sys
from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox
EVENT = "hack-2026-11"
TEMPLATE = "hack-2026"
def provision(team: str) -> dict:
sbx = Sandbox.create(
template=TEMPLATE,
persistent=True, # exempt from the idle reaper: lunch is not abandonment
metadata={"event": EVENT, "team": team},
)
password = secrets.token_urlsafe(12)
sbx.exec("mkdir -p /work && cp -r /opt/starter/. /work/", check=True)
# Start the IDE detached, with its own password on top of the URL.
sbx.exec(
f"PASSWORD='{password}' setsid -f code-server --auth password "
"--bind-addr 0.0.0.0:8080 /work > /var/log/code-server.log 2>&1",
check=True,
)
return {
"team": team,
"sandbox_id": sbx.id,
"ide": sbx.preview_url(8080),
"ide_password": password,
"demo": sbx.preview_url(3000), # whatever they serve on :3000
}
def main(path: str) -> None:
teams = [t.strip() for t in open(path) if t.strip()]
out = csv.DictWriter(sys.stdout, ["team", "sandbox_id", "ide", "ide_password", "demo"])
out.writeheader()
with ThreadPoolExecutor(max_workers=8) as pool:
for row in pool.map(provision, teams):
out.writerow(row)
sys.stdout.flush()
if __name__ == "__main__":
main(sys.argv[1])
Two choices in that script deserve a defence. persistent=True exempts the sandbox from the idle reaper entirely, which is what you want at an event: a team that goes to lunch for ninety minutes has not abandoned its machine, and losing an afternoon's uncommitted work to an idle timer is the kind of thing people write blog posts about. The alternative — an ephemeral sandbox with a long ttl_seconds — is covered in the limits section below. The second is the metadata tags. Every later script (the watchdog, the freeze, the teardown) finds the event's machines by metadata["event"] rather than by a list of ids in a spreadsheet somebody might lose.
The UUID is the credential: what that means for sharing
sbx.preview_url(port) returns https://<port>-<sandbox-id>.<suffix>. That URL works without logging in, for the lifetime of the sandbox. There is no token to mint and no account for the attendee to create; the random sandbox UUID in the hostname is the bearer credential. That is precisely what makes it pleasant at an event — paste the demo link into the judging form, done — and it has consequences you should tell attendees out loud:
- Anyone with the link can reach that port. A demo URL in the event Discord is a demo URL for everyone in the Discord, which for the demo port is usually the point.
- The same is true of the IDE port. That is why the script starts code-server with its own password: the URL gets you to a login page, the password gets you a shell. Two factors that leak through different channels — a screenshot of a browser tab shows the URL, not the password.
- There is no "rotate the link". The UUID is the sandbox's identity, so a leaked URL stays valid until the sandbox is gone. If an IDE link leaks, change the password (
sbx.execa newPASSWORDand restart code-server); if a demo link leaks, it was going to be public at 4 p.m. anyway. - Do not hand attendees your API key so they can use the CLI's terminal. An API key is scoped to your workspace, not to one sandbox, so a team holding it can list, exec into and kill every other team's machine. PandaStack does not have per-sandbox scoped credentials today; the browser IDE behind its own password is the access path that does not require one.
Strangers with root, and the miner "for the demo"
An event is a multi-tenant system where the tenants are strangers, they all have root, and they have been encouraged since breakfast to try things. The isolation question is not hypothetical. It is whether one tenant's root can become anything more than root of their own machine.
With containers on a shared host, a team's root is a process on your kernel, filtered by namespaces, cgroups and seccomp — good mechanisms, all of them, and all of them enforced by the same kernel that a local privilege escalation targets. The history of that attack surface is a long one; Container Escape CVEs, by Class: What the Pattern Tells You walks through it. With a microVM, a team's root is root of a guest kernel that only their machine runs. To reach the host or another team, they need an escape through the virtual machine monitor and KVM, a far smaller surface designed for exactly this adversary. Nobody at your hackathon is burning one of those on a sticker prize.
The boundary is the floor, not the whole plan. The limits that do the rest of the work:
- Memory is fixed by the template. A team cannot allocate past the guest RAM you baked; a runaway process triggers the guest's own OOM killer inside their machine, not on your host.
- CPU is shared fairly under contention. Each guest gets cgroup CPU weight, so a miner on one team's machine burns that team's share of the cores, not everyone's. It still burns money — see the cost section — which is why you watch for it rather than merely tolerate it.
- Networking is per machine. Each sandbox has its own network namespace, so a team that port-scans finds itself, the internet, and nothing else on your side. It cannot see the venue subnet either, because it is not on it.
- Snapshots are evidence. When you find the miner,
sbx.pause()freezes the machine in place. The conversation with the team can then happen with the process list on screen rather than after they have deleted it.
Finding it is a five-line script. A miner is never idle, so no idle timer will catch it; load average will:
# watch.py -- run every few minutes during the event.
from pandastack import Sandbox
EVENT = "hack-2026-11"
for sbx in Sandbox.list():
if sbx.metadata.get("event") != EVENT:
continue
r = sbx.exec("cat /proc/loadavg; ps -eo pcpu,comm --sort=-pcpu | sed -n 2,4p", timeout_seconds=10)
load1 = float(r.stdout.split()[0])
if load1 > 6:
print(f"{sbx.metadata['team']}: load {load1}\n{r.stdout}")
# sbx.pause() # freezes it in place for the conversation, and keeps the evidence
Tune the threshold to the event. A team running a test suite or a build will spike briefly; a miner sits at full load for an hour. Two consecutive hits a few minutes apart is a reasonable trigger for a human to walk over. Note that exec counts as activity on the sandbox, which does not matter here because persistent sandboxes are not idle-reaped anyway.
Persistent, or ephemeral with an idle TTL?
For a one-day event I use persistent sandboxes and tear them down by script. For a drop-in workshop where people wander off and never come back, an idle TTL does the teardown for you. The trade-off:
| persistent=True | ttl_seconds=N (ephemeral) | |
|---|---|---|
| Idle reaper | Never reaps it | Deletes it after N seconds with no activity |
| A two-hour lunch | Survives | Survives only if N is longer than lunch |
| The attendee who never returns | Runs, and bills, until you kill it | Reaped automatically N seconds after their last activity |
| A miner | Runs until you notice | Also runs until you notice: it is never idle |
| Snapshots on reap | Not applicable | An idle reap keeps the snapshots; an explicit kill() deletes them |
| Best for | Hackathons with a scripted teardown | Self-paced workshops and drop-in labs |
Remember that ttl_seconds is an idle clock, not a wall-clock lifetime: activity resets it, which is why a busy machine never hits it. If you need a hard end time, the teardown script is the hard end time. How to control sandbox lifetime: TTL, idle, and cleanup goes deeper on the lifecycle knobs, and you can change a running sandbox's TTL or persistence later with sbx.set_ttl() and sbx.set_persistent().
Freeze at the deadline, judge from copies
This is the part that makes organisers happiest, because it replaces trust with mechanism. At the deadline, snapshot every team's sandbox and pause it. A PandaStack snapshot is the machine's memory and disk together, captured with the guest briefly paused, so what you have is not the repository at 4:00 — it is the running dev server, the seeded database, the half-applied migration and the browser-ready app, exactly as they were.
# freeze.py -- run at the submission deadline.
import json, time
from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox
EVENT = "hack-2026-11"
def freeze(sbx: Sandbox) -> dict:
snap_id = sbx.snapshot() # memory + disk, captured with the guest paused
sbx.pause() # no more commits after the bell
return {"team": sbx.metadata["team"], "sandbox_id": sbx.id,
"snapshot_id": snap_id, "frozen_at": time.time()}
teams = [s for s in Sandbox.list() if s.metadata.get("event") == EVENT]
with ThreadPoolExecutor(max_workers=8) as pool:
frozen = list(pool.map(freeze, teams))
json.dump(frozen, open("frozen.json", "w"), indent=2)
# Judging: each judge gets a fresh machine restored from the frozen state,
# so a judge clicking around cannot change what the next judge sees.
def judge_copy(entry: dict, judge: str) -> str:
j = Sandbox.create(from_snapshot=entry["snapshot_id"],
metadata={"event": EVENT, "team": entry["team"], "judge": judge})
return j.preview_url(3000)
Then each judge gets a fresh sandbox created from_snapshot. It restores with the team's processes already running, under a new sandbox id and so a new preview URL, and nothing a judge does — clicking delete, submitting the form with an emoji in it, discovering the admin panel — touches the frozen original or what the next judge sees. If a demo crashes in front of judge three, judge four restores the same 4:00 state again. Teams can still present from their own paused machine if you resume it for their slot, but the score is given against the frozen copy.
A few honest details about restored machines:
- Snapshots are not instant on a multi-GiB guest, because the full memory image is written out; the SDK uses a generous timeout for that reason. Run the freeze in parallel, as above, and start it at the bell rather than a minute after.
- The guest clock is resynchronised on restore, so the judge's copy has the correct time, not 4:00. An app that cached a short-lived OAuth token at 3:55 may need its tokens refreshed — the same thing would happen if the team closed their laptop lid for an hour.
- Anything the app reaches outside the sandbox is not frozen. A demo that calls a sponsor's API, or a managed database outside the machine, sees that service's state at judging time. Data inside the sandbox is frozen; data elsewhere is whatever it is now.
snapshot()is the right tool here, notfork(). Afork()copies the disk and cold-boots the child, so the dev server is not running in it; the snapshot restore is what brings the processes back.
sbx.kill() on a team's sandbox also deletes the snapshots taken from it, which are your judging copies' source. Kill team machines last: after scoring, after winners have pushed their code somewhere they own, after the photo. An idle-TTL reap of an ephemeral sandbox does not cascade, but persistent machines are never idle-reaped, so for a hackathon the teardown script is the only thing that ends them — and the only thing that ends the bill.Teardown, and what it costs
Teardown is the provisioning loop in reverse, keyed on the same metadata tag: judge copies first, team machines last.
# teardown.py -- only after results are announced and winners have their code.
from pandastack import Sandbox
EVENT = "hack-2026-11"
# Judge copies first, then the team machines. kill() on a team sandbox also
# deletes the snapshots taken from it, so this is the last step, not the first.
mine = [s for s in Sandbox.list() if s.metadata.get("event") == EVENT]
for sbx in sorted(mine, key=lambda s: "judge" not in s.metadata):
print("killing", sbx.metadata.get("team"), sbx.metadata.get("judge", "team"), sbx.id)
sbx.kill()
Now the honest part. A persistent sandbox costs money for every hour it is running, whether its team is coding, at lunch, or asleep under the table during an overnight event. Memory is billed for what the machine holds — the guest RAM you baked into the template — and CPU by the CPU-seconds actually burned, so a miner is visible on the bill as well as in your watchdog. The arithmetic for an event is teams times hours times the template's GiB, plus whatever CPU they use; check pricing for the current rates and do that multiplication before you commit to a template size. A 2 GiB template for fifty teams over a ten-hour event is a known, boring number. Leaving fifty machines running for a week because nobody ran teardown.py is a less boring one.
Three things keep it boring. Size the template to the event rather than to the most ambitious team. Pause or hibernate machines overnight on a multi-day event if teams are not working through the night (a paused machine still holds its memory; check pricing for how paused and hibernated sandboxes are billed). And put the teardown on the schedule like the pizza order, with a named owner.
When this is overkill
A two-hour intro-to-Python workshop with twenty people and one notebook does not need a machine per attendee. A browser-only runtime such as Pyodide or JupyterLite runs entirely on the laptop, needs no backend, costs nothing, and is safe because there is no server to attack; the trade is that it cannot run a real web server, install arbitrary native packages, or give anyone a shell. The education buyer's guide, The Best Code Execution Sandboxes for Education Platforms in 2026, lays out that ladder. If what you need is grading — running untrusted submissions against tests you keep secret — that is a different system with different limits, described in Building an autograder that runs student code safely in microVMs.
The per-team microVM earns its keep when the event looks like real development: multiple languages, dev servers, databases, people with root who are strangers to you and to each other, a demo that has to run on a judge's screen, and a deadline that should mean something. That describes most hackathons I have been to, and none of the forty broken Python installs.