Running CTF challenges and cyber ranges on microVMs

Most infrastructure is built on the assumption that users are trying to use it correctly. A CTF platform, a university offensive-security course, a red-team range, a vendor certification lab — these all start from the opposite assumption. You have invited a few hundred strangers to attack your infrastructure and told them there are points in it. The entire pedagogical value of the exercise is that participants get root, escalate privileges, break out of things, and generally do everything your production runbook exists to prevent. "Please don't touch that" is not a security control when touching that is worth 500 points.

I'm Ajay — I build PandaStack, which runs Firecracker microVMs as a service, so I have an obvious bias and I'll try to be specific about where it applies and where it doesn't. This post is infrastructure guidance for the people running the range, not the people playing in it: how to shape per-participant instances, which challenge categories are structurally impossible on a shared kernel, how to make "reset my box" a button rather than a support ticket, how to give a challenge a real internal network that isn't yours, and how to stop your training range from becoming a free attack platform pointed at the rest of the internet.

The threat model is inverted, and that changes the architecture

In a normal multi-tenant SaaS, exploitation is an edge case you defend against. In a range, exploitation is the product. Every participant is expected to obtain root inside their environment on day one, because that's what the lesson is. That single fact invalidates most of the isolation strategies you'd otherwise reach for.

Unix user separation? Gone the moment the challenge is solved — and "solved" is the goal. Filesystem permissions? Same. Application-level tenancy checks? They live inside the thing being attacked. Anything you enforce at or above the operating-system level in the participant's environment is not a boundary, it's a puzzle piece. The only boundaries that survive are the ones below the participant's kernel or outside their machine entirely: the hypervisor, the network, and the egress policy.

One instance per participant, not one box per challenge

The classic starter architecture is a shared jump box or a single hosted instance of each challenge that everyone connects to. It is cheap, it is simple, and it fails in the most demoralising way possible: the first player to solve a challenge is standing inside the same machine as everyone still working on it. From there they can read other people's shells, watch other people's exploit attempts land, corrupt the state everyone else is relying on, or — the classic — get the flag, then break the service so nobody else can.

That last one isn't even always malice. Pwn challenges are about corrupting memory. A participant probing an off-by-one will crash the service dozens of times before they land it. On a shared instance, every one of those crashes is an outage for everyone else, and your organiser Slack fills with "is challenge 4 down again?" It is down again. It will be down again in ninety seconds.

• Flag confidentiality — on a shared instance, the flag is a file on a machine that many mutually untrusted people have code execution on. Per-participant instances make flag theft between players a network problem instead of a filesystem problem.
• Availability — one participant's crash loop, fork bomb, or 12 GB allocation shouldn't be everyone else's outage. Isolate the failure domain to the person who caused it.
• Attribution — when something does go wrong, per-participant instances mean your logs already say who. Shared boxes turn incident response into forensics.
• Honest scoring — if a solve can be observed, copied, or blocked by a neighbour, the scoreboard is measuring proximity rather than skill.
• Clean resets — you can hand a trashed instance back to a known-good state without coordinating a maintenance window with everyone else on it.

The obvious objection is cost, and it's a real one — until instances only exist while someone is actually playing. That's the part the boot time decides, and it's why this whole argument keeps landing on microVMs rather than on "just give everyone an EC2 instance."

Pwn and kernel categories: a challenge that needs a kernel you're allowed to break

Containers are the default answer for hosting challenges, and for a large chunk of a typical CTF they are a perfectly reasonable one. Web challenges, crypto challenges, forensics, most misc and reversing categories — these mostly need a process and a port, and a well-locked-down container with a read-only rootfs, dropped capabilities, a seccomp profile, and a user namespace is a sensible fit. Docker-based challenge frameworks are popular for exactly this reason, and Google's kCTF project exists specifically because hosting exploitation challenges on Kubernetes is hard enough to warrant purpose-built tooling. If you're going that route, read their threat model carefully and verify the current guidance against their own docs rather than any blog post, including this one.

But there's a category boundary you cannot engineer your way past. Containers share the host kernel. Some of the most valuable things you can teach share the host kernel too:

• Kernel exploitation and driver bugs — the entire exercise is "here is a deliberately vulnerable kernel module; achieve privileged code execution." On a shared kernel there is no such thing as a scoped version of that.
• Container escape as a taught topic — you cannot honestly teach escaping a container while relying on that container as the boundary that keeps your platform intact.
• Rootkits, LKM loading, syscall-level tampering — a participant needs to load code into a kernel and watch what happens to it.
• Anything with a deliberately vulnerable kernel version — training on a historical bug means shipping the kernel that has it, which you cannot do if everyone shares one.
• Malware detonation and defence-evasion labs — where the whole point is to let something hostile run to completion and observe it, ideally without it observing you back.

A microVM changes the shape of this. Each instance boots its own guest kernel under a hardware-virtualized boundary, so the kernel the participant is attacking is theirs. When they win — and you want them to win, that's the lesson — they get root on a kernel whose blast radius is one throwaway VM. The next boundary out is the VMM, which on Firecracker is a deliberately tiny device model (virtio-net, virtio-blk, virtio-vsock, serial, a keyboard controller good for exactly one thing) plus a jailer and a seccomp filter on the VMM process itself.

One caveat worth checking before you build a curriculum around it: shipping a specific vulnerable kernel version per challenge depends on whether your platform lets you supply guest kernels per template, and many managed sandbox providers (PandaStack included, today) pin one guest kernel across templates. If your course requires CVE-specific kernel builds, verify that capability explicitly before committing — it's the difference between "kernel exploitation category" and "kernel exploitation category, with the one kernel we have."

The reset button: snapshot-restore instead of a support ticket

Participants will destroy their environments. Not occasionally — routinely, and usually on purpose, because "what happens if I do this" is the correct instinct to be cultivating. Someone will `rm -rf /`, someone will fill the disk, someone will fork-bomb it, someone will land a heap-corruption primitive that leaves the box in a state no reboot fixes. In a course setting this is the single largest source of interruptions: a student's box is unusable, the student is stuck, and now an instructor is triaging infrastructure in the middle of a lab.

The fix is making reset so cheap that it's the first thing anyone tries. If creating a fresh instance means cold-booting a VM and re-running provisioning, reset takes minutes and people avoid it. If creation is a snapshot restore, it's a click. On PandaStack, creating a sandbox restores a baked Firecracker snapshot rather than booting: p50 179ms, p99 around 203ms, with the restore step itself roughly 49ms. The first-ever spawn of a template still does a real cold boot at around 3 seconds, and after that snapshot is baked, every instance of that challenge comes off it.

The design consequence is bigger than the latency number. When restore is the normal creation path, "reset" and "create" are the same operation, so there is no separate cleanup code path to get wrong. You don't repair a trashed instance — you delete it and stamp a new one from the same snapshot. Whatever the participant did to the old one goes away with it, including any persistence they cleverly established, which is a nice property when the challenge was about establishing persistence.

import hashlib, hmac, os

from pandastack import Sandbox

# Per-participant flags. Deriving the flag from a server-side secret means a
# leaked flag identifies whose instance it came from, which turns flag-sharing
# from an unsolvable social problem into a lookup.
RANGE_SECRET = os.environ["RANGE_FLAG_SECRET"].encode()

def flag_for(team_id: str, challenge_id: str) -> str:
    digest = hmac.new(
        RANGE_SECRET, f"{challenge_id}:{team_id}".encode(), hashlib.sha256
    ).hexdigest()[:32]
    return f"range{{{digest}}}"


def provision(team_id: str, challenge_id: str) -> Sandbox:
    """One instance per (team, challenge). Never shared, never reused."""
    sbx = Sandbox.create(
        template="base",
        # The lab session ends, the instance ends. This is the backstop for
        # every case where your control plane forgets to clean up: a crashed
        # worker, a closed browser tab, a student who went to lunch in 2019.
        ttl_seconds=3600,
        metadata={
            "team": team_id,
            "challenge": challenge_id,
            "event": "autumn-range",
        },
    )

    # Flag goes in AFTER the snapshot was baked, so the flag is never part of
    # a shared template image. One participant reading their own flag file
    # teaches them nothing about anyone else's.
    sbx.filesystem.write("/srv/flag.txt", flag_for(team_id, challenge_id))
    sbx.filesystem.write("/srv/challenge/README", "good luck\n")

    setup = sbx.exec(
        "chown root:root /srv/flag.txt && chmod 0400 /srv/flag.txt && "
        "systemctl start challenge.service",
        timeout_seconds=30,
    )
    if setup.exit_code != 0:
        # Never hand out a half-provisioned instance -- a participant cannot
        # tell "broken challenge" from "hard challenge", and they will spend
        # four hours proving it to you.
        sbx.kill()
        raise RuntimeError(f"provision failed: {setup.stderr}")

    return sbx


def reset(team_id: str, challenge_id: str, old: Sandbox) -> Sandbox:
    """There is no repair path. Destroy and re-stamp from the snapshot."""
    old.kill()
    return provision(team_id, challenge_id)


# Grading and scripted checks want the opposite lifecycle: a disposable VM
# that exists for the length of one command and then doesn't.
with Sandbox.create(template="base", ttl_seconds=300) as checker:
    checker.filesystem.write("/tmp/solve_check.sh", submitted_solution)
    result = checker.exec("sh /tmp/solve_check.sh", timeout_seconds=30)
    passed = result.exit_code == 0
# checker is destroyed on block exit, whatever the submission did to it

Two details in there are load-bearing. First, the flag is written after creation, not baked into the template — a flag inside a shared snapshot is a flag that every instance of that challenge contains, which makes "solve it once, share the file" trivially scalable. Second, the TTL. Ranges leak instances constantly, because sessions end in every way except the tidy one, and a server-side TTL is the only cleanup mechanism that doesn't depend on your own control plane behaving.

"Scan the internal network" needs an internal network that isn't yours

Half of what makes range training realistic is lateral movement: land on a web host, discover a database on an adjacent subnet, pivot, escalate to a domain controller. Which means the exercise involves a participant scanning a network and moving through it — and if that network is your production VPC, congratulations, you have built an insider-threat simulator with real stakes and no consent form.

The right primitive is a network per participant, not a firewall rule per participant. On PandaStack each sandbox gets its own Linux network namespace with its own veth pair, tap device, and NAT rules, allocated from a pool of 16,384 pre-allocated /30 subnets per agent. Pre-allocation is a performance decision (creating a namespace cold costs far more than patching a MAC on a warm one), but the isolation property is what matters here: instances aren't sharing a bridge, so a participant scanning their own segment finds their own machine.

For a multi-machine scenario — the pivot chain — you compose that deliberately: several instances for one team, connected to each other on an overlay you create for that team, with nothing joining them to any other team's overlay or to anything of yours. That's real work, and it's worth being clear that no sandbox API hands it to you for free. What the per-sandbox namespace gives you is the default: instances are isolated unless you connect them, rather than connected unless you separate them. Building a lab network on top of "isolated by default" is design work. Building one on top of "one flat bridge everyone shares" is a whack-a-mole exercise you will lose during the event, in front of people whose hobby is winning it.

Any range where the answer to "can they reach X?" is a firewall rule will eventually meet a participant who is better at firewalls than you are. Make the answer be "there is no route," not "there is a route and a rule."

Egress: don't hand three hundred strangers a free attack platform

This is the part that turns a bad day into a legal one. An instance with unrestricted outbound internet, handed to a few hundred people who are actively practising offensive tooling, is an open proxy with your organisation's name on the WHOIS record. Most participants won't misuse it. You don't need most. You need one person to point a scanner at a third party, or to use your range as a hop for something they shouldn't, and now you are explaining your event to an abuse desk — or to your cloud provider, who will act first and read your explanation later.

Default-deny egress, with a narrow allowlist, is the only posture that survives contact. And it has to be enforced outside the participant's machine, because inside it they are root and any in-guest control is a challenge rather than a control.

# Egress policy shape, enforced on the HOST side of each instance's veth --
# outside the guest, where a participant with root cannot reach it. Adapt to
# your own platform's enforcement point (nftables, VPC firewall, cloud egress
# rules); the ordering and the default are what matter.

nft add table inet range
nft add chain inet range forward '{ type filter hook forward priority 0; policy drop; }'

# 1. Established flows first, so replies to allowed traffic come back.
nft add rule inet range forward ct state established,related accept

# 2. Explicit allowlist, and keep it embarrassingly short. Package mirrors and
#    an update host, if the exercise genuinely needs them. Everything a
#    challenge depends on should be baked into the template instead -- an
#    "apt-get during the event" dependency is also an outage waiting for the
#    mirror to have a bad afternoon.
nft add rule inet range forward ip daddr @range_allowlist accept

# 3. RFC1918 and link-local are the interesting denials: without these, the
#    lateral-movement lesson escapes the lab and finds your real network.
#    Cloud metadata endpoints belong here too -- 169.254.169.254 is the first
#    thing anyone curls on a new box, and it is a credential vending machine.
nft add rule inet range forward ip daddr { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16 } drop

# 4. Rate-limit what remains, so a scanner pointed outward is slow and loud
#    rather than fast and anonymous. Log drops with the instance id: when an
#    abuse report arrives, you want to already know which instance it was.
nft add rule inet range forward limit rate over 200/second drop
nft add rule inet range forward log prefix "range-egress-drop " drop

# Verify from the participant's side of the boundary, not from your laptop.
# A policy you have not tested from inside the guest is a policy you are
# describing, not enforcing.

Two additions worth the effort. Rate-limit and log outbound connections per instance, so that if an abuse report ever lands you can answer "which instance, which participant, what time" in minutes rather than reconstructing it from cloud flow logs at 2am. And put the metadata endpoint in the deny list explicitly. On a cloud host, `169.254.169.254` is the shortest path from "I have a shell in your lab" to "I have your instance role," and it is the very first thing a competent participant checks, mostly out of professional curiosity.

Teardown, flag rotation, and the state you forgot you kept

Ranges accumulate. Sessions end abruptly, browsers close, cohorts finish, and the instances stay up because the only thing that would have removed them was a client-side cleanup call that never fired. Every one of those orphans is a machine a participant has root on, with your flags in it, running unattended after anyone stopped watching. Server-side TTLs are non-negotiable for this reason: your control plane should be the backstop, not the mechanism.

1. Give every instance a TTL at creation, sized to the session plus slack. If a participant needs longer, they can extend it — an explicit extension is a much better default than an implicit forever.
2. Derive flags per participant from a server-side secret (the HMAC pattern above) so that a leaked flag is attributable, and a flag pulled from one instance is worthless on another.
3. Rotate the secret between cohorts. Reusing an event's flags for next term's course means last term's writeups are this term's scoreboard.
4. Never bake flags into the template snapshot. The snapshot is shared by every instance of the challenge; the flag should exist only in one participant's running VM.
5. Tag instances with team, challenge, and event at creation, so cleanup and incident response are both a query rather than an archaeology project.
6. Sweep for orphans on a schedule anyway, and alert if the count is non-zero. A leak you never look for is a leak that grows until it becomes a bill or a breach.
7. After the event, delete instances and snapshots deliberately. A stale challenge snapshot from three events ago is an unpatched machine nobody owns.

import { Sandbox } from "@pandastack/sdk";

// Challenge-provisioning handler for the range's web app. The interesting
// logic is not the create call -- it's the two guards around it.
type Slot = { sandboxId: string; expiresAt: number };
const slots = new Map<string, Slot>(); // key: `${teamId}:${challengeId}`

export async function POST(req: Request): Promise<Response> {
  const { teamId, challengeId, action } = await req.json();
  const key = `${teamId}:${challengeId}`;

  // GUARD 1: one live instance per team per challenge. Without this, a held
  // refresh key is a resource-exhaustion primitive, and someone WILL find
  // that out before they find the intended solution.
  const existing = slots.get(key);
  if (existing && action !== "reset") {
    return Response.json({ status: "already-running", ...existing });
  }

  // GUARD 2: a per-team rate limit on creation. Reset is meant to be cheap
  // for the participant, not free for you.
  if (!(await allowCreate(teamId))) {
    return Response.json({ error: "reset rate limit" }, { status: 429 });
  }

  if (existing) {
    // Reset is delete-and-restamp. There is no in-place repair, because
    // "repair" means trusting the state a participant just spent an hour
    // deliberately corrupting.
    await new Sandbox(existing.sandboxId).kill().catch(() => {});
    slots.delete(key);
  }

  const sb = await Sandbox.create({
    template: "base",
    ttlSeconds: 3600, // server-side backstop; never trust client cleanup
    metadata: { team: teamId, challenge: challengeId, event: "autumn-range" },
  });

  // Flag is injected per instance, after creation -- never baked into the
  // shared snapshot every other team is also running.
  await sb.filesystem.write("/srv/flag.txt", flagFor(teamId, challengeId));
  const boot = await sb.exec("systemctl start challenge.service", {
    timeoutSeconds: 30,
  });

  if (boot.exitCode !== 0) {
    await sb.kill().catch(() => {});
    return Response.json({ error: "provision failed" }, { status: 500 });
  }

  const slot = { sandboxId: sb.id, expiresAt: Date.now() + 3_600_000 };
  slots.set(key, slot);
  return Response.json({ status: "running", ...slot });
}

Shared box vs container vs microVM vs full VM, per participant

Four architectures, honestly compared. The PandaStack latency figures are our measured numbers on our platform; treat everything said about other systems as a qualitative shape to verify against their own documentation, because defaults and hardening options change.

• Shared jump box for everyone — Pros: trivially cheap, one machine to maintain, works fine for a small trusted classroom doing web and crypto categories. Cons: flags are readable by anyone who solves first, one participant's crash is everyone's outage, no honest scoring, and no per-person reset. Unsuitable for anything where participants get root, which is most of the interesting material.
• Container per participant — Pros: fast to start, dense, cheap, and the ecosystem for CTF-style deployment is mature (Docker-based challenge frameworks and kCTF exist precisely to make this tractable). Good for web, crypto, forensics, and most reversing. Cons: shared host kernel. Kernel exploitation, LKM/rootkit, container-escape, and vulnerable-kernel-version categories are structurally out of scope, and hardening becomes an ongoing arms race against the exact skill your event is teaching.
• MicroVM per participant — Pros: own guest kernel, so exploitation categories that need a breakable kernel become hostable; own network namespace by default; snapshot-restore makes create and reset the same sub-second operation (PandaStack: p50 179ms create, ~49ms restore step); instances only exist while someone is playing. Cons: it's still a VMM you're trusting, per-instance memory is real memory, and whether you can ship a specific vulnerable kernel per challenge depends on the platform — verify before designing a curriculum around it.
• Full VM per participant — Pros: the most familiar model, maximum control over kernel and image, and every commercial hypervisor's tooling applies. Cons: boot and provisioning measured in minutes, so reset is a coffee break and instances get left running because restarting them hurts; per-VM overhead makes "one per participant per challenge" expensive fast; and pre-provisioning a pool for an event means paying for peak all week.

The pattern across the four: containers optimise for density and lose the kernel boundary, full VMs keep the boundary and lose the speed, and microVMs are an attempt to keep the boundary while getting the speed back through snapshot-restore instead of booting. For a range specifically, the kernel boundary is not a nice-to-have — it's the difference between which categories you can run at all.

Cost: instances that exist only while someone is playing

The economics of a range are brutally spiky. A CTF is 48 hours of peak followed by weeks of nothing. A university course is three hours on Tuesday afternoons. A certification lab is whenever someone books a slot. If your unit of compute takes minutes to provision, you cannot follow that curve — you pre-provision a pool sized for peak and pay for it while it's idle, or you make people wait at exactly the moment they're most impatient, which is the start of the event.

Sub-second creation removes the choice. There is no warm pool because there's nothing to warm: creation restores a baked snapshot on demand, so an instance can be created when a participant clicks "start" and destroyed when they click "stop" or when the TTL fires. Idle participants cost nothing because idle participants don't have instances. A challenge nobody has attempted this term has exactly one artifact on disk — its snapshot — and zero running machines.

Two more levers worth knowing. Copy-on-write forking clones a warm, already-set-up machine rather than provisioning a fresh one: on PandaStack a same-host fork lands in 400–750ms (1.2–3.5s cross-host), which is a good fit for scenarios where every participant needs an identical mid-scenario state — a machine already compromised to a given stage, say, for a defensive or forensics exercise. And if the scenario needs a real database rather than a fixture file, a managed Postgres instance takes 30–90s to create, so provision it alongside the lab rather than at the moment the participant hits the login page.

The short version

If you're building or auditing range infrastructure, the questions worth asking are these, roughly in order of how badly they hurt when the answer is wrong.

1. Does each participant get their own instance for each challenge, with a flag that's theirs alone and derived from a server-side secret?
2. Do any of your challenge categories require the participant to attack a kernel? If so, that kernel cannot be shared with anyone, including you.
3. Is reset a button that completes in seconds, or a support ticket? Whichever it is, that's how often participants will experiment.
4. Is egress default-deny, enforced outside the guest, with the cloud metadata endpoint and RFC1918 space explicitly blocked, logged, and rate-limited?
5. Does a "scan the internal network" challenge scan a network you built for it, or one you'd rather it didn't find?
6. Does every instance have a server-side TTL, so that a closed laptop is cleaned up by your platform rather than by your monthly invoice?
7. After the event: can you enumerate and delete everything you created, including snapshots, from tags you set at creation time?

None of this makes a range unbreakable, and you shouldn't build as though it did. What it does is move the boundary to somewhere the exercise isn't allowed to reach: below the participant's kernel, outside their network, and enforced by something they don't have root on. Everything above that line is fair game — which is the whole point, because that's the part you're trying to teach.