Per-Room WebRTC Media Servers in MicroVMs

A selective forwarding unit sounds like the easiest server you'll ever write. Participants publish RTP streams, the SFU forwards them to the other participants, and unlike an MCU it doesn't even have to decode anything. Then you actually run one. It decrypts and re-encrypts every SRTP packet, maintains jitter buffers, answers RTCP feedback, runs bandwidth estimation per subscriber, switches simulcast layers when someone's Wi-Fi wobbles, and — the moment a product manager says the word "recording" — starts muxing and writing media to disk while all of the above continues on a deadline. mediasoup, Janus, LiveKit-class servers: different codebases, same physics.

I'm Ajay, I built PandaStack. This post is about a specific deployment shape: one Firecracker microVM per room (or per tenant), created when the room opens and destroyed when the last person leaves. It's a genuinely good fit for real-time media — and it comes with a networking tax that most posts about microVM isolation get to ignore. I'm going to be honest about that tax, because if you skip it you will ship a product where every call spins on "connecting…" forever and nobody can tell you why.

One hot room ruins fifty calls

Real-time media is not a throughput workload, it's a deadline workload. A packet that misses its playout window isn't slow, it's garbage — the jitter buffer throws it away and the participant hears a chirp or sees a smear. That changes what "load" means. A build farm under pressure gets slower; a media server under pressure gets worse, permanently, for the seconds it was busy. You can't make it up later.

Now put every room on one process on one box. The failure modes stack up fast. A four-person standup that grows into a forty-person all-hands turns forwarding fan-out from trivial into the dominant cost, because each publisher's stream goes out to everyone else. Someone flips on cloud recording and you've added encoding and disk I/O to a host that was already pacing packets. A participant joins from a browser that needs a codec you don't have natively and something, somewhere, starts transcoding. Each of those is a legitimate feature. Collectively they mean one room's Tuesday can degrade every other room's Tuesday, and the affected customers have no idea they're sharing anything.

Then there's the blast radius of an actual crash. SFUs are native-code-heavy and they parse hostile input for a living — RTP headers, RTCP compound packets, DTLS handshakes, all arriving from browsers you don't control. A worker that segfaults doesn't drop one call, it drops every call it was hosting, and your incident channel fills with variations of "is anyone else frozen?" The people in the room that caused it and the people in forty unrelated rooms get exactly the same experience: silence, then a reconnect storm that lands on whatever capacity is left.

What a VM per room actually buys

• Hard resource ceilings. The room's SFU gets the vCPU and RAM the guest was booted with, full stop. A room that turns on recording and transcoding hits its own wall instead of eating into the headroom other rooms needed to keep their buffers fed.
• Fate isolation. A segfault, an OOM kill, a wedged worker, or a bad SFU build takes down one room. Its participants reconnect into a fresh VM; nobody else notices. That's the difference between a support ticket and an incident.
• Per-tenant egress accounting at the boundary. In an SFU, bandwidth is the bill — egress scales with subscribers, not publishers. With a network namespace per room you can meter and cap bytes at the VM boundary instead of trusting application counters that stop being emitted the moment the process dies.
• Independent versioning. Upgrading a shared media server means draining live calls off a box. When each room is its own VM, new rooms start on the new build and old rooms retire naturally when their calls end. Rollback is "stop creating rooms on that image."
• Clean teardown. Killing the VM reclaims transports, buffers, half-written recordings, stray ffmpeg children, and anything else the room leaked. There is no next room on this machine, so there's nothing to leak into.

The egress point deserves more than a bullet. For most SFU businesses, media bandwidth is the single largest variable cost, and it is wildly unevenly distributed across tenants — one customer running large webinars can out-consume a thousand customers doing 1:1 calls. If your only measurement is application-level counters aggregated across a shared process, you are estimating your own cost of goods sold. Per-room network namespaces turn that into a measured number you can attribute, cap, and bill.

The networking is the hard part (and I'm not going to pretend otherwise)

Most workloads you'd put in a microVM talk outbound over TCP and are perfectly happy behind NAT. Media servers are the opposite: they need to be reachable from the public internet on a wide range of UDP ports, and they have to tell clients — accurately — where to find them. Adding a per-VM network namespace puts one more translation hop between the SFU and reality. That hop is where per-room isolation goes wrong.

UDP port ranges and the extra hop

An SFU wants a contiguous UDP port range to hand out to transports. Inside a per-room guest that range has to be reachable from outside, which leaves you two workable designs. The clean one is a public IP per room VM — no port games, no mapping, and if you can afford the addresses it is by far the least surprising thing to operate. The cheaper one is a shared public IP with a disjoint slice of the UDP port space assigned to each room and forwarded to that room's guest.

If you take the shared-IP route, the mapping must be 1:1 — external port equals internal port. That isn't a style preference. The SFU advertises the ports it bound; if the outside world sees different ones, every ICE candidate it publishes is a lie and connectivity checks fail. Address space itself is rarely the constraint: a PandaStack agent pre-allocates 16,384 /30 subnets, so private addressing for rooms is not what runs out. Public UDP ports are what runs out. Divide the usable port space on one public IP by however many ports you give each room, and that quotient is your rooms-per-IP ceiling — plan it deliberately rather than discovering it during a launch.

// SFU config inside a per-room microVM (mediasoup-flavoured; the same idea
// applies to Janus and LiveKit-class servers under different key names).
//
// The guest binds a PRIVATE address behind the VM's netns. If it advertises
// that address, every client hangs on "connecting" forever, because nothing
// on the internet can route to 10.200.x.x.

const PUBLIC_IP = process.env.ROOM_PUBLIC_IP;      // injected at room create
const RTC_MIN = Number(process.env.ROOM_RTC_MIN);  // this room's port slice
const RTC_MAX = Number(process.env.ROOM_RTC_MAX);

const worker = await mediasoup.createWorker({
  rtcMinPort: RTC_MIN,
  rtcMaxPort: RTC_MAX,
});

const router = await worker.createRouter({ mediaCodecs });

const transport = await router.createWebRtcTransport({
  listenInfos: [
    {
      protocol: "udp",
      ip: "0.0.0.0",               // bind everything inside the guest
      announcedAddress: PUBLIC_IP, // ...but publish the routable address
    },
    {
      protocol: "tcp",             // fallback for UDP-hostile networks
      ip: "0.0.0.0",
      announcedAddress: PUBLIC_IP,
    },
  ],
  enableUdp: true,
  enableTcp: true,
  preferUdp: true,
});

# Host side: give this room's guest a disjoint slice of the public UDP port
# space, mapped 1:1. Do NOT remap ports -- the SFU already told clients
# which ports it bound, and ICE has no sense of humour about it.
iptables -t nat -A PREROUTING -d "$PUBLIC_IP" -p udp --dport "$RTC_MIN:$RTC_MAX" -j DNAT --to-destination "$GUEST_IP"

# Return path leaves from the same public address the client connected to,
# so the source of the media matches the candidate it was paired with.
iptables -t nat -A POSTROUTING -s "$GUEST_IP" -j SNAT --to-source "$PUBLIC_IP"

# Sanity check from OUTSIDE the host, not from the host itself. Half of all
# "it works on my machine" WebRTC bugs are someone testing from inside NAT.
nc -zvu "$PUBLIC_IP" "$RTC_MIN"

ICE candidates have to tell the truth

This is the failure that costs teams the most time, because it is completely silent. The SFU comes up healthy, signaling works, the room joins, the participant list populates — and then no media flows, forever. Everything logs green. What happened is that the server offered candidates pointing at an address only it can reach, ICE ran its checks, every pair failed, and WebRTC did what WebRTC does: nothing, quietly.

So the announced address is a first-class piece of per-room configuration, injected at create time along with the port slice, and it must be verified from outside your network. Don't have the guest discover its own address via STUN — from in there, the answer is whatever your NAT says, which is the thing you were trying to learn. And accept that per-room isolation doesn't remove the need for a TURN tier: some fraction of users sit behind firewalls that permit nothing but TCP 443, and they will relay. TURN is shared infrastructure that lives outside the room VMs, and you still have to run it.

Jitter matters more than throughput

The tempting way to size media hosts is by aggregate bandwidth. The correct way is by scheduling determinism. What ruins a call isn't a host running near its bandwidth capacity — it's the vCPU that didn't get scheduled for a few milliseconds while a packet sat in a queue. That has three consequences worth internalising. Don't oversubscribe vCPU on media hosts the way you would for batch work; a VM ceiling caps what a room can take, it does not conjure cores that aren't there. Care about CPU pinning and steal time, because both show up as jitter rather than as a number on a throughput dashboard. And measure the netns/veth hop's cost on your own hardware with your own traffic instead of trusting anyone's blog post, including this one.

Rooms are ephemeral and bursty, which is the good news

Everything above is the cost side. Here's what makes it worth paying: the lifecycle is a perfect match. Rooms are created, used for a bounded stretch, and abandoned. They're also brutally spiky in a way that follows human calendars — a few hundred rooms open in the same ninety seconds at the top of the hour, then near-nothing until the next one. Provisioning a warm pool for a workload shaped like that means paying for peak all day and still being caught short when a big customer schedules an all-hands.

Snapshot restore gives you the other option: create on room-open. Every create restores a baked memory image rather than cold-booting, which puts the create fast path at p50 179ms and p99 203ms, with the restore step itself around 49ms. A genuine cold boot (roughly 3s) only happens the first time a template has no snapshot yet. Be precise about what that buys, though — the VM being up is not the room being ready. The SFU still initialises transports and the client still completes signaling, ICE, and DTLS. What changes is that machine provisioning stops being the long pole and becomes noise next to round trips you were paying anyway.

And because there's no warm pool, an idle room costs nothing. Not "a little" — nothing, because there is no VM until someone opens the room, and there is no VM after the last participant leaves. Meeting-shaped traffic spends most of the day idle, so that's not a rounding error in your bill; it's most of it.

from pandastack import Sandbox

SFU_BOOT = """#!/bin/bash
set -euo pipefail
# Written by the control plane before we start: this room's public address
# and its slice of the UDP port space.
set -a; . /etc/room.env; set +a

setsid node /opt/sfu/server.js > /var/log/sfu.log 2>&1 < /dev/null &

# "Ready" means the signaling port is listening. ICE and DTLS happen after
# this, with the client, and are not ours to wait on.
for _ in $(seq 1 100); do
  nc -z 127.0.0.1 "$SIGNAL_PORT" && exit 0
  sleep 0.1
done
echo "sfu never bound $SIGNAL_PORT" >&2
exit 1
"""


def open_room(room_id: str, tenant: str, public_ip: str, ports: tuple[int, int]) -> str:
    """One microVM per room. Created on room-open, killed on room-close."""
    lo, hi = ports
    sbx = Sandbox.create(
        template="base",
        ttl_seconds=4 * 60 * 60,          # backstop: nobody meets for four hours
        metadata={"room": room_id, "tenant": tenant, "kind": "sfu"},
    )

    # The announced address is configuration, not something the guest can
    # discover -- from in there, STUN just describes our own NAT back to us.
    sbx.filesystem.write(
        "/etc/room.env",
        f"ROOM_ID={room_id}\nROOM_PUBLIC_IP={public_ip}\n"
        f"ROOM_RTC_MIN={lo}\nROOM_RTC_MAX={hi}\nSIGNAL_PORT=7880\n",
    )
    sbx.filesystem.write("/opt/sfu/boot.sh", SFU_BOOT)

    boot = sbx.exec("bash /opt/sfu/boot.sh", timeout_seconds=30)
    if boot.exit_code != 0:
        sbx.kill()                        # never leave a half-born room around
        raise RuntimeError(f"room {room_id}: {boot.stderr[-2000:]}")
    return sbx.id


def close_room(sbx: Sandbox) -> dict:
    """Last participant left: read the meter, then delete the machine."""
    tx = sbx.filesystem.read("/sys/class/net/eth0/statistics/tx_bytes")
    egress = int(tx.decode().strip())     # this tenant's media egress, measured
    sbx.kill()                            # transports, buffers, stray ffmpeg: gone
    return {"egress_bytes": egress}


def transcode_recording(recording_url: str) -> bytes:
    """Recording/transcoding is the CPU spike that wrecks live rooms. Run it
    in a throwaway VM instead of on the box that is carrying a call."""
    with Sandbox.create(
        template="base",
        ttl_seconds=1800,
        metadata={"kind": "recording-postprocess"},
    ) as job:
        job.filesystem.write(
            "/work/run.sh",
            f"set -eux\ncurl -fsS -o /work/in.webm '{recording_url}'\n"
            "ffmpeg -y -i /work/in.webm -c:v libx264 -c:a aac /work/out.mp4\n",
        )
        r = job.exec("bash /work/run.sh", timeout_seconds=1500)
        if r.exit_code != 0:
            raise RuntimeError(r.stderr[-2000:])
        return job.filesystem.read("/work/out.mp4")

Forking is less load-bearing here than in stateful simulation workloads, and I'd rather say so than oversell it — a restore is already fast enough for room creation, and an SFU's value isn't in warm accumulated state. Where it does earn its place is when your room image loads something expensive and identical every time: a noise-suppression model, a big per-tenant config bundle, a codec cache. Fork a parent that already has those resident and each room inherits them copy-on-write. Same-host forks land in 400–750ms; cross-host is 1.2–3.5s when you're deliberately spreading load.

Shared SFU vs. container per room vs. microVM per room

• Shared SFU process, all rooms — Density: highest, and the operational surface is one server you know well. Isolation: none that matters; one room's recording spike, transcode, or forty-person fan-out degrades every other room, and a worker crash drops all of them at once. Networking: simplest — one public address, one port range, no per-room mapping. The right answer far more often than isolation advocates admit.
• Container per room — Density: high, startup fast, per-room CPU/memory limits via cgroups. Isolation: shared host kernel, so a kernel-level bug reachable from hostile RTP/DTLS parsing crosses rooms, and noisy-neighbour effects still leak through shared kernel-side networking and scheduling. Networking: you're already doing per-room port mapping, so you pay most of the UDP/ICE tax without getting a guest kernel for it.
• MicroVM per room — Density: lowest per host, but idle rooms cost nothing because they don't exist, and creation is a snapshot restore (p50 179ms) rather than a cold boot. Isolation: a hardware-virtualised guest kernel per room, so a crash, OOM, or exploit is contained to one call, with egress measurable and cappable at the VM's own network namespace. Networking: the most work — public addressing, a per-room UDP port slice, correct announced addresses, and a TURN tier you still have to run separately.

When a shared SFU is simply the right answer

If you're single-tenant — your own product, your own users, one SFU build, modest concurrency — the isolation buys you very little and the networking costs you a lot. You'd be trading a well-understood single server for N media servers, which means N sets of metrics, N things to upgrade, N chances for the announced address to be wrong, and a port-allocation subsystem you now maintain. Run the shared SFU. Put your effort into capacity headroom and into moving recording and transcoding off the media path, which is where most of the spikes actually come from anyway.

The calculus tips when tenants are strangers to each other, when their usage patterns are wildly asymmetric, when a single customer's webinar can measurably degrade everyone else, when you need defensible per-tenant bandwidth attribution, or when a compliance story requires that one customer's media never share a kernel with another's. There's also a middle setting people forget: a VM per tenant rather than per room. You keep the fate and cost isolation between customers, amortise the VM across all of that tenant's concurrent rooms, and slash the number of public-port slices you have to manage. For most B2B video products that's the sweet spot, and it's where I'd start.

One honest note on alternatives: LiveKit Cloud, Daily, Twilio, Agora, Vonage and friends sell the managed version of this problem, and their isolation models, egress pricing, and per-tenant guarantees differ from each other in ways that matter — verify them against each vendor's own documentation rather than against anyone's summary. Buying is very often the correct call. Building per-room microVMs makes sense when you need the hardware isolation boundary itself, when your egress economics only work if you control the media path, or when you're the one selling the platform.