Per-User Notebook Kernels: Isolating a Hosted Jupyter Product

If you're building a hosted notebook product — the Deepnote / Hex / Colab shape, or any data-science or BI tool where users write and run cells in the browser — you have quietly built a multi-tenant remote-code-execution service. Every cell a paying customer runs is arbitrary Python your platform executes on their behalf, and it runs somewhere. The entire security posture of the product comes down to one question: whose kernel is that, and what else can it reach? This post is about the answer that holds up when tenant A and tenant B are both companies paying you real money — one Firecracker microVM per user session — and about making it cheap with hibernate-on-idle, because nobody runs cells twenty-four hours a day.

I'm Ajay; I built PandaStack, a Firecracker microVM platform. There's a companion post on sandboxing a single untrusted notebook. This one is specifically about the product problem: you're not running one stranger's notebook, you're running a fleet of paying tenants' kernels side by side, forever, and the failure mode is a data leak between customers rather than a crashed box. That changes the architecture, the economics, and what you have to get right.

The threat: a shared kernel pool between paying tenants

The tempting first architecture for a hosted notebook product is a warm pool of kernels: keep N Python interpreters hot in a cluster, hand one to whichever user runs a cell next, reset it when they're done. It's fast, it amortizes startup, and it's a data-leak waiting to happen — because the reset is never as clean as you think and the kernel is a full interpreter with a shell one import away. Picture the most innocent-looking cell one of your customers could type:

import os
# "just poking around the runtime"
print(os.environ)                       # your DB URLs, your API keys, maybe another tenant's
os.listdir("/proc")                     # what else is running on this box?
# and the cell that ends up in a postmortem
for pid in os.listdir("/proc"):
    if pid.isdigit():
        try:
            env = open(f"/proc/{pid}/environ").read()
            if "TENANT" in env or "TOKEN" in env:
                print(pid, env)          # peek at a neighbor's process env
        except OSError:
            pass

On a shared kernel or a shared host, `/proc` is a directory of everyone else's secrets. Walk the process table, read a neighbor's `environ`, find the connection strings and bearer tokens the platform handed the tenant next door, and you've exfiltrated one customer's data through another customer's account. Nothing here is an exploit — it's the standard library reading files that are right there. And that's the polite attacker. The impolite one runs `while True: pass` across every core, or allocates until the OOM killer starts shooting neighbors, and takes down the shared pool for everybody.

Why the kernel process boundary isn't the boundary

The obvious upgrade is "give each user their own kernel process," maybe in their own container. Better — but the process and the container are not the trust boundary you need, and the reason is architectural, not a tuning gap. A Linux container is namespaces and cgroups wrapped around a process that still shares the host's single kernel with every other tenant on the machine. That shared kernel is millions of lines of privileged C, and it is exactly the surface a malicious cell probes to escape. Per-user kernel processes stop the casual `/proc` walk from landing on a neighbor inside the same container, but they do nothing about the day a kernel CVE drops or your seccomp policy has a gap — and on that day the blast radius is every tenant on the host.

You can harden a long way — seccomp, user namespaces, dropped capabilities, read-only rootfs, gVisor or Kata in front — and you should do the network and resource pieces regardless. But every one of those is a mitigation layered on a boundary that was never designed to contain adversarial code between paying tenants. For a product whose whole value proposition includes "your data is yours," the boundary has to be a separate kernel behind a hardware virtualization wall the CPU enforces, so a compromised kernel is trapped inside a guest that can't see the host at all. That's a microVM — the same boundary AWS chose for Lambda and Fargate rather than trusting containers between customers.

A per-user kernel process isolates one Python interpreter from another. A per-user microVM isolates one guest kernel from another. In a product where the tenant is a stranger with a credit card, only the second one is a promise you can keep.

The model that holds: one microVM per user session

The architecture that survives a real customer base is one hardware-isolated VM per user session (or per notebook). When a user opens a notebook, you create a sandbox; their kernel lives inside it; their cells execute there; their files sit on its filesystem. Tenant A's VM and tenant B's VM are different guest kernels on different copy-on-write memory — there is no shared `/proc` to walk, no neighbor's disk to traverse into, no shared scheduler to starve. An escape would have to break the hypervisor itself, a vastly smaller and more audited surface than the full Linux syscall table a container sees.

The historical objection is startup cost: nobody wants their notebook to hang for eight seconds while you cold-boot a VM. That's the objection PandaStack is built to kill. Every create restores a baked snapshot on demand instead of cold-booting, so a session comes up at p50 179ms (about 49ms for the snapshot-restore step itself), p99 ~203ms. The very first spawn of a template is a ~3s cold boot that captures the snapshot; every session after that takes the fast path. VM-grade isolation between tenants, container-ish latency to the user.

Hibernate on idle, wake on the next cell: the scale-to-zero economics

Here's the economic reality of a notebook product: people open a notebook, run a few cells, then go to a meeting, go to lunch, or close the tab and forget it exists. The median kernel spends the overwhelming majority of its life idle. If every open notebook is a VM burning RAM, your bill scales with tabs-left-open, which is a terrible thing to scale with. The fix is scale-to-zero at the session level: when a session goes idle, hibernate it — snapshot memory plus disk, then stop the VM so it holds no compute — and wake it on the next cell run.

Because a wake is a snapshot restore, it's fast enough to sit in front of a "Run" click. The user's session comes back on the same restore fast path a fresh create uses, with their kernel state and files intact from the snapshot. A classroom of 200 students who all wandered off, or 5,000 trial accounts with a notebook open and abandoned, cost you almost nothing while idle instead of 200 or 5,000 live VMs. You're renting rooms only while someone is in them.

• Idle detection: after N minutes with no cell execution and no active connection, mark the session idle. Don't let a background poll from the UI count as activity, or nothing ever hibernates.
• Hibernate: snapshot the VM (memory + disk) and stop it. The session now holds zero vCPU and zero RAM on the host — it's bytes on disk, not a running machine.
• Wake on demand: the next cell run (or the user re-opening the notebook) restores the snapshot on the fast path and resumes the kernel mid-instruction, so live objects and imported state survive.
• Reap the truly dead: pair hibernation with a TTL so a session nobody ever comes back to is eventually deleted, not hibernated forever.

A persistent volume per user, so their files outlive the kernel

A notebook product needs durable per-user files: uploaded CSVs, saved outputs, the notebook itself. You don't want those living only in a kernel's ephemeral rootfs that dies when the session is reaped. The pattern is to attach a persistent volume scoped to the user (or the workspace) to their session VM. The kernel is disposable; the volume is not. When the session hibernates or gets torn down, the files stay; the next session mounts the same volume and the user's `data.csv` is right where they left it.

Crucially, that volume belongs to exactly one tenant. It is not a shared NFS mount where a path like `/home/*/notebooks/*` walks into a neighbor's files — that's the shared-kernel disk-traversal leak wearing a nicer hat. One user, one volume, mounted only into that user's VM. If a tenant needs a real database rather than files, give them their own: a managed Postgres microVM provisions in 30–90s and is its own isolated VM, not a schema on a shared instance every tenant can enumerate.

The per-user kernel loop in code

Here's the shape with the Python SDK: create a per-user session VM on the code-interpreter template (pandas, numpy, matplotlib, scikit-learn, and an IPython kernel are baked in, so there's no per-session pip install), write the user's cell to the guest, exec it with a hard timeout, stream output back to the notebook UI, and hibernate the session when it goes idle. Set PANDASTACK_API_KEY in the environment first. The `metadata` tag attributes the VM to a tenant for auditing and egress policy; `ttl_seconds` is a backstop for the tab the user closed without logging out (they always do).

from pandastack import Sandbox

def open_session(user_id: str, notebook_id: str) -> Sandbox:
    """One microVM per user session. TTL is a backstop for abandoned tabs."""
    return Sandbox.create(
        template="code-interpreter",
        ttl_seconds=3600,                     # reap a forgotten session after an hour
        metadata={"user": user_id, "notebook": notebook_id, "kind": "kernel"},
    )

def run_cell(sbx: Sandbox, source: str, n: int) -> int:
    """Execute one cell in the user's OWN kernel VM, streaming output live."""
    path = f"/workspace/cell_{n}.py"
    sbx.filesystem.write(path, source)
    # ALWAYS pass a timeout. User cells loop more than you'd believe.
    for event in sbx.exec_stream(f"python3 {path}", timeout_seconds=30):
        if event.type == "stdout":
            push_to_notebook_ui(event.data)   # your websocket to the browser
        elif event.type == "stderr":
            push_to_notebook_ui(event.data, stream="stderr")
        elif event.type == "exit":
            return event.exit_code
    return -1

# --- one user's session ---------------------------------------------------
sbx = open_session(user_id="u_42", notebook_id="q3-revenue")

cell = '''
import pandas as pd
df = pd.read_csv("/data/revenue.csv")     # from THIS user's persistent volume
print(df.groupby("region")["amount"].sum())
'''
code = run_cell(sbx, cell, n=1)
print("cell exit:", code)

# The user goes quiet (meeting, lunch, closed the tab). Hibernate the session:
# snapshot memory + disk, stop the VM. It now holds zero compute on the host.
sbx.hibernate()

# ...next cell run (or the user comes back). Wake it on the restore fast path —
# the kernel resumes mid-instruction, so `df` from cell 1 is still in memory.
sbx.wake()
code = run_cell(sbx, 'print(df.shape)', n=2)   # df survived the nap

Note the two safety rails, because they're load-bearing, not decoration. The `timeout_seconds` on each exec is a circuit breaker for the cell that loops forever; the `ttl_seconds` on create is the backstop for the session that never cleanly closes. And the hibernate/wake pair is the whole economic story: between the first cell and the second, the user paid nothing to keep their kernel warm, but the second cell still saw the dataframe from the first because wake resumes the snapshot rather than starting clean.

Package installs and egress that don't leak

Two things every notebook product has to let users do — install packages and reach the network — are also the two places a shared architecture leaks. Per-user VMs fix both by construction, but you still have to set the policy.

• Package installs stay in the tenant's VM: a user running pip install lands packages on their own guest filesystem, not a shared site-packages every tenant imports from. No dependency-confusion or malicious-package blast radius beyond the one VM — and none of it survives into another user's session, because it was never in another user's VM.
• Bake the common case: put pandas, numpy, matplotlib, scikit-learn, and the kernel into the template snapshot so the median session needs zero installs and comes up instantly. Only the long-tail dependency triggers a pip install, and it's contained.
• Default-deny egress for untrusted tenants: the VM contains execution, but a cell can still POST computed data to an attacker if it can open a socket. Restrict outbound at the network layer, not in the notebook code.
• Block the private plane and metadata endpoint: if a tenant needs egress, still deny RFC1918 (10/8, 172.16/12, 192.168/16) and link-local 169.254.0.0/16 — which covers the cloud metadata IP at 169.254.169.254 — so a cell can never reach your database, control plane, or IAM credentials.
• Never inject platform secrets into the guest: a VM isolates the user's code from your host; it does not protect a connection string you handed straight to that user's kernel. Keep your credentials out of the tenant's environment entirely.

VM-per-session vs. container-per-session vs. shared kernel pool

Three architectures for where a hosted notebook's kernel runs, side by side rather than picked dogmatically. Verify the specifics of any competitor's runtime against their own docs — I'm describing the shapes, not claiming to know exactly how each vendor is built:

• Isolation boundary — VM-per-session: a hardware-virtualized guest kernel per user; an escape has to break the hypervisor. Container-per-session: namespaces + cgroups on one shared host kernel; a kernel CVE or container escape reaches every tenant. Shared kernel pool: one interpreter reused across users; the weakest wall, wrong for adversarial multi-tenant code.
• Cross-tenant data leak — VM-per-session: nothing to reach; separate VMs, separate memory, separate disks. Container-per-session: a shared /proc and often a shared disk are one traversal away. Shared kernel pool: a botched reset or a leaked variable is a direct breach.
• Runaway cell (infinite loop, OOM, fork bomb) — VM-per-session: capped to one VM's vCPU/RAM; the fleet never notices. Container-per-session: cgroups help but a shared kernel can still be starved. Shared kernel pool: one noisy user degrades or kills everyone on the box.
• Idle cost — VM-per-session: near zero if you hibernate idle sessions (snapshot + stop, wake on next cell). Container-per-session: you pay to keep it warm or eat a cold start. Shared kernel pool: low compute cost, paid for in blast radius.
• Startup / wake latency — VM-per-session: snapshot-restore create and wake at p50 179ms (~203ms p99), ~3s only on the first cold boot of a template. Container-per-session: sub-second to spawn, but state doesn't survive a reset. Shared kernel pool: instant, because you skipped the boundary you needed.
• Best fit — VM-per-session: any product where tenants are companies or the public and "your data is yours" is a promise. Container-per-session: semi-trusted internal users where the company is the trust boundary. Shared kernel pool: a demo, a single-tenant install, or trusted-employees-only.

Does a VM per user actually scale?

The capacity fear is usually misplaced. A single agent pre-allocates 16,384 /30 subnets, so networking is not the ceiling — host memory and CPU are. And because idle sessions hibernate down to bytes on disk, the number of live VMs tracks the number of users actively running cells right now, not the number of open tabs. You scale tenants horizontally by adding hosts, and the snapshot-restore path means each host stands sessions up in ~179ms rather than minutes.

If you want every user to start from an identical pre-warmed environment — a golden kernel with your libraries loaded and a dataset already in memory — snapshot that state once and fork it per session: a same-host fork is 400–750ms and shares memory copy-on-write (cross-host 1.2–3.5s). That's how you give thousands of users a rich, pre-loaded kernel without paying to boot each one from scratch.

When a shared kernel pool is actually fine

To be fair to the simple architecture: if every user of your notebook product is a trusted employee on the same team, a shared pool or per-user containers on a shared host is reasonable, and a per-session VM is overkill. The trust boundary is the company, and you've implicitly accepted that risk already. The model breaks the instant "users" means "customers" — different companies, prospects, students, the public — because then a leak isn't an internal accident, it's tenant A reading tenant B's revenue data through your product. The moment you can't vouch for the person writing the cell, the kernel they run on has to be theirs alone. Per-session microVMs, hibernated when idle, are how you make "theirs alone" cheap enough to actually ship.