Running Customer Trading Strategies in Isolated microVMs

You built a platform where customers write their own strategies. Maybe you are a broker who shipped an algo feature, a signals or copy-trading service, a crypto bot host, or a research platform that graduated from backtests to paper and then to live. The product is the same in every case: the customer writes Python, you run it, and when it decides to buy something, something buys it. The code is theirs. The machine is yours. The credential that turns a decision into a fill is also yours, and it is sitting somewhere very close to their code.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, so read this as opinionated. I want to talk about the live-execution shape specifically — strategies running as a continuous multi-tenant production workload — rather than backtest throughput, which I've written about separately. Execution has a different threat model, a different failure mode, and a much less forgiving audience when it goes wrong. The industry's default answer is still "we run it in a container and audit the imports," and I'd like to explain why both halves of that sentence are weaker than they sound.

The threat model is unusual because the payload is valuable

In most untrusted-code products the attacker wants your infrastructure. They want a crypto miner, a spam relay, a foothold in your VPC, credentials to a cloud account. Your infrastructure is the prize, and everything you read about sandboxing is written with that assumption baked in: protect the host, protect the control plane, protect the metadata endpoint.

Here the prize is on the other side of the boundary. What is valuable on a strategy-execution platform is not your Kubernetes cluster; it is the other tenants' source code. Alpha is the asset. A strategy that reads another customer's strategy file has stolen the actual product — not a stepping stone to something better, the thing itself. And a very close second prize is whatever moves money: the broker API key, the exchange HMAC secret, the withdrawal-enabled token somebody added "temporarily" in 2024.

This inverts the usual defensive posture in a way that matters. Host compromise is still bad, but it is no longer the worst case, and it is no longer the case an attacker needs to reach. A payload that never touches your kernel, never escapes anything, and simply reads /srv/strategies/*.py or os.environ has already won. Which means the standard reassurance — "container escapes are rare and hard" — is answering a question your attacker was never going to ask.

On most platforms the attacker wants in. On this one they want what is already standing next to them: someone else's edge, and the key that turns a decision into a fill.

Then there is the other half of the threat model, which is not adversarial at all and which you will meet far more often. Somebody's strategy has a bug. It loops. It allocates until the box swaps. It submits the same market order four thousand times because a boolean was inverted and the position check compared a string to a float. None of that is an attack. A customer's infinite loop is not a security incident, it is a Tuesday. But a while True that submits market orders is a boundary problem with a P&L attached, and the boundary is the only thing that can stop it.

The shape: one machine per strategy, not one process

The baseline design is a disposable guest per strategy run — its own kernel under KVM, its own memory ceiling, its own network namespace — holding the strategy, a market-data client, and nothing else worth taking. On PandaStack that is a snapshot restore rather than a boot: about 179ms p50 and 203ms p99 end to end, with the restore step itself near 49ms. The very first cold boot of a template is around 3 seconds, paid once, and then never again.

That latency number is what makes the design practical rather than theoretical. A per-strategy machine you can create in under a fifth of a second is a machine you can create per session, per market open, per restart-after-crash, and per disputed-run replay. A per-strategy machine that takes forty seconds to provision quietly becomes a long-lived shared worker within about a month of production traffic, because someone will get paged and the reuse will look like a fix.

from pandastack import Sandbox

SESSION_SECONDS = 6 * 60 * 60          # one trading session
GRACE_SECONDS = 120                    # TTL backstop beyond our own timeout


def start_strategy(tenant_id: str, strategy_id: str, source: str):
    """Give one customer's strategy its own machine for one session."""

    # ttl_seconds is a backstop, not the primary lifetime control. If our
    # scheduler dies mid-session, the platform still reaps the guest. Cleanup
    # that depends on our own code running is cleanup that fails during an
    # incident, which is the only time we needed it.
    sbx = Sandbox.create(
        template="code-interpreter",
        ttl_seconds=SESSION_SECONDS + GRACE_SECONDS,
        metadata={
            "tenant": tenant_id,
            "strategy": strategy_id,
            "mode": "paper",           # promoted to "live" by a separate path
        },
    )

    # The harness is ours and is baked into the template. The strategy is
    # theirs and arrives at run time. Note what is NOT written here: no
    # broker key, no exchange secret, no database URL, no cloud role. The
    # guest has nothing to steal because we never gave it anything.
    sbx.filesystem.write("/run/strategy.py", source)
    sbx.filesystem.write("/run/universe.json", allowed_symbols_for(tenant_id))

    # Streamed so intents come back as they are emitted rather than at exit.
    # A strategy that runs for six hours and prints at the end is a strategy
    # whose orders you learn about six hours late.
    return sbx, sbx.exec_stream(
        "python /opt/harness/run.py /run/strategy.py",
        timeout_seconds=SESSION_SECONDS,
    )

Two details in there are load-bearing. The first is that the strategy file is written at run time into a guest restored from a pristine snapshot, so nothing from the previous tenant's run is on that disk or in that memory. The second is that the guest is given the tenant's symbol universe as data rather than being trusted to respect it — which is the theme of the next two sections, and honestly the theme of the whole post.

Credential proximity is the real bug

Here is the architecture almost everyone starts with, because it is the obvious one and it works on day one. The worker process holds the broker credential in an environment variable. The customer's strategy is imported into that process, or spawned as a child of it, and calls a helper that signs and sends the order. It is clean, it is fast, and it is one os.environ read away from being over.

If a broker API key is an environment variable in the process that runs customer code, the isolation boundary is decorative. It does not matter whether that process lives in a container, a jail, a gVisor sandbox or a Firecracker guest. The code does not need to escape anything; it needs to call getenv. Every hardening measure you apply is protecting a perimeter that has the crown jewels sitting inside it. And a child process inherits the environment by default, so "we spawn it as a subprocess" moves the key precisely nowhere.

The correct shape is that the strategy never sees a credential, because the strategy never places an order. It emits an intent: a structured description of what it wants to happen, written to a narrow channel. Outside the guest, a service you wrote and the customer cannot edit reads that intent, validates it against risk limits and account state, and — if and only if it passes — signs and places the order with a key that has never been inside a guest. The guest's most powerful primitive becomes "ask politely," which is a much better maximum privilege than "transact."

# ============================================================
# INSIDE THE GUEST  --  /opt/harness/broker.py
# The customer's strategy does "import broker" and calls submit().
# There is no key in this file, in the environment, or on this disk.
# ============================================================
import json, sys, time, uuid


def submit(symbol, side, qty, order_type="market", limit_price=None):
    """Emit an order INTENT. Returns a client id, NOT a fill.

    This function cannot place an order. It cannot sign anything. All it
    can do is write a line to stdout with a sentinel prefix. The most
    hostile possible implementation of this function is still just a
    process writing bytes to a pipe.
    """
    client_id = str(uuid.uuid4())
    sys.stdout.write("PS-INTENT " + json.dumps({
        "client_id": client_id,
        "symbol": symbol,
        "side": side,
        "qty": qty,
        "type": order_type,
        "limit_price": limit_price,
        "emitted_at": time.time(),
    }) + "\n")
    sys.stdout.flush()
    return client_id


# ============================================================
# OUTSIDE THE GUEST  --  the order gateway
# Holds the credential. Owns the limits. Customer cannot edit it,
# read it, reach it over the network, or crash it from inside.
# ============================================================
def on_guest_line(tenant, run, line, book, broker):
    if not line.startswith("PS-INTENT "):
        log_stdout(run, line)          # ordinary print(), not an order
        return

    try:
        intent = json.loads(line[len("PS-INTENT "):])
    except ValueError:
        return reject(run, "malformed intent")

    # Everything below is enforced here because here is the one place the
    # customer's code has no vote. Each of these has a real incident
    # behind it, and none of them are exotic.
    ok, why = risk_check(tenant, intent, book)
    if not ok:
        record(run, intent, decision="rejected", reason=why)
        if why in ("rate", "notional"):
            halt_strategy(run, reason=why)   # a loop does not get 4000 tries
        return

    # First time a secret appears anywhere in this flow, in a process the
    # guest cannot see, cannot reach, and does not know the address of.
    fill = broker.place(tenant.account_id, intent, key=broker.key_for(tenant))
    record(run, intent, decision="accepted", broker_order_id=fill.id)

The channel does not have to be stdout. A vsock socket, a unix socket bridged by the host, an HTTP endpoint reachable only from that guest's network namespace — all fine. What matters is that it is narrow, that it is parsed by your code, and that the guest cannot address anything else. Stdout has one underrated property for this job: it is trivially auditable, it is already streaming, and there is no plausible confusion about what the guest can do with it.

Risk limits belong outside the guest, without exception

The same argument applies to every control you care about, and it is worth stating in its most annoying form: a risk limit enforced inside the guest is a risk limit enforced by the code you have already decided not to trust. Position size, order rate, notional caps, symbol allowlists, max drawdown, market-hours gating — if any of those live in a Python module the strategy imports, they are suggestions. Not because customers are malicious, though some are, but because a helper function is one monkeypatch, one shadowed name, or one sincere bug away from not running at all.

So push all of it into the gateway. The list below is not a security wishlist; it is a list of things that have gone wrong on real platforms, and the shared property is that the guest is the wrong place to catch any of them.

• Symbol allowlist — the intent names an instrument this account is permitted and funded to trade. Cheap to check, and the check has to be against your record of the account, not against the JSON you handed the guest at startup.
• Notional and quantity caps — per order and per session, denominated in currency rather than share count, because a quantity that is sane for one instrument is a career-ending typo on another.
• Order rate — a token bucket per strategy, with the important behaviour being what happens when it empties. Rejecting order 4001 and letting the loop continue is not a fix; it is a slower version of the same incident. Halt the run.
• Position and exposure ceilings — computed from your own book, not from a counter the guest keeps, and evaluated before the order goes out rather than after the fill comes back.
• Market state — reject orders outside the session, during a halt, or into an instrument that stopped trading, because the strategy's view of the clock came from a guest and I would not trust that clock (see below).
• Kill switch — one flag, per strategy and per tenant and globally, checked on every intent. When it flips, intents are recorded and rejected. Being able to stop everything in one place is worth more than every other control on this list combined.

The fat-finger loop is more likely than the attack, by a wide margin, and it is the one that gets written up afterwards. A strategy that submits thousands of duplicate market orders in a few seconds does not need to be hostile to be expensive, and the only thing that can stop it is something the customer cannot edit. Build the gateway for the sincere bug and it will hold against the malicious one for free; build it for the attacker only and the bug will still get through, because the bug is doing something the attacker never bothered to try.

Fairness: the strategy that pegs a core at the open

Multi-tenant execution has a resource problem that backtesting does not, and it is entirely about timing. Every strategy on your platform wants CPU at the same instants: the open, the close, an economic print, a liquidation cascade. That is precisely when one tenant's poorly-vectorised pandas loop, or a numpy call that quietly spawned a thread per core, degrades everybody else's latency — and "everybody else's latency" on this platform means their fills.

Cgroups can express these limits, and on a well-run container platform they usually do. My objection is narrower and mostly operational: a cgroup limit is a promise made in configuration, applied by an orchestrator, on a kernel shared with the workload it is constraining, and it is one templating mistake away from not being applied. A microVM's memory ceiling is not a policy that someone remembered to set — it is how much RAM the machine has. There is no configuration path in which a guest gets more, because the guest cannot perceive more.

The practical consequence is the one you want during a market open: a strategy that leaks memory OOMs inside its own guest, kills its own Python process, and produces a failed run for one tenant. It does not evict a neighbour, and it does not put the host's page cache into a state that makes everyone else slow. On PandaStack that ceiling comes from the template the snapshot was baked at, which has a pleasant side effect — the limit travels with the machine image rather than living in a deployment manifest that can drift. Being honest about what this does not fix: hyperthread contention, memory bandwidth, and the storage layer are still shared, so a guest with its own kernel is isolated, not immune. Capacity planning does not go away.

Determinism, replay, and the customer who disputes a fill

Eventually a customer says the platform did something their strategy did not ask for. Sometimes they are right. Answering that question requires re-running the exact code against the exact data and getting the exact answer, and the number of platforms that can actually do this is smaller than the number that believe they can.

Snapshot and fork are genuinely good at this, and it is the strongest technical argument for the model. Bake a snapshot with the toolchain, library versions and the run's data warm in memory; fork it per run. A same-host fork takes 400–750ms — copy-on-write guest memory and a reflink rootfs clone, so it is O(metadata) rather than O(dataset) — and cross-host is 1.2–3.5s. Every run starts from a byte-identical machine state. The environment is not described by a lockfile that you hope still resolves the same way; the environment is the snapshot.

from pandastack import Sandbox

def replay(run_record):
    """Re-run a disputed session from the exact machine state it used."""

    # The snapshot the original run forked from. Pin its id on every run
    # record at execution time. A replay against "the current template" is
    # not a replay, it is a new experiment with a familiar name.
    warm = Sandbox.snapshot_ref(run_record["snapshot_id"])

    sbx = warm.fork(ttl_seconds=1800)
    try:
        sbx.filesystem.write("/run/strategy.py", run_record["source"])
        sbx.filesystem.write("/run/ticks.jsonl", fetch_recorded_feed(run_record))

        # A restored guest wakes up believing it is the instant the snapshot
        # was taken, with the SAME RNG state it had then. For a backtest that
        # is a curiosity. For anything that timestamps or seeds an order it
        # is a correctness bug, so refresh both explicitly, first thing.
        sbx.exec("chronyc -a makestep || sudo hwclock -s")
        sbx.exec("python /opt/harness/reseed.py --seed " + run_record["seed"])

        # Replay the recorded feed instead of the live one. Same bytes, same
        # order, same gaps -- including the 40-second gap the customer is
        # actually complaining about.
        r = sbx.exec(
            "python /opt/harness/run.py /run/strategy.py --replay /run/ticks.jsonl",
            timeout_seconds=1500,
        )
        return parse_intents(r.stdout)
    finally:
        sbx.destroy()

The rest of determinism is not the platform's job and I will not pretend otherwise. Floating-point results vary with BLAS thread counts and CPU features; dict iteration order is stable in modern Python but set iteration is not; a strategy calling time.time() or random.random() without a seed is non-deterministic by construction and no hypervisor can save it. What the snapshot gives you is the elimination of the environment as a variable, which in my experience is where the majority of "it doesn't reproduce" cases actually live. The rest is a harness discipline problem: pin the seed, record the feed, forbid wall-clock reads outside a provided clock object.

Egress: default-deny, one allowlist entry, and an alarm

A strategy needs market data. That is the entire list. It does not need the metadata endpoint, your internal network, your database, your control-plane API, PyPI at run time, or an arbitrary host on the internet. So the network policy writes itself: default-deny outbound, with a narrow allowlist to the data feed and nothing else, enforced outside the guest where the guest's code cannot renegotiate it.

On PandaStack each sandbox gets its own network namespace with a veth pair and TAP device — 16,384 pre-allocated /30 subnets per agent — so filtering is host-side rather than a setting inside the guest. The mechanism matters less than the placement: any enforcement point the guest can reach is an enforcement point the guest can eventually argue with.

The best property of this policy is not the blocking, it is the signal. A strategy has no legitimate reason to resolve an unfamiliar hostname, and on a platform where the valuable thing is other people's source code, an outbound connection attempt to somewhere unexpected is one of the highest-quality alerts you will ever get. Log every denied flow with the tenant and strategy attached. Most of them will be a library phoning home for telemetry, which is worth knowing anyway. The rest will be the most interesting ticket of your quarter.

Be honest about the residual. If the data feed is allowlisted, the data feed is a channel — subscription symbols, request timing and query patterns can all carry a few bits per second to a determined party. A tight allowlist bounds the risk and makes exfiltration an engineering project instead of a one-line requests.post. It does not reduce it to zero, and anyone selling you a design that does is measuring a different thing.

Audit and evidence, briefly and without legal advice

I am not a lawyer, this is not legal or compliance advice, and PandaStack makes no certification claims — check your own obligations with people qualified to tell you what they are. What I will say is an engineering observation: obligations around per-tenant execution isolation and per-run record-keeping are dramatically easier to evidence when a run is a discrete machine with a discrete lifecycle.

"Tenant A's code cannot read tenant B's code" is an argument about namespaces, mount propagation and orchestrator configuration in a shared-kernel design, and it needs to be re-made every time the deployment changes. In a guest-per-run design it is an argument about whether the machines are different machines. "Here is exactly what ran" is a lockfile and a hope on a mutable runner; it is a snapshot id, a source hash, an intent log and a start and stop timestamp when the run is a VM. Immutability comes free from the fact that the machine is destroyed rather than reused, so there is no drift to reconcile — nothing wrote to it afterwards because it stopped existing.

Record the run as a record: tenant, strategy id and source hash, snapshot id, template, start and stop time, every intent with its accept-or-reject decision and reason, and every broker order id that came back. That set answers almost every dispute you will get, and it is a natural by-product of the gateway you already had to build for risk limits.

Same-process, container, microVM, or a separate account

Softest boundary to hardest. The container column describes general architectural properties and common defaults rather than any particular platform — a carefully configured setup with per-job network policy, dropped capabilities and ephemeral runners closes several of these gaps, so verify against your own platform's current docs. The only measured numbers here are PandaStack's.

• Same-process eval or import — Isolation: none; every strategy shares an interpreter, so one os.environ or gc.get_objects() walk reads the other tenants. Credential exposure: total, the key is in the address space. Cost: effectively zero, which is exactly why it survives to production. Cold start: none, and that is the whole temptation.
• Container per tenant — Isolation: namespaces and cgroups on a shared kernel; good against accidents, weaker against a payload that never tries to escape. Credential exposure: whatever is in the environment or a mounted secret, which is usually more than intended and inherited by every child process. Cost: low, one image cached per node. Cold start: milliseconds warm, seconds on a cold image pull.
• microVM per strategy run — Isolation: an own-kernel guest under KVM (5.10, Firecracker v1.16), its own network namespace, RAM fixed by the machine rather than by a policy file. Credential exposure: zero if you use the intent pattern, because the key never enters the guest. Cost: a VM's worth of RAM per running strategy, which is the real bill and worth pricing honestly. Cold start: about 179ms p50 and 203ms p99 from a baked snapshot, roughly 3 seconds for the one-time cold boot, 400–750ms for a same-host fork.
• Separate cloud account or broker sub-account per tenant — Isolation: the strongest available, and the one auditors like most. Credential exposure: contained by construction, since a leaked key is scoped to one customer's own account. Cost: high in operational overhead — provisioning, quotas, billing and lifecycle per tenant — and it does not solve the in-account problem of what the strategy can do with its own key. Cold start: minutes to days, depending on who has to approve the account.
• What each one does with a runaway while True — Same-process: takes the worker and every strategy in it. Container: cgroups throttle it if they were set, and the shared kernel and page cache still feel it. microVM: it exhausts its own guest's RAM and dies alone. Separate account: it runs beautifully, at full speed, until the customer's own risk limits or margin call stop it.

These compose better than they compete. The design I would actually build is a microVM per strategy run for isolation and reproducibility, an out-of-guest gateway for credentials and risk, and separate broker sub-accounts per tenant where the broker supports them, because the last one turns a bad day into one customer's bad day.

What to take away

Strategy execution is the highest-consequence version of the untrusted-code problem because the valuable things are inside the perimeter rather than outside it. The other tenants' source code is the product, the broker key moves money, and the attacker does not have to escape anything to reach either one. Design accordingly: a real machine boundary per run so one tenant's code cannot read another's; an intent channel so the guest asks rather than transacts; risk limits, rate limits and the kill switch in a gateway the customer cannot edit; default-deny egress with one allowlist entry and an alert on everything else; and a snapshot id plus an intent log per run so a dispute is answerable.

Snapshot-restore is what makes the per-run machine affordable at 179ms p50, and fork is what makes replay pleasant at 400–750ms. Refresh the clock and the entropy pool on resume or you will ship a very confusing class of bug. And accept the framing that makes the rest of it obvious: the customer's infinite loop is not a security incident, it is a Tuesday — you just have to make sure Tuesday costs one machine that was going to be deleted anyway, rather than four thousand market orders and a phone call.