How to Give a DSPy Program a Code Execution Sandbox

DSPy has one of the better pitches in the field, and it happens to be true: stop hand-tuning prompt strings, write the program's structure and a metric, and let an optimiser search for the prompts. It works. It also has a property almost nobody thinks about until it bites them, which is that the optimiser runs your program hundreds or thousands of times. If your program executes model-written code, the compile step quietly turns a small helper function into a machine that generates and runs unreviewed programs at industrial volume — and then points a search algorithm at it.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, so this shape reaches me fairly often: a DSPy program with a tidy little `exec()` helper that was completely fine in development, run through an optimiser for the first time, and now the laptop is behaving oddly and the scores make no sense. This post is about where code execution belongs in a DSPy program, and specifically about the compile-time multiplier — which turns out to be a correctness argument at least as much as a security one.

What DSPy actually is, in one section

If you have only heard the pitch: DSPy asks you to write the structure of an LLM program rather than its prompts. You declare a signature — the string form `"question -> answer"`, or a `dspy.Signature` subclass when you want typed fields and descriptions — and wrap it in a module. `dspy.Predict` for a single call, `dspy.ChainOfThought` when you want reasoning in the trace, `dspy.ReAct` when the model should be able to call tools, which it takes as `tools=[...]`. You compose those into your own `dspy.Module` with a `forward()`, the same way you would compose layers in PyTorch.

Then you write a metric — a function that scores a prediction against an example — and hand both to an optimiser (`BootstrapFewShot`, `MIPROv2`, and relatives) along with a training set. The optimiser searches. It proposes candidate instructions, bootstraps demonstrations by running your program and keeping the traces that scored well, tries combinations, and returns a compiled program with the winning prompts baked in. The mental model is a compiler, not a prompt-engineering session: you wrote the source, the optimiser produced the artefact, and `.compile(program, trainset=trainset)` is the build.

Where code execution enters the picture

Two places, mostly. The first is `dspy.ReAct` with a Python-execution tool in its `tools` list — the most powerful thing you can put there, which is exactly why people put it there. The second is a custom module whose `forward()` runs generated code as part of the program's own logic: the model writes a pandas script and you execute it to get the answer, or it writes a function and you run unit tests to decide whether it worked.

Both are good designs. Code as an action format beats JSON tool calls for anything involving arithmetic, data manipulation, or multi-step composition, and "run the tests" is a far better metric than "ask another model whether it looks right". The problem is not the design. The problem is what the optimiser does to it. Here is the version everyone writes first.

# The version everyone writes first.
import io
import contextlib
import dspy


def run_python(code: str) -> str:
    """Execute Python and return whatever it printed."""
    buf = io.StringIO()
    with contextlib.redirect_stdout(buf):
        exec(code, {})   # your process. your cwd. your environment variables.
    return buf.getvalue()


agent = dspy.ReAct("question -> answer", tools=[run_python])

# One run:     one piece of model-written code, in your process.
#
# One compile: optimiser.compile(agent, trainset=trainset)
#              -> candidates x examples x trials x tool-calls-per-episode
#              -> thousands of unreviewed programs, many of them generated
#                 from instructions the optimiser invented precisely because
#                 you would never have written them yourself.
#
# The line of code did not change. What it does changed completely.

The compile-time multiplier

Do the arithmetic. A modest compile might be twenty training examples, eight candidate prompt configurations, and a handful of trials — and DSPy will cheerfully evaluate far more if you let it. Every evaluation executes your program end to end, which means it executes your code tool, and a ReAct agent calling that tool two or three times per episode multiplies it again. You are now in the low thousands of executions of model-written Python, and you got there by running one line.

The count is not the interesting part. This is: during optimisation, most of that code is generated from prompts the optimiser invented. The entire point of the search is to try instructions you would not have written — strange ones, contradictory ones, badly worded ones — because sometimes strange instructions score well. Deliberately perturbing the prompt to see what falls out is not an attack technique here. It is the algorithm. You have automated adversarial prompt generation as a build step, wired its output to an interpreter, and left it running overnight.

In a single run, exec() is a risk you took. During a compile, it is a service you are operating.

The genuinely funny part is that the optimiser is the most responsible component in the pipeline. It is doing exactly what it promised: searching hard, faithfully, and without the faintest concept of what it is searching through — and what it is searching through happens to be your process, your working directory, and the environment variables holding your API keys. `rm -rf` is not the failure mode I worry about most, because at least that one announces itself. The one that gets you is a piece of generated code that quietly deletes the wrong file during trial 6 of 40, at 2am, while you are asleep and the score keeps ticking along.

The stronger argument is not security. It is correctness.

Security gets the headline, but the argument that should actually change your architecture is that a shared, mutable execution environment makes your optimisation scores meaningless.

Consider one long-lived interpreter reused across evaluations. Example A's generated code runs a `pip install`. It scores well. Example B, evaluated four minutes later, imports that package and passes — in an environment where a cold run would have failed. The optimiser records that B's prompt was good. It was not; the environment was. That demonstration gets bootstrapped into the compiled program as a few-shot example, and every future run is now steered toward an assumption that only holds on the machine where you compiled.

• Score inflation — one evaluation installs a package, warms a cache, or leaves a helper file behind, and a later evaluation passes for a reason that has nothing to do with the candidate being scored.
• Score deflation — one evaluation pins an incompatible version, corrupts a config, or leaves a lock file, and every subsequent evaluation fails through no fault of its prompt.
• Order dependence — the same trainset in a different order produces a different compiled program, so "my compile is reproducible" degrades into "my compile is reproducible provided nothing changed, which I cannot verify".
• Silent drift — process-global state (a monkeypatched module, a changed working directory, a mutated random seed) survives between evaluations and is invisible in the traces you inspect afterwards.

The optimiser cannot see any of this. It has exactly one channel — the metric's return value — and it treats that number as ground truth about the prompt. Contaminate the number and you have not merely degraded the search, you have redirected it. A compile that optimises against a dirty environment is worse than no compile at all, because it hands you an artefact that looks tuned while carrying a hidden dependency on a filesystem that no longer exists.

Optimisers parallelise. Shared environments do not.

DSPy evaluation is embarrassingly parallel — every (candidate, example) pair is independent — and the optimisers exploit that with a thread count you configure. This is where a shared execution environment stops being merely incorrect and becomes slow as well. One interpreter, one working directory, one `/tmp`: your options are to serialise behind a lock, throwing away the parallelism you asked for, or to let evaluations race, which gets you every corruption above non-deterministically. The racing version is much harder to debug, and much more likely to be the one you ship.

A sandbox per evaluation sidesteps the question entirely. Each worker gets its own machine with its own kernel, filesystem, and network namespace. Nothing is shared, so nothing needs a lock, and the parallelism knob goes back to being what you wanted it to be: a throughput setting rather than a correctness gamble.

A sandbox-backed execution tool for ReAct

The tool version is a small change. Same shape — code in, text out — with the execution happening inside a microVM you can throw away.

import dspy
from pandastack import Sandbox


def make_python_tool(sbx: Sandbox):
    ctx = sbx.create_code_context(language="python")  # persistent kernel

    def run_python(code: str) -> str:
        """Execute Python in an isolated microVM and return its output."""
        try:
            ex = ctx.run_code(code, timeout_seconds=30)
        except Exception as e:
            # Hand failures back as text. ReAct can recover from a NameError;
            # it cannot recover from an exception that ends the episode.
            return f"execution failed: {e!r}"
        logs = ex.logs
        parts = [logs.get("stdout", ""), logs.get("stderr", "")]
        out = "\n".join(p for p in parts if p)
        return out[:4000] or "(no output)"

    return run_python


sbx = Sandbox.create(template="code-interpreter", ttl_seconds=900)
try:
    agent = dspy.ReAct("question -> answer", tools=[make_python_tool(sbx)])
    print(agent(question="What is the 200th prime number?"))
finally:
    sbx.kill()

Three details matter more than they look. Errors are returned rather than raised, because a ReAct loop that sees a traceback in the tool output will usually fix its own code on the next step, whereas an exception escaping the tool ends the episode and, during a compile, costs you the example. Output is truncated, because a model that calls `df.to_string()` on a large frame will otherwise consume your context window and a meaningful slice of the token budget in a single action. And the timeout lives on the execution rather than on your patience — more on that below.

A module whose forward() executes in a fresh sandbox

For the generate-then-check pattern, the sandbox belongs inside `forward()`: one per call, created and destroyed around the execution. This is the shape that makes the metric honest, because the environment each evaluation sees is identical by construction, and does not exist afterwards.

import dspy
from pandastack import Sandbox


class CodeAndRun(dspy.Module):
    """Write a program, run it in a fresh microVM, return the result as data."""

    def __init__(self, snapshot=None):
        super().__init__()
        self.snapshot = snapshot  # fork a prepared env when we have one
        self.write = dspy.ChainOfThought("task, data_preview -> python_code")

    def _fresh_sandbox(self) -> Sandbox:
        if self.snapshot is not None:
            return Sandbox.fork(self.snapshot.id, ttl_seconds=300)
        return Sandbox.create(template="code-interpreter", ttl_seconds=300)

    def forward(self, task: str, data_preview: str = ""):
        pred = self.write(task=task, data_preview=data_preview)
        code = pred.python_code.strip()  # strip markdown fences here if needed

        sbx = self._fresh_sandbox()
        try:
            sbx.filesystem.write("/work/solution.py", code.encode())
            r = sbx.exec("cd /work && python solution.py", timeout_seconds=60)
            ok = r.exit_code == 0
            return dspy.Prediction(
                code=code,
                stdout=r.stdout[-4000:],
                stderr=r.stderr[-2000:],
                ok=ok,
                error="" if ok else f"exit_{r.exit_code}",
            )
        except Exception as e:
            # A broken execution is a datum. If this escapes, you do not lose
            # one example -- you lose the trial and everything it paid for.
            return dspy.Prediction(code=code, stdout="", stderr=repr(e),
                                   ok=False, error="sandbox_error")
        finally:
            sbx.kill()

Note what `forward()` returns on failure: a `Prediction` with `ok=False`, not an exception. That is the difference between a metric that scores one example zero and a compile that dies in trial 19 of 40 after ninety minutes of paid inference. Model-written code fails constantly — syntax errors, missing imports, infinite loops, confident nonsense — and during optimisation those failures are signal rather than incidents. Catch them, encode them as data, score them as failures, and let the search route around them.

Forking a prepared snapshot for identical starting state

Creating a sandbox per evaluation buys isolation. Forking a prepared snapshot per evaluation buys isolation and a pinned starting state, which is the thing the metric actually needs.

The move is the one you would make in any eval harness: do the slow, network-dependent setup once — install pandas and scipy, drop the dataset on disk, check out the repo — snapshot that VM, then start every evaluation as a copy-on-write fork of the snapshot. A same-host fork is 400-750ms and shares the parent's memory and disk copy-on-write, so the per-evaluation cost is a clone rather than a provision. More importantly, every evaluation in every trial begins from a byte-identical environment, so when candidate 7 beats candidate 3, the difference is the prompt and nothing else.

It also removes a whole class of flakiness from the compile itself. A `pip install` inside an evaluation means your optimisation run depends on PyPI's availability at 3am on the night you happened to start it, and a yanked version mid-compile shows up as a mysterious cliff in the scores of the last two trials — which you will spend an afternoon attributing to the prompts. Bake once, fork per evaluation, and the network stops being a variable in your search.

Putting it together: compile with setup and teardown

import os
import dspy
from dspy.teleprompt import BootstrapFewShot
from pandastack import Sandbox

dspy.configure(lm=dspy.LM(os.environ["DSPY_MODEL"]))

# ---- setup: bake the evaluation environment exactly once ----------------
base = Sandbox.create(template="code-interpreter", ttl_seconds=1800)
base.exec("pip install pandas numpy scipy", timeout_seconds=600)
with open("dataset.csv", "rb") as f:
    base.filesystem.write("/work/dataset.csv", f.read())
PREPARED = base.snapshot()   # every evaluation forks from this exact state


def metric(example, pred, trace=None) -> float:
    """Score one evaluation. Must never raise -- a raise loses the trial."""
    if not pred.ok:
        return 0.0   # failed or timed-out execution scores zero, not an error
    return float(pred.stdout.strip() == example.expected.strip())


try:
    optimiser = BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
    compiled = optimiser.compile(CodeAndRun(snapshot=PREPARED),
                                 trainset=trainset)
    compiled.save("compiled_code_program.json")
    print("compiled against snapshot", PREPARED.id)
finally:
    base.kill()  # scaffolding; the snapshot is the artefact worth keeping

The structure to copy is the `try/finally`. The bake sandbox is scaffolding and must be killed whether the compile succeeds, fails, or is interrupted at 3am by someone shutting a laptop lid. Set `ttl_seconds` on the forks as a backstop too: if the compile process dies, the TTL reaps the orphans instead of leaving a fleet of sandboxes running through the weekend on the strength of a `KeyboardInterrupt` you did not handle. And save the snapshot id next to the compiled artefact — that pair, not the JSON alone, is what makes the run reproducible.

Timeouts, and why a hung evaluation is worse than a failed one

Model-written code hangs. It writes a `while True` whose break condition never fires, reads from stdin that will never arrive, or opens a socket to a host that black-holes packets. In an interactive run you notice and press Ctrl-C. In a compile, a hung evaluation holds a worker slot, and if enough of them accumulate the optimiser stops making progress while continuing to look busy — the worst available failure mode, because it burns wall-clock and money and produces nothing you can debug.

Every execution needs a hard timeout at the sandbox boundary, not a soft one inside the generated code, because the generated code is precisely the thing you have decided not to trust with a limit. Then treat the timeout as a scored failure and move on. The optimiser will learn to prefer candidates whose code terminates, which is a genuinely useful thing for it to learn and something you would otherwise have had to write into the prompt by hand.

What this costs during a compile

The instinct is that a fresh VM per evaluation must be too expensive to do thousands of times. Check it against the bill rather than the instinct. On PandaStack there is no warm pool of idle VMs — every create restores a baked Firecracker snapshot, landing around 179ms p50 and 203ms p99, and a same-host fork is 400-750ms. Billing is per second at $0.054 per active vCPU-hour and $0.0162 per GiB-hour, so a sandbox that spends most of an evaluation waiting on the model bills close to nothing for the waiting.

Now compare that with the other line on the invoice. A compile is, by construction, thousands of model calls carrying long instructions and bootstrapped demonstrations. The tokens dominate, and it is not close. Choosing a weaker execution boundary to save compute during an optimisation run is optimising a rounding error, and the thing you traded away was the trustworthiness of the number the whole exercise exists to produce.

Four shapes, honestly compared

• In-process exec() — Single run: works, and the blast radius is one piece of code you can read afterwards. Compile: thousands of unreviewed programs in your process with your credentials, plus a shared mutable environment silently corrupting the scores.
• One long-lived sandbox, reused — Single run: fine, and genuinely convenient for interactive development. Compile: solves the isolation problem and not the determinism problem — state accumulates across evaluations exactly as it would in-process, just somewhere safer.
• Sandbox per evaluation — Single run: a little overhead you will not notice next to model latency. Compile: isolation plus a clean starting state, parallelises without locks, and costs one create plus per-second compute per evaluation.
• Fork of a prepared snapshot per evaluation — Single run: overkill unless setup is slow. Compile: the strongest option — identical byte-level starting state across every candidate and trial, no per-evaluation installs, and no dependency on the network staying up for the length of the run.

A checklist before you start a compile

• Execution is out of process for anything a compile will touch, which is all of it.
• A fresh environment per evaluation, forked from a prepared snapshot when setup is non-trivial.
• Nothing in the sandbox you would not want printed into an optimiser trace: no API keys, no cloud credentials, no ambient network the task does not need.
• A hard timeout on every execution, enforced by the sandbox rather than by the generated code.
• Failures returned as data — ok=False scores zero; an exception loses the trial.
• Outputs truncated before they reach the model, so one to_string() cannot eat a trial's token budget.
• Teardown in a finally, plus a TTL backstop for the day the compile process dies mid-search.
• The compiled artefact versioned alongside the snapshot id it was compiled against. That pair is your reproducibility contract.

DSPy's core idea — specify the program, let a search find the prompts — is right, and the ergonomics are good enough that it is easy to forget you are running a search at all. The compile step takes every design decision in your program and makes it thousands of times. For most decisions that is fine. For "where does model-written code execute", it is the whole ballgame: the multiplier converts a risk you accepted into a system you are operating, and converts a bit of environmental sloppiness into a number you cannot trust. A disposable machine per evaluation fixes both, and costs less than the tokens you were already going to spend.