Tracing agent runs when the code runs in a sandbox

The trace looks great. The LLM span has the prompt, the token counts and the latency. The retrieval span has the query and the document ids. The tool span in the middle — the one where the agent actually did the thing the user asked for — says: ran code, 4.2 seconds, returned 900 bytes. That is the entire record of the only step that changed anything.

This is the tax on doing the right thing. The moment you move execution out of your process and into an isolated sandbox, which you should, your tracer stops being able to see through the boundary. It can no longer wrap the function call, because there is no function call — there is an HTTP request to a machine your instrumentation has never met. So the trace develops a hole at exactly the point where you most want detail. I build PandaStack, so the sandbox calls below are ours, but the boundary problem is identical for every provider, every self-hosted container runner, and every remote executor you might build yourself.

Two systems, one story, no shared clock

In-process execution gives you observability for free by accident. The code runs under your tracer, on your stack, with your context variables in scope, so anything it does inherits the current span whether you thought about it or not. Out-of-process execution takes all of that away at once. You now have two processes with no shared memory, no shared context, and — importantly — no shared clock you should trust to the millisecond.

The instinct is to try to restore the old world: get the tracer inside the sandbox, have it emit spans, stitch them into the parent trace and pretend nothing changed. You can do that, and there is a right way to do it, but it is not the first thing to do. The first thing to do is admit that the boundary is now the most interesting object in the system, and instrument the boundary itself properly. Most of the debugging value is there, and it costs you nothing in trust, because it is all measured by code you control.

Instrument the boundary before you instrument the guest

Everything in this list is observable from your side, without cooperation from anything running inside the sandbox. That is what makes it trustworthy, and it is also why it should be the layer you get right first.

• Provisioning time, separated from execution time. Did the sandbox take 200 milliseconds to exist or 12 seconds? Folding that into 'tool latency' is how teams end up optimising their prompt to fix a cold-start problem.
• Whether the sandbox was created, restored, forked, or reused. These have wildly different latency profiles, and a p99 that only moves on one of them is a diagnosis, not a mystery.
• Exit code, as an integer. Not a boolean. 0, 1, 2, 124 and 137 mean five different things, and collapsing them into 'failed' throws away the answer before you have asked the question.
• Bytes in and bytes out, plus an explicit truncation flag. Silent truncation is how the one line that mattered disappears between the sandbox and the model.
• Timeout and kill status, distinct from each other and from a non-zero exit. A process the kernel killed for memory pressure looks exactly like a crash unless you go and check.
• Teardown: whether you destroyed the sandbox, whether the TTL reaper did, or whether nobody did. The third case is a bill, and it is invisible unless you record it.
• The sandbox identity and the host it ran on, so the span can later be joined against the platform's own lifecycle events and logs.

Here is the whole of that as one span, using the OpenTelemetry Python API. Nothing exotic: create a span, set attributes on it, and let whatever exporter you have configured — an OTLP collector, a vendor backend, LangSmith's OTel endpoint — pick it up.

# pip install pandastack opentelemetry-api opentelemetry-sdk
import time
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from pandastack import Sandbox

tracer = trace.get_tracer("agent.tools")


def run_in_sandbox(code: str, run_id: str) -> dict:
    with tracer.start_as_current_span("sandbox.exec") as span:
        span.set_attribute("agent.run_id", run_id)
        span.set_attribute("sandbox.template", "code-interpreter")

        t0 = time.perf_counter()
        with Sandbox.create(
            template="code-interpreter",
            ttl_seconds=900,
            metadata={"agent_run_id": run_id},
        ) as sbx:
            provision_ms = (time.perf_counter() - t0) * 1000
            span.set_attribute("sandbox.id", sbx.id)
            span.set_attribute("sandbox.provision_ms", round(provision_ms, 1))

            sbx.filesystem.write("/workspace/main.py", code)

            t1 = time.perf_counter()
            res = sbx.exec("python /workspace/main.py", timeout_seconds=60)
            client_ms = (time.perf_counter() - t1) * 1000

            # Two different numbers on purpose: what the sandbox says it spent
            # running, and what we waited. The gap is transport and scheduling.
            span.set_attribute("sandbox.exec_ms", res.duration_ms)
            span.set_attribute("sandbox.wall_ms", round(client_ms, 1))
            span.set_attribute("sandbox.exit_code", res.exit_code)
            span.set_attribute("sandbox.stdout_bytes", len(res.stdout))
            span.set_attribute("sandbox.stderr_bytes", len(res.stderr))

            if res.exit_code != 0:
                span.set_status(Status(StatusCode.ERROR, "non-zero exit"))

            return {
                "stdout": res.stdout,
                "stderr": res.stderr,
                "exit_code": res.exit_code,
            }

The two duration attributes are the single highest-value thing in that snippet. One is what the platform reports the command actually spent running; the other is what your process waited. When those diverge you are looking at provisioning, queueing or network, and when they do not, you are looking at the agent's own code. Teams who record only one of them spend a lot of time optimising the wrong system.

Getting signal out of the guest

The outside layer tells you that something took four seconds and exited 1. It does not tell you that the code spent 3.8 of those seconds retrying a DNS lookup for a host your egress policy blocks. For that you need the inside, and the inside is where the care is required, because everything it produces is downstream of model-generated — and therefore potentially attacker-influenced — code.

The approach that has held up best for me is a structured sidecar. Have the wrapper script inside the sandbox write a JSON file describing what happened, read that file out over the filesystem API after the run, and treat it as untrusted input that you parse and validate rather than as telemetry you forward.

• Stdout and stderr, tailed rather than whole, with the byte counts recorded separately so you know what you dropped.
• Files created or modified under the workspace, as paths and sizes. This is frequently the actual output of the run, and it is invisible from the outside.
• Outbound network attempts, including the blocked ones. A denied connection is a far more interesting event than a successful one, and it is the cheapest prompt-injection tripwire you will ever install.
• Package installs the agent performed at runtime, which are both a reproducibility problem and a supply-chain event worth a span attribute.
• Peak memory, so an exit code 137 stops being a surprise the second time.
• The wrapper's own view of start and end time, useful for spotting clock weirdness rather than for building the timeline.

Note what is not on that list: a full transcript of everything the code printed. That belongs in a log store, keyed by the sandbox id you already put on the span, not in the trace. More on why in a moment.

Propagate the context, then distrust what comes back

If you do want spans emitted from inside the sandbox — reasonable for long-running builds or multi-stage jobs — you propagate context the same way you would to any other service: inject the W3C trace context into the environment the child process starts with, and let an SDK inside pick it up.

from opentelemetry.propagate import inject

carrier: dict[str, str] = {}
inject(carrier)                      # {"traceparent": "00-<trace>-<span>-01"}

traceparent = carrier.get("traceparent", "")

# Hand it to the guest as an environment variable. An OTel SDK inside the
# sandbox will pick it up and parent its spans under this tool span.
res = sbx.exec(
    f"TRACEPARENT={traceparent} python /workspace/main.py",
    timeout_seconds=60,
)

That is the easy half. The hard half is the return trip, and it is where people get this wrong. Anything the sandbox sends back — a span id, a trace id, a parent reference, a timestamp — was produced inside a process whose entire purpose is to run code you did not write. Treat any of it as authoritative and you have handed a prompt injection the ability to write into your observability system.

That is not a theoretical attack so much as a very boring one. Code inside the sandbox emits spans claiming a parent id belonging to a different customer's trace. Or it emits a thousand spans a second and your ingest bill becomes the incident. Or it emits one span, carefully shaped, that says the run succeeded. None of these require sophistication; they require you to have wired the sandbox's exporter directly at your backend and gone to lunch.

Sandbox output is attacker-influenced data

We spent years learning to escape user input before it reaches SQL, HTML and shells. Then observability arrived and we started piping arbitrary text into log lines and span attributes without a second thought, because logs felt like a write-only medium where nothing could go wrong.

Log injection is real and quietly nasty. Output that contains newlines forges log entries in a line-oriented pipeline. Output containing ANSI escapes rewrites what an engineer sees in their terminal while they are triaging an incident. Output shaped like your own structured log format gets parsed as your own structured log format, complete with a severity field of its choosing. A model that has read an attacker's web page and then printed what it was told to print is a perfectly good delivery mechanism for all three.

The fix is unglamorous and about fifteen lines. Redact known secret shapes, strip control characters, cap the length, and record that you did.

import re

_SECRET_PATTERNS = [
    re.compile(r"sk-[A-Za-z0-9]{16,}"),              # OpenAI-style keys
    re.compile(r"gh[pousr]_[A-Za-z0-9]{20,}"),       # GitHub tokens
    re.compile(r"AKIA[0-9A-Z]{16}"),                 # AWS access key ids
    re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
    re.compile(r"eyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}"),  # JWTs
]

MAX_ATTR_CHARS = 512


def safe_attr(value: str) -> str:
    """Make sandbox output safe to hang off a span attribute."""
    for pattern in _SECRET_PATTERNS:
        value = pattern.sub("[redacted]", value)
    # No newlines, no ANSI escapes, no NUL: nothing that can forge a log
    # record or repaint someone's terminal at 3am.
    value = re.sub(r"[\x00-\x1f\x7f]", " ", value)
    if len(value) > MAX_ATTR_CHARS:
        return value[:MAX_ATTR_CHARS] + "…[truncated]"
    return value


tail = res.stderr[-2000:]
span.set_attribute("sandbox.stderr_tail", safe_attr(tail))
span.set_attribute("sandbox.stderr_truncated", len(res.stderr) > 2000)

Two details worth defending. The tail rather than the head, because the useful part of a failure is almost always at the end. And the explicit truncation flag, because an engineer reading a clipped stderr with no marker will confidently reason about a program that ended somewhere it did not.

Where each signal comes from, and how far to trust it

It helps to keep a mental column for provenance. Anything measured by your process is fact; anything the sandbox reports is a claim.

• Provision latency — Where it comes from: your client, timing the create call. Trust level: full. Nothing inside the sandbox can influence it, which also makes it the cleanest signal you have for platform regressions.
• Exit code — Where it comes from: the platform's exec API, not the guest program. Trust level: full for control-flow decisions. The program chooses the number, so it is not a security signal, but it is an honest report of what the program returned.
• Wall-clock duration — Where it comes from: your client, and separately the platform's own measurement. Trust level: full for both, and the difference between them is the interesting quantity.
• Byte counts and truncation flags — Where it comes from: your client, measuring what it received. Trust level: full. Measure length before you truncate, not after, or the flag lies.
• Stdout and stderr — Where it comes from: the guest process. Trust level: none. Sanitise, redact, cap, and never parse it into structured fields you then act on.
• Spans emitted inside the sandbox — Where it comes from: an SDK running next to model-generated code. Trust level: none. Accept only through your process, re-parented and rate-limited, or not at all.
• Network-attempt records — Where it comes from: the platform's egress policy, if it enforces one; the guest, if it does not. Trust level: full in the first case, none in the second. Know which one you have.
• Resource usage — Where it comes from: the hypervisor or the platform's metering. Trust level: full, and it is what your invoice is computed from anyway.

Traces are where secrets go to be archived forever

A trace backend is the one datastore in your architecture that nobody threat-models. It is append-only, it is retained for months, it is usually SaaS, and — this is the part that should worry you — read access to it is typically granted to anyone on call, which over a couple of years means everyone who has ever worked there.

Agent traces are unusually rich in things you would not choose to archive. Prompts contain customer data by construction. Tool arguments contain whatever the model decided to pass along, which includes credentials surprisingly often. And sandbox stderr is a firehose of environment dumps, connection strings in exception messages, and the occasional full traceback with an API key sitting in a local variable.

• Redact on the way out, in your instrumentation layer, not on the way in at the backend. Vendor-side masking is a fine second layer and a terrible first one, because by the time it runs the data has already crossed a network boundary you do not own.
• Never forward your process environment into the sandbox wholesale. It is the single largest source of secrets ending up in traces, and it fails safe by default if you simply do not do it.
• Set retention deliberately. Ask how far back anyone has actually looked when debugging. The answer is usually days, and the default is usually forever.
• Treat trace access as customer-data access in your access model, because that is what it is, and because it is a finding waiting to happen at your next audit.

Do not put an agent's stdout in a span attribute

The temptation, once the redaction is in place, is to attach everything. Full stdout, full stderr, the generated source, the file listing. It works in development, where a run produces four kilobytes, and it becomes a problem in production, where an agent calls a method that prints a large dataframe and produces four megabytes on a Tuesday afternoon.

Two separate failure modes hide in there. Payload size is a cost problem: trace backends bill on ingest volume, and per-run payloads are the fastest way to make observability spend outrun the model spend it was supposed to explain. Cardinality is a query problem: an attribute whose value is unique per run — a sandbox id, a generated command, a hash — is fine as a span attribute you filter by occasionally, and catastrophic the moment it becomes a metric dimension. That is how a dashboard starts timing out and nobody can say why.

• Bounded, low-cardinality values on spans: template name, exit code, truncation flags, size buckets, error class.
• High-cardinality identifiers on spans as attributes you can filter by — sandbox id, run id — but never as metric labels.
• Large payloads in a log or object store, referenced from the span by id. The span says where the bytes are; it does not carry them.
• Sample deliberately: keep every errored run, sample the successful ones. Full fidelity for every green run is a cost decision people make by accident.

Where LangSmith fits, and where plain OTel does

Everything above is deliberately vendor-neutral, because the boundary problem is the same everywhere and because SDK surfaces move faster than blog posts. The practical split, as it stands today, is that LLM-native tools like LangSmith are shaped around the agent run — a tree of prompts, tool calls and outputs you can read, annotate, and turn into a test case — while OpenTelemetry is shaped around the request path, and puts your agent in the same trace as the API gateway, the database and the deploy that broke it.

You usually want both views, and the good news is that they increasingly meet in the middle: the major LLM observability vendors now accept OTLP, and the semantic conventions for generative AI are converging, however slowly. That means the pragmatic architecture is to emit standard OTel spans from your own code, add the LLM-specific instrumentation your framework provides on top, and fan out at the collector rather than instrumenting twice. Check the current ingest endpoints and attribute names against your vendor's own documentation before you build on them — this is the fastest-moving corner of the ecosystem and any specific field name I print here has a decent chance of being wrong by the time you read it.

What does not move is the shape of the problem. Whichever backend you choose, the sandbox boundary is a trust boundary, and your instrumentation has to know that.

The short version

1. Instrument the boundary from your side first: provision time, exec time, exit code, byte counts, timeout and kill status, teardown.
2. Record two durations — what the platform says it spent and what you waited — because the gap between them is a different bug from either one.
3. Propagate trace context into the sandbox as an environment variable. That direction is safe.
4. Never let the sandbox export spans directly to your backend. Read them out, re-parent them under a span you own, validate and cap them.
5. Sanitise every byte that comes back: strip control characters, redact secret shapes, truncate, and flag the truncation.
6. Keep payloads out of spans. Attach identifiers and sizes; put the bytes in a log store and reference them.
7. Redact before export, not at the vendor, and set retention like the data is customer data — because it is.

None of this is difficult. It is just work that nobody schedules, because the trace looks fine right up until the afternoon you need it, and then the one span that would have told you what happened says: ran code, 4.2s, 900 bytes.