The best AI agent observability and tracing platforms in 2026

Here is a trace I have looked at more times than I would like. A parent span for the agent run. A child span for the model call, with the prompt and the completion rendered nicely. A child span named run_code, duration 8.4 seconds, status ERROR. And on that span, one attribute: exit 1.

That is not observability. That is a receipt for a failure. The model wrote some code, something ran it, the something said no, and the trace has preserved exactly none of the information that would tell you why. No stderr. No stdout. No wall time separated from queue time. No idea whether the process was killed by the OOM killer or exited on its own. The one span in the whole trace that describes contact with the real world is the one span carrying no data about it.

I'm Ajay, I build PandaStack. Conflict of interest declared up front and then set aside: we are not an observability vendor, we do not sell a trace viewer or an eval harness, and nothing in this post is trying to sell you one. We run the Firecracker microVMs that agent tool calls execute inside. That gives me a specific and slightly unusual vantage point — I spend a lot of time with people whose traces are beautiful right up to the moment their agent touches a machine, at which point the trace goes dark.

So this is two things. First, an honest comparison of the platforms in this category and what each one is genuinely optimised for, because they are much less interchangeable than the category name suggests. Second, the part almost every roundup skips: how to instrument the execution side so the interesting spans stop being empty.

Agent observability is three jobs sold as one product

The single most useful thing to understand before evaluating anything here is that the phrase covers three distinct jobs. Teams buy one product hoping for all three, then discover six weeks in that their chosen tool is excellent at one, adequate at the second and a nuisance for the third.

• Tracing. Reconstructing a single run: what the model saw, what it decided, which tools it called with what arguments, what came back, and where the seconds went. This is a debugging job. You reach for it when something specific went wrong and you need to see the run as it happened.
• Evaluation and scoring. Answering whether a change made things better. Datasets of inputs, scorers that grade outputs, and a way to compare run A against run B without squinting. This is an experimentation job, and it is the one that decides whether you can ship prompt changes with any confidence at all.
• Cost and latency accounting. Tokens, dollars and seconds attributed to a user, a tenant, a feature or an API key. This is a finance and capacity job, and it is usually the reason someone outside engineering asks about observability in the first place.

There is a fourth that sneaks up on teams after about a quarter: human annotation. Somebody has to look at a hundred production traces and mark which ones were bad, and those labels become the dataset the eval job runs against. Tools differ enormously in whether that is a first-class workflow with a queue and a reviewer role, or a CSV export and a spreadsheet.

The reason this framing matters is that it converts a vague comparison into a short question: which of the three is my actual bottleneck right now? If you cannot debug a bad run, you want the trace viewer. If you cannot tell whether last week's prompt change helped, you want the eval harness. If your model bill tripled and nobody knows which feature did it, you want the accounting layer, and you probably want it as a gateway rather than as an SDK you have to thread through forty call sites.

Nobody has all three problems equally badly at the same time. Pick for the one that is hurting, and check that the other two are at least possible.

The trace layer is quietly becoming portable

For a couple of years every vendor in this space had its own SDK, its own span shape and its own idea of what a generation was. Adopting one meant writing integration code you could not reuse. That is changing, and it is the most consequential thing to happen to this category.

OpenTelemetry has a GenAI semantic conventions effort defining standard attribute names and span shapes for model calls, tool calls and agent runs — things like the operation name, the system you called, the request model, and token usage split into input and output. Several of the platforms below ingest OTLP directly, and several of the instrumentation libraries emit those attributes. Arize's Phoenix is built on OTel from the ground up. Langfuse accepts OTLP. The practical upshot is that the trace layer is starting to look like a commodity, in the good sense: you emit standard spans, and where they land becomes a deployment decision rather than a rewrite.

# A model call as an OTel span, using GenAI-flavoured attribute names.
# This is deliberately hand-rolled -- most people will use an
# auto-instrumentation package -- but writing it once shows you exactly
# what your backend is going to receive, which is worth ten minutes.
from opentelemetry import trace

tracer = trace.get_tracer("agent.llm")

def chat(client, messages, *, tenant: str, feature: str):
    model = "claude-sonnet-5"
    with tracer.start_as_current_span("chat " + model) as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.system", "anthropic")
        span.set_attribute("gen_ai.request.model", model)
        span.set_attribute("gen_ai.request.max_tokens", 4096)

        # Your OWN dimensions. These are what make the cost job answerable
        # later, and no vendor can add them for you after the fact.
        span.set_attribute("app.tenant_id", tenant)
        span.set_attribute("app.feature", feature)

        resp = client.messages.create(model=model, max_tokens=4096,
                                      messages=messages)

        span.set_attribute("gen_ai.response.model", resp.model)
        span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)
        span.set_attribute("gen_ai.response.finish_reasons",
                           [resp.stop_reason or "unknown"])
        return resp

That gives you the real lock-in question for this category, and it is a better question than the ones on most comparison pages. Not is this vendor open source. Not can I self-host. The question is: if I want to leave in eighteen months, are my traces in a format something else can read, and can I get them out?

The answers vary. A platform that ingests OTLP and stores something close to standard spans is one export away from portable. A platform whose value is a proprietary eval-experiment model with its own scoring history is much stickier — not maliciously, just because the thing you would be moving does not have a standard shape yet. Evals are where lock-in genuinely lives in 2026, and it is worth being deliberate about that rather than surprised by it.

The options, qualitatively

No prices, no feature matrices and no benchmark numbers below. This category is moving faster than any other part of the AI stack — pricing models, free tiers, retention windows and self-hosting terms have all changed within the last year for several of these — and any specific number I wrote today would be wrong by the time you read it. What follows is shape and centre of gravity, which move slowly. Verify everything else against current docs before you commit.

1. Langfuse

The open-source default, and the one most often chosen by teams whose first requirement is owning the data. Core is open source and genuinely self-hostable, there is a managed cloud, and the product is trace-viewer-first: sessions, nested observations, prompt management with versioning, scores attached to traces, datasets, and evaluation on top. It ingests OTLP as well as its own SDKs, which matters for the portability question above.

Centre of gravity: job one, with a credible job two. If your bottleneck is that you cannot see what happened inside a run, and you have a policy reason to keep prompts and completions on your own infrastructure, this is the obvious first stop.

The honest catch is operational, and it is the thing people underestimate. Self-hosting Langfuse is not one container. Current versions want Postgres for transactional data, ClickHouse for the trace volume, a Redis-compatible cache, and S3-compatible blob storage. That is a real stack. I wrote a whole separate post about where to put it, linked at the bottom, because the hosting decision turns out to be an entirely different problem from the choosing decision.

2. LangSmith

From the LangChain team, and the deepest integration story in the category if you are already on LangChain or LangGraph — traces show up with essentially no instrumentation work, with the framework's own abstractions rendered as first-class objects rather than as generic spans. It is not LangChain-only; the SDK traces arbitrary Python and TypeScript. But the fit is noticeably tighter inside that ecosystem, which is both the selling point and the thing to think about.

Centre of gravity: jobs one and two together, with a strong prompt-iteration loop. Trace debugging, datasets built from production traces, evaluators, a playground for comparing prompt variants, and human annotation queues that are a real workflow rather than an afterthought. If your team's day looks like read a bad trace, turn it into a test case, change the prompt, check the score, that loop is what this is built around.

It is SaaS-first. Self-hosted deployment exists but sits under enterprise terms rather than being the default path, which is a different posture from Langfuse or Phoenix. If you have a hard data-residency constraint, start that conversation early rather than assuming.

3. Braintrust

The eval-first one, and the clearest example of why this category needs disambiguating. Braintrust does log and trace, but the product is organised around the experiment: datasets, scorers, and side-by-side comparison of run against run so you can see which examples got better and which regressed when you changed something. The playground and the scoring model are the centre, not the trace list.

Centre of gravity: job two, hard. Pick it when your actual pain is that nobody can tell whether the change you shipped last Tuesday made the product better or worse, and arguments about prompt quality are being settled by whoever is most confident in the room. That is a real and expensive failure mode, and a trace viewer does not fix it.

The trade is the one I flagged earlier: an eval platform accumulates history — datasets, scorer definitions, months of experiment results — that has no standard interchange format. That is not a criticism of the product, it is a property of the problem. Just know you are putting down roots there, and ask about export before you have two years of experiments in it.

4. Helicone

The gateway. Instead of adding an SDK and instrumenting call sites, you change your provider base URL to point at Helicone, and requests flow through it — logged, attributed, cached, rate-limited and costed on the way past. It is open source and self-hostable, and there are async logging paths if you do not want a proxy in the request path.

Centre of gravity: job three, with a fast path to job one. Nothing else in this list gets you from zero to per-user, per-key cost attribution as quickly, because there is no instrumentation work to do — the integration is a config change. For a team whose problem is that the model bill went up and nobody can decompose it, that is exactly right.

Two things to weigh. A proxy in the request path is a real dependency with real latency and real failure semantics, so read carefully about what happens when it is unavailable and decide whether you are comfortable — the async path exists for exactly this reason. And a gateway sees requests, which is a slightly different unit from a multi-step agent run; it has session grouping, but the tool-first products model an agent trace more natively. Verify the current shape of both against their docs; this product has moved a lot.

5. Arize Phoenix

The OpenTelemetry-native one, and my pick for teams who philosophically object to a new SaaS account for something that is fundamentally tracing. Phoenix is open source, runs locally — including inside a notebook — as well as as a server, and is built on OpenInference, an OTel-compatible convention for LLM spans. It does tracing, evals and datasets, and it will happily be the thing you run on a laptop while developing.

Centre of gravity: job one with OTel purity, and a good job two. Behind it sits Arize's commercial platform, which comes from the older ML-observability world — drift, data quality, production monitoring at scale — so the upgrade path leans toward organisations with an ML platform team rather than toward application developers.

The local-first story is genuinely underrated. Being able to run the trace viewer next to the agent you are developing, with no account and no egress of prompts to anywhere, changes how often you look at traces during development. Tools you have to log into get looked at after the incident. Tools running on localhost get looked at during the bug.

6. W&B Weave

From Weights and Biases, and the continuity play. If your organisation already runs W&B for training and fine-tuning, Weave puts LLM application traces and evaluations in the same place as your model runs, under the same accounts and permissions. Decorate a function, get a trace; build evaluations against datasets; compare.

Centre of gravity: jobs one and two, viewed through an experiment-tracking lens rather than an APM one. The mental model is inherited from ML experimentation, which is a real advantage when the people debugging the agent are the same people who fine-tuned the model it calls, and a mild impedance mismatch when they are backend engineers who think in traces and services.

If nobody at your company uses W&B today, this is unlikely to be the one that wins on its own merits for a pure application-engineering team. If half your company already lives there, the integration argument is strong and you should weigh it heavily.

7. Plain OpenTelemetry into the stack you already run

The baseline everything above should have to beat, and the option that gets dismissed too quickly. Emit GenAI-flavoured spans from your app, send them via OTLP to whatever you already operate — Tempo, Jaeger, SigNoz, Honeycomb, Datadog, your existing collector — and your agent traces sit next to your HTTP handlers, your database queries and your queue workers, in one tool, with one on-call rotation and no new vendor review.

That last part is the strongest argument and it is not a technical one. An agent run is rarely just model calls. It is an HTTP handler that authenticated a user, three database queries, a queue hop, four model calls and two sandboxed executions. A dedicated LLM tool shows you the middle. Your existing tracing backend shows you all of it, including the 900ms your own authorisation check spent on a cold cache — which, in my experience, is a surprisingly common answer to why is the agent slow.

What you give up is the LLM-shaped interface. A generic span viewer renders a prompt as a long string attribute rather than as a conversation. There is no dataset, no scorer, no experiment comparison, no annotation queue. Several of the general APM vendors have shipped LLM-specific views recently and the gap is closing — check what yours has before assuming — but the eval half is still mostly absent, and eval is the half that stops you shipping regressions.

A pattern I like and see working: OTel to your normal backend for everything, plus a dedicated eval tool for job two. Two systems, but each doing the job it is good at, and the trace half stays portable.

Side by side

• Langfuse — Optimised for: tracing, with solid evals. Shape: open-source core plus cloud, OTLP ingest, prompt management, self-hostable if you accept Postgres plus ClickHouse plus a cache plus blob storage. Best for: teams whose first requirement is owning the trace data.
• LangSmith — Optimised for: tracing plus the prompt-iteration and annotation loop. Shape: SaaS-first, framework-agnostic SDK but noticeably deeper if you are on LangChain or LangGraph. Best for: teams whose day is read a bad trace, make it a test case, change the prompt, check the score.
• Braintrust — Optimised for: evaluation and experiment comparison. Shape: datasets, scorers, side-by-side run diffing, playground; logging present but not the centre. Best for: teams who cannot currently tell whether a change helped.
• Helicone — Optimised for: cost and usage accounting, fastest possible integration. Shape: a gateway you point your base URL at, open source and self-hostable, with async logging if you do not want a proxy in the path. Best for: the bill went up and nobody can decompose it.
• Arize Phoenix — Optimised for: OTel-native tracing you can run locally. Shape: open source, OpenInference conventions, notebook or server, evals and datasets included, commercial Arize platform behind it. Best for: teams who want traces without a SaaS account, and ML-platform organisations.
• W&B Weave — Optimised for: tracing and evals inside an existing experiment-tracking world. Shape: decorator-based tracing, evaluations, same home as your training runs. Best for: organisations already standardised on Weights and Biases.
• Plain OpenTelemetry — Optimised for: seeing the agent in the context of everything else it touches. Shape: standard spans into the backend you already run; no eval harness, no LLM-shaped UI, no new vendor. Best for: teams with a working observability stack and discipline, especially paired with a separate eval tool.
• PandaStack — Optimised for: none of the above, deliberately. We are the Firecracker microVM the tool calls execute inside, and our job in this story is to make the execution spans contain something. Pair us with one of the seven above; we do not replace them.

The self-hosting question is really a data question

Self-hosting comes up early in every one of these evaluations and is usually framed as a preference. It is not a preference. It is a consequence of one fact people notice late: your traces contain the full text of every prompt and every completion.

Think about what that means concretely. If your agent reads support tickets, your trace store contains support tickets. If it summarises medical documents, your trace store contains medical documents. If a customer pastes an API key into a chat box — and they will — your trace store contains their API key, indefinitely, searchable, visible to everyone with a login to your observability tool. This is the single most commonly underestimated fact in the category, and it converts a tooling decision into a data-processing decision with contractual consequences.

So the ordering is: work out what your traces will contain, then decide where they may live, then choose a product from the ones that can live there. Doing it in the other order is how teams end up ripping out a tool they liked.

• Redact before you export, not after. A masking hook in your instrumentation layer that strips known secret shapes and PII fields costs an afternoon. Deleting leaked data out of a third-party trace store costs a support ticket and a bad week.
• Set retention deliberately and early. The default instinct is to keep everything forever. The useful question is how far back anyone has actually looked — most debugging happens within days, most analysis is aggregate over months, and payload-heavy trace data is the most expensive kind of data to hoard.
• Sample production, keep everything in staging. Full-fidelity traces for every production run is a cost decision disguised as a fidelity decision. Tail-based sampling that keeps every errored run and a fraction of successful ones is usually the right default.
• Decide who can read prompts. Trace access is customer-data access. If your access model does not treat it that way, it is wrong, and it is a finding waiting to happen in your next audit.

The part most roundups skip: the tool call is where agents actually fail

Now the thing that makes this a post from a compute company rather than a rewrite of a comparison page.

Read a hundred failed agent runs and count the causes. A few are genuine reasoning failures where the model chose a stupid plan. A few are prompt problems. The overwhelming majority are the model calling a tool and the tool doing something the model did not expect: the command failed, the package was not installed, the file was not where it thought, the network call was blocked, the process was killed for using too much memory, the thing that worked in the last run does not work in this one because the previous run left a file behind.

Every platform above will faithfully record that the tool was called and that it returned something. Almost none of them can tell you what happened inside it, because that is not their job — it is yours, and it lives in your instrumentation of the execution boundary. A trace whose tool span says run_code, ERROR, 8.4s is technically a trace and practically a shrug.

Here is what a tool span should carry when the tool is code execution. None of this is exotic; it is just work nobody assigns.

• The exact command, and the bytes of source if the model wrote it. Not a summary — the literal thing that ran, because the difference between the intended command and the executed one is a whole genre of bug.
• The exit code as an integer, not a boolean success flag. 0, 1, 2, 124 and 137 mean entirely different things and collapsing them into failed throws away your diagnosis.
• Both output streams, truncated on purpose, with a flag recording that they were truncated. A silently clipped stderr is how you lose the one line that mattered.
• Wall time as reported by the thing that ran the command, separate from the wall time your client observed. The gap between those two is provisioning, queueing and network, and confusing it with execution time sends people optimising the wrong thing.
• Kill and OOM status. A process killed by the kernel for memory pressure looks identical to a crash unless you go and check, and it is the single most misdiagnosed agent failure I see.
• The identity of the machine it ran on, so the span can be joined back to the environment's own logs and lifecycle events later.

This is the code. It is a real tool implementation against our Python SDK, with OTel spans around it, and the structure transfers to any execution backend you use.

# tools/run_code.py -- the tool the model calls, properly instrumented.
#
# The test for this span: can somebody reading it six hours from now, with
# the machine long gone, say what ran, what came back and why it stopped?
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
from pandastack import Sandbox

tracer = trace.get_tracer("agent.tools")

MAX_CAPTURE = 4000  # bytes of each stream we attach to the span


def run_code(source: str, session_id: str) -> dict:
    with tracer.start_as_current_span("execute_tool run_code") as span:
        trace_id = format(span.get_span_context().trace_id, "032x")

        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", "run_code")
        span.set_attribute("gen_ai.tool.type", "sandbox")
        span.set_attribute("code.source.bytes", len(source.encode()))

        # Snapshot-restore create: a fresh microVM with its own guest kernel,
        # p50 around 179ms, so per-call isolation is affordable. The trace id
        # rides along in metadata -- that is the join key later.
        sbx = Sandbox.create(
            template="code-interpreter",
            ttl_seconds=300,
            metadata={"trace_id": trace_id, "session_id": session_id},
        )
        span.set_attribute("sandbox.id", sbx.id)
        span.add_event("sandbox_created", {"sandbox.id": sbx.id})

        try:
            sbx.filesystem.write("/work/step.py", source)
            r = sbx.exec("python /work/step.py", timeout_seconds=120)

            # The three numbers that make the span diagnosable.
            span.set_attribute("process.exit_code", r.exit_code)
            span.set_attribute("process.duration_ms", r.duration_ms)
            span.set_attribute("process.stdout_bytes", len(r.stdout))
            span.set_attribute("process.stderr_bytes", len(r.stderr))

            # Truncate on purpose, keep the END of stderr (that is where the
            # traceback is) and the START of stdout, and record that you cut.
            span.set_attribute("process.stdout", r.stdout[:MAX_CAPTURE])
            span.set_attribute("process.stderr", r.stderr[-MAX_CAPTURE:])
            span.set_attribute(
                "process.output_truncated",
                len(r.stdout) > MAX_CAPTURE or len(r.stderr) > MAX_CAPTURE,
            )

            if r.exit_code != 0:
                span.set_status(Status(StatusCode.ERROR,
                                       "exit code " + str(r.exit_code)))
                _classify_failure(span, sbx, r)

            return {"stdout": r.stdout, "stderr": r.stderr,
                    "exit_code": r.exit_code}

        except Exception as exc:
            # A failure to RUN the tool is a different thing from the tool
            # failing, and the span should say which one happened.
            span.set_attribute("tool.harness_error", True)
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, str(exc)))
            raise
        finally:
            sbx.kill()

Three details in there are load-bearing and worth arguing about.

The trace id goes into the sandbox's metadata at create time, not into a log line afterwards. That means the join works from both directions: from a span you can find the machine, and from a machine you can find the span. If you only log the sandbox id onto the span, you can go one way, and the day you need the other way is the day you have a runaway VM and no idea which user request produced it.

Truncation keeps the end of stderr and the start of stdout. Tracebacks are at the end. Progress output is at the start. Taking the first 4KB of both is the default and it is wrong for one of them.

The harness_error attribute separates the tool failed from we failed to run the tool. Those look the same in a status code and mean completely different things — one is the agent's problem, one is yours — and the number of dashboards that conflate them is high.

Ask the kernel before you throw the machine away

Exit code 137 is 128 plus 9, which means the process received SIGKILL. In a sandbox with its own guest kernel that is very often the OOM killer, and it is the failure people spend the longest misdiagnosing because from the agent's point of view the tool simply stopped talking. The model then retries, gets killed again, and burns tokens narrating its confusion.

You can confirm it, but only while the machine still exists. This is the classifier the previous snippet called, and the ordering matters: read the diagnosis, then tear down.

def _classify_failure(span, sbx, r) -> None:
    """Turn a non-zero exit into something a human can act on.

    Everything here has to happen BEFORE the sandbox is destroyed. The
    finally block that kills the VM is also the finally block that destroys
    your evidence, which is a trade people make by accident exactly once.
    """
    code = r.exit_code

    if code == 124:
        # GNU timeout's convention, and what most runners return on deadline.
        span.set_attribute("process.timed_out", True)
        return

    if code == 137:
        span.set_attribute("process.killed_signal", "SIGKILL")
        # Ask the guest kernel instead of guessing. One extra exec.
        oom = sbx.exec(
            "dmesg 2>/dev/null | grep -i -m1 'killed process' || true"
        )
        line = oom.stdout.strip()
        span.set_attribute("process.oom_killed", bool(line))
        if line:
            span.set_attribute("process.oom_kernel_line", line[:400])
            span.add_event("oom_killed", {"kernel.line": line[:400]})
        return

    if code == 127:
        # Command not found. Nearly always a missing dependency in the
        # template, which is a platform bug wearing an agent bug's coat.
        span.set_attribute("process.missing_command", True)

Long tools: stream the output, summarise it into the span

Buffering the entire output of a ten-minute build and attaching it to a span at the end is bad in three ways: you see nothing while it runs, you lose everything if it dies, and you will blow past your exporter's attribute size limits and have the whole span silently dropped. Stream instead, ship the lines to your log pipeline, and keep only a bounded tail for the span.

import collections
from opentelemetry import trace

tracer = trace.get_tracer("agent.tools")


def run_build(sbx, cmd: str, log_sink) -> int:
    with tracer.start_as_current_span("execute_tool run_build") as span:
        span.set_attribute("gen_ai.tool.name", "run_build")
        span.set_attribute("process.command", cmd)

        trace_id = format(span.get_span_context().trace_id, "032x")
        tail = collections.deque(maxlen=200)

        def on_line(chunk: str) -> None:
            tail.append(chunk)
            # Every line goes to the log pipeline stamped with the trace id,
            # so the full output is queryable there. Do NOT make each line a
            # span event -- a chatty build will exceed the event limit and
            # you will lose the span, not just the extra events.
            log_sink.write({"trace_id": trace_id, "sandbox_id": sbx.id,
                            "line": chunk})

        exit_code = sbx.exec_stream(
            cmd, on_stdout=on_line, on_stderr=on_line, timeout_seconds=900,
        )

        span.set_attribute("process.exit_code", exit_code)
        span.set_attribute("process.output_tail", "".join(tail)[-4000:])
        return exit_code

Joining a trace to a machine, from either end

Once the trace id is in the sandbox metadata, the join works both ways and costs nothing. From a trace in your observability tool, find the machine:

# Trace id from the span -> the sandbox that ran the tool call.
pandastack sandbox list \
  | jq -r '.[] | select(.metadata.trace_id == "4bf92f3577b34da6a3ce929d0e0e4736")
           | {id, template, status, created_at}'

# And the host-side log for that machine, which is a different stream from
# your application's stdout -- this is the VMM's own view.
pandastack sandbox logs sbx_01hq... --stream both

# Going the other way: you found a sandbox burning CPU and want the request
# that created it. The metadata is right there on the object.
pandastack sandbox get sbx_01hq... | jq '.metadata'

Do the same for whatever you run. The general rule is that any resource an agent creates should carry the trace id of the run that created it, in whatever tagging mechanism the resource has, at creation time. Sandboxes, temporary buckets, database branches, queue messages. It is a one-line change per resource and it is the difference between an investigation and an archaeology dig.

Why per-run isolation is an observability feature

This is the part I care most about and it is rarely framed as an observability concern at all.

If your agent's tool calls run in a shared, long-lived environment — one container that serves many runs, a persistent worker with a working directory, a notebook kernel that stays warm — then the logs from any given run are not a description of that run. They are a description of that run plus whatever every previous run left behind. A pip install from an hour ago. A file at the path the model expected to be empty. An environment variable someone's earlier code exported. A background process still holding a port.

The symptom is the worst kind of bug report: this trace shows it failing, but when I replay it, it passes. Which is not a mystery — the replay ran in a different accumulated state than the original did. The trace was never a complete description of the conditions, so it was never replayable, so it was never really evidence.

Per-run isolation fixes that, and it fixes it in a way that is invisible until you compare. When every tool call gets a fresh machine restored from the same known snapshot, the state at the start of the run is a constant. The logs then genuinely describe the run. Replaying reproduces. A regression in your eval suite is a regression in your agent, not an artefact of which task happened to run before it on the same box.

The reason people do not do this is that historically it was too slow to spend a container boot on every tool call. Our whole architecture is a bet against that constraint: there is no warm pool, every create restores a baked snapshot on demand, and p50 create latency is around 179 milliseconds with p99 around 203. Forking a running sandbox — copy-on-write memory plus a reflinked disk — is 400 to 750 milliseconds on the same host. At those numbers, per-run isolation stops being a purity argument and becomes the default you would pick anyway.

The eval consequence is direct. If your scores move because task 41 poisoned the environment for task 42, your eval harness is measuring your infrastructure. I wrote that up separately for the eval-harness case; it is linked at the end.

Cost accounting: tokens are not the whole bill

Job three deserves a warning of its own, because every tool in this category measures the same slice of the cost and none of them measures the rest.

They all count tokens and multiply by a rate. That is genuinely useful and it is a real part of the bill. But for an agent that executes code, the model spend is frequently not the largest line. A run that calls the model four times for a few thousand tokens and then spends nine minutes of CPU installing dependencies and running a test suite has a compute cost that dwarfs its token cost, and every LLM observability tool on the market will report that run as cheap.

• Compute seconds per run, from the execution layer. Sum your tool span durations per trace and you have it approximately; take it from your compute platform's own metering and you have it properly.
• Egress. An agent that downloads a large model file or scrapes a site pays for bytes, and nothing in your trace viewer knows that number.
• Storage that outlives the run. Snapshots, artefacts, uploaded files. This is the cost that accumulates silently because no single run looks expensive.
• Retries. A run that failed and retried three times cost four times as much, and if your retry loop creates a new trace each time, your per-run cost dashboard is quietly lying by a factor of four.

The fix is unglamorous: put your own cost attributes on your own spans, using the same tenant and feature dimensions you put on the model spans, and let the sum be the answer. Whichever platform you pick, it can only aggregate the attributes you gave it. The dimension you did not record at 3pm on Tuesday is not recoverable on Wednesday.

Where PandaStack is the wrong answer

To be completely unambiguous: PandaStack is not an observability platform and I am not proposing it as an alternative to anything in the list above. We do not have a trace viewer. We do not have an eval harness. We do not have datasets, scorers, annotation queues, prompt management or a playground. If you came looking for a place to put your agent traces, the answer is one of the seven options above, not us.

What we are is the runtime the tool calls execute in — Firecracker microVMs with their own guest kernel, created by snapshot restore, forkable, and costing nothing when idle. Our contribution to your observability story is narrow and specific: making the execution span contain real data, and making per-run isolation cheap enough that the data means something. That is it. You still need a platform for the other three jobs.

There are also places where we are the wrong compute answer, which is worth saying in the same breath.

• If your agent's tools are all API calls — search, retrieval, a CRM write — there is no code execution, no isolation boundary to buy, and this entire section is irrelevant to you. Instrument your HTTP client and go home.
• If you want to self-host Langfuse on us specifically, be aware we do not offer managed ClickHouse today. We will run the app tier and the Postgres dependency comfortably; the analytical store is yours to bring.
• If your execution workload is one steady long-lived process rather than many short bursts, scale-to-zero and snapshot-restore buy you nothing and you should use a boring always-on VM.
• If you need GPUs in the sandbox, we do not do that.

Choosing, in about ten minutes

1. Name which of the three jobs is actually hurting today. Cannot debug a run, cannot tell whether a change helped, or cannot decompose the bill. Write it down before you look at a single landing page, because every one of them is designed to convince you that you need all three.
2. Work out what your traces will contain and where that data is allowed to live. This eliminates candidates faster than any feature comparison, and it is the constraint people discover last.
3. Check whether you can emit OTLP. If you can, the trace half of your decision becomes reversible, which lowers the stakes on everything else.
4. If you already have a working tracing backend, seriously price the plain-OTel baseline plus a dedicated eval tool. One pane for the whole request path is worth more than most people credit, and the LLM-specific views in general APM tools have improved a lot.
5. Instrument the tool calls before you finish the evaluation. Do it against whichever candidate you are trialling. A trace viewer evaluated with empty execution spans will make every candidate look equally fine, because you will be comparing the rendering of the part that already works.
6. Check that per-run state is actually isolated. If your tool calls share an environment, fix that before you trust any number the eval half produces — otherwise you are measuring your infrastructure's memory, not your agent's quality.

The short version

Langfuse if you want to own the data and can live with its dependency stack. LangSmith if you are in the LangChain ecosystem or want the tightest read-a-trace-to-a-test-case loop. Braintrust if your bottleneck is that nobody can prove a change helped. Helicone if the bill is the problem and you want it solved this afternoon. Phoenix if you want OTel-native traces running on localhost with no account. Weave if your company already lives in Weights and Biases. And plain OpenTelemetry into your existing backend if you have one and enough discipline to add an eval tool beside it.

Then, whichever you pick, go and look at your tool spans. If the span for the step where your agent touched a real machine says nothing but a status and a duration, you have bought a very good viewer for the half of the problem you did not have. The model's reasoning is usually fine. It is the shell command that failed, and nobody wrote down why.