all posts

Audit Trails When the Machine Is Gone in 200 Milliseconds

Ajay Kumar··11 min read

A sandbox that lived for eleven seconds is an excellent security property and a terrible witness. Everything that made it safe — it was isolated, it was disposable, it is now unrecoverable bytes on a host you would struggle to name — is exactly what makes the question "what did the agent actually do at 03:14?" unanswerable. The machine is not under investigation. The machine does not exist.

That question gets asked. By a customer who found a table truncated and wants to know which of your agent's tool calls did it. In your own incident channel at 03:40, by someone who needs to know whether the thing that egressed two gigabytes was a model mistake or a person. By a SOC 2 auditor, who is not testing whether your system is secure and is testing, intensely, whether you can show a control operated every single time. And eventually, if you are successful enough to be worth suing, by a lawyer, in writing, with a date range.

I'm Ajay; I build PandaStack, which runs code in Firecracker microVMs where a snapshot-restore create lands at a p50 of 179 ms. Short-lived by design, down to the create path. This post is the uncomfortable half of that design: what you must emit before the machine dies, where the record has to be born, and what it costs. There is one genuinely good idea here and I will put it up front so you can stop reading early: on a security trigger, snapshot the sandbox before you kill it, and treat the snapshot as the evidence bag.

The record has to be born outside the blast radius

Start with the thing that disqualifies most of what teams build first. A log written inside a guest the agent controls is not evidence. It is testimony from the subject of the investigation — it can be edited, truncated, backdated or simply not written, and the code with the means to do all three is the code you are trying to characterise. This is the same reason you do not let an application grade its own audit log. In-guest `auditd`, in-guest eBPF, a logging shim in the interpreter: useful colour, never the record of account.

The second disqualifier kills more designs: shipping. Suppose your in-guest collector forwards faithfully. A line emitted but not yet shipped when the VM is killed is simply gone. There is no flush — a microVM teardown is not a graceful shutdown with a chance to drain buffers. So every in-guest log has a missing tail of unknown length, and it goes missing precisely when things went wrong, because "things went wrong" and "the VM was killed abruptly" are the same event.

So the shape falls out on its own. The record of account is born in your control plane — the process that decided to create the sandbox, decided which command to run, and holds the identity of whoever asked. That process survives the machine by construction. The guest's own output is supporting material: captured where you can, never load-bearing.

One consequence inverts how people usually write logging code. Record intent before execution and outcome after, as two events linked by hash. An event written only after the fact cannot describe a command that killed its own recorder, and "the log just stops" is the most common shape of a real incident. Two events per action is the difference between knowing what was attempted and knowing only what succeeded.

The minimum useful event set

Here is the list I would hold a design review against — roughly the smallest set from which you can answer a real question without a phone call.

  • Who or what initiated it, resolved to three things rather than one: the human principal, the agent run id, and the specific tool-call id. "The agent did it" is not an actor; it is a shrug with a UUID.
  • The resolved identity and tenant, written at emit time from the same resolution your authorisation check used. If your ban check and your audit log resolve the tenant by different predicates, one of them is lying.
  • Provenance: template, snapshot, and parent. On a snapshot platform this is a tree, not a field — a forked sandbox whose record does not name its parent is unusable, because the real question is "which ancestor introduced this?"
  • The command text, redacted at emit time, plus the resolved interpreter and working directory. Not the environment. Never the environment.
  • The exit status, the duration, and — separately — whether you actually observed an exit. A client timeout is not evidence the command stopped.
  • Bytes in and out, and egress destinations. Volume tells you the shape of an incident; destinations tell you whether it is one.
  • Every credential handed in, by name and never by value. "What could this process reach?" is answered from this field or by guesswork.
  • TTL, termination reason, and the wall-clock plus monotonic timing of each event. The reason field is the one everybody skips.

The provenance point is sharper on snapshot platforms than people expect. On PandaStack, `fork()` is a disk-and-memory snapshot restore of the parent, so two forked siblings come back byte-identical: same RNG state, same clock, same in-memory session ids. A feature for reproducibility, a trap for identity. If you relied on anything the guest generates to tell two sandboxes apart — a UUID minted at startup, a random token, a timestamp — you now have two machines with the same one. The id that distinguishes them is the one the control plane assigned.

The question, the event that answers it, and where the event has to be born
What you will be askedThe event that answers itWhere it must be emitted from
Which tool call deleted that row?exec.requested with command, actor, agent run id and tool-call idController, before the call
Did a human approve this?approval.granted, hash-linked from the exec event in the same chainYour approval service, into the same per-tenant chain
What could this process reach?exec.requested with credentials named, plus the egress destination eventsController (credentials) and host (egress)
Why did this sandbox disappear?sandbox.destroyed with a reason — ttl_expired, user_delete, quarantine, oomControl plane, never the guest
What did it print?exec.completed with captured stdout/stderr, redacted at emitController, for the session classes you chose to capture
Which ancestor introduced the bad state?sandbox.created with from_snapshot and parent idControl plane, at create
What was on disk at 03:14?Nothing. Unless you snapshotted — see the evidence bagNot reconstructable from logs
import hashlib
import json
import os
import re
import time
from contextlib import contextmanager

from pandastack import Sandbox

# ---------------------------------------------------------------- redaction
# At EMIT time, always. Redacting at read time means the plaintext was
# already durable somewhere, and "somewhere" is the only part an incident
# cares about. This regex is a starting point, not a control -- the real
# control is never passing a secret as an argv token in the first place.
_SECRET = re.compile(
    r"(?i)(?:bearer\s+[A-Za-z0-9._\-]{8,}"
    r"|(?:sk|pk|ghp|gho|xox[baprs])-[A-Za-z0-9._\-]{8,}"
    r"|AKIA[0-9A-Z]{16}"
    r"|(?:api[-_]?key|token|passwd|password|secret)\s*[=:]\s*\S+)"
)


def redact(s: str, limit: int = 8192) -> str:
    s = _SECRET.sub("[REDACTED]", s)
    if len(s) <= limit:
        return s
    return s[:limit] + f"...[+{len(s) - limit} bytes elided]"


# -------------------------------------------------------------- the chain
# Everything in this file runs in the CONTROLLER process. The guest never
# sees the chain, never sees the key, and is never asked what time it is.
class Chain:
    """Append-only, hash-chained event log. One chain per tenant."""

    def __init__(self, tenant: str, sink, head: str = "0" * 64, seq: int = 0):
        self.tenant = tenant
        self.sink = sink   # your append-only writer; must be DURABLE on return
        self.head = head   # hash of the last event for THIS tenant
        self.seq = seq

    def emit(self, kind: str, **fields) -> str:
        ev = {
            "tenant": self.tenant,
            "seq": self.seq,
            "kind": kind,
            # The control plane's clock. A snapshot-restored guest comes back
            # with its clock frozen at bake time until something re-syncs it,
            # so a guest-generated timestamp in an audit record is worse than
            # no timestamp: it is confidently wrong.
            "wall_ns": time.time_ns(),
            # Monotonic orders two events that a wall-clock step would swap.
            # It is process-local and meaningless across restarts -- that is
            # fine, it is a tiebreaker, not an identity.
            "mono_ns": time.monotonic_ns(),
            "prev": self.head,
            **fields,
        }
        body = json.dumps(ev, sort_keys=True, separators=(",", ":"))
        ev["hash"] = hashlib.sha256(body.encode()).hexdigest()
        self.sink(ev)
        self.head, self.seq = ev["hash"], self.seq + 1
        return ev["hash"]


class AuditedSandbox:
    def __init__(self, sbx, chain, run_id, capture_output):
        self._sbx, self._chain, self._run = sbx, chain, run_id
        self._capture = capture_output

    def exec(self, cmd, *, tool_call_id, timeout_seconds=None, credentials=()):
        # INTENT, before the call. An event written only after the fact cannot
        # describe a command that killed its own recorder.
        requested = self._chain.emit(
            "exec.requested",
            sandbox_id=self._sbx.id,
            agent_run_id=self._run,
            tool_call_id=tool_call_id,
            command=redact(cmd),
            # Name the credentials you handed in. Never the values. And never
            # os.environ -- a full environment in an audit record is a breach
            # with a retention policy attached.
            credentials=sorted(credentials),
            # timeout_seconds is a CLIENT deadline; neither exec endpoint
            # enforces it server-side. Record it as what it is, and put the
            # hard limit in the shell (`timeout`, `ulimit`) where it binds.
            client_deadline_s=timeout_seconds,
        )
        t0 = time.monotonic_ns()
        try:
            r = self._sbx.exec(cmd, timeout_seconds=timeout_seconds)
        except BaseException as exc:
            # A client-side timeout is NOT evidence the command stopped. Say
            # exactly what you know: we stopped waiting.
            self._chain.emit(
                "exec.unresolved", sandbox_id=self._sbx.id,
                tool_call_id=tool_call_id, requested=requested,
                error=type(exc).__name__,
                duration_ns=time.monotonic_ns() - t0,
            )
            raise
        self._chain.emit(
            "exec.completed",
            sandbox_id=self._sbx.id,
            tool_call_id=tool_call_id,
            requested=requested,          # links outcome to intent, by hash
            exit_code=r.exit_code,
            duration_ns=time.monotonic_ns() - t0,
            stdout_bytes=len(r.stdout or ""),
            stderr_bytes=len(r.stderr or ""),
            # Metadata for every session; full output only for the classes
            # that matter. You cannot turn this on retroactively, so the
            # decision has to be made from the REQUEST, at create time.
            stdout=redact(r.stdout or "") if self._capture else None,
            stderr=redact(r.stderr or "") if self._capture else None,
        )
        return r

    def __getattr__(self, name):
        # Deliberately NOT a transparent proxy for lifecycle calls: kill(),
        # snapshot() and fork() all have to go through the chain, so they are
        # handled explicitly elsewhere rather than leaking through here.
        if name in {"kill", "snapshot", "fork", "fork_tree", "hibernate"}:
            raise AttributeError(f"{name}() must go through the audit wrapper")
        return getattr(self._sbx, name)


@contextmanager
def audited_sandbox(chain, *, template, actor, run_id, tool_call_id,
                    approval_hash=None, capture_output=False, **create_kw):
    chain.emit(
        "sandbox.create.requested",
        template=template,
        # Who, resolved: the human, the agent run, and the specific tool call.
        # "the agent did it" is not an actor.
        actor=actor,
        agent_run_id=run_id,
        tool_call_id=tool_call_id,
        # The human-in-the-loop approval, by hash, in THIS chain. An approval
        # living in your app DB and an execution living in your event store
        # prove nothing together -- you cannot show the approval preceded
        # THIS command.
        approval_hash=approval_hash,
    )
    sbx = Sandbox.create(
        template=template,
        metadata={"audit_tenant": chain.tenant, "agent_run_id": run_id},
        **create_kw,
    )
    reason = "unknown"
    try:
        chain.emit(
            "sandbox.created",
            sandbox_id=sbx.id, template=sbx.template,
            # Provenance is a TREE, not a field. A forked or restored sandbox
            # whose record does not name its parent snapshot is unusable as
            # evidence -- and on a platform where fork() restores the parent's
            # memory, two siblings are byte-identical at birth.
            from_snapshot=sbx.from_snapshot, boot_mode=sbx.boot_mode,
            cpu=sbx.cpu, memory_mb=sbx.memory_mb,
            guest_ip=sbx.guest_ip, created_at=sbx.created_at,
        )
        yield AuditedSandbox(sbx, chain, run_id, capture_output)
        reason = "completed"
    except BaseException as exc:
        reason = f"error:{type(exc).__name__}"
        raise
    finally:
        # The destruction event, with a REASON. This is the field everyone
        # forgets and the field every investigation needs: a TTL expiry and an
        # operator deleting a mistake look IDENTICAL in a log that only
        # records that the sandbox stopped existing.
        chain.emit("sandbox.destroy.requested", sandbox_id=sbx.id, reason=reason)
        try:
            sbx.kill()
            chain.emit("sandbox.destroyed", sandbox_id=sbx.id, reason=reason)
        except BaseException as exc:
            # Record the failure to destroy too. An undeleted sandbox is a
            # finding, and silence here is how it becomes a surprise invoice.
            chain.emit("sandbox.destroy.failed", sandbox_id=sbx.id,
                       reason=reason, error=type(exc).__name__)
            raise


# -------------------------------------------------------------------- usage
def sink(ev):
    # Replace with an append-only writer: an INSERT into a table whose role
    # has no UPDATE or DELETE grant, plus an object-storage copy under a
    # write-once retention lock. The sink must be durable BEFORE it returns --
    # a buffered log line is not evidence, it is an intention.
    print(json.dumps(ev))


chain = Chain(tenant="acme", sink=sink)

with audited_sandbox(
    chain,
    template="code-interpreter",
    actor={"type": "user", "id": "u_1f0a9c", "email_hash": "sha256:..."},
    run_id="run_7c21",
    tool_call_id="call_01",
    approval_hash=None,
    capture_output=True,       # untrusted code: capture everything
    ttl_seconds=900,
) as sbx:
    sbx.exec("python /work/agent_task.py", tool_call_id="call_02",
             timeout_seconds=120, credentials=["gh_app_token"])

The two things everyone forgets

The destruction event, with a reason

Most audit logs for ephemeral compute record creation beautifully and record destruction as an absence. The row stops appearing, the events stop arriving, you infer the end. This is a hole shaped exactly like a cover-up: a sandbox that vanished because its TTL expired is indistinguishable, in a log that records only that it stopped existing, from one an operator killed at 03:39 to make a mistake go away. Both look like silence.

So emit the destruction, and emit the reason, and make the reason enumerated rather than prose: `ttl_expired`, `user_delete`, `quarantine`, `oom`, `host_lost`, `idle_reaped`. Emit the failure to destroy too — an undeleted sandbox is a finding in its own right, and also how you discover a billing surprise. The `finally` block below exists to make this unskippable, because the one path that will not run your tidy logging code is the path where something threw.

The clock, which is lying to you

This one cost us a real incident, so I will state it as a defect we had rather than a practice we invented. A snapshot-restored guest comes back with the clock it had at bake time, because the guest's idea of the current time is in the memory image Firecracker faithfully restores. Until something re-syncs it, the guest is confidently living in the past — which is how we got TLS handshakes failing inside restored sandboxes for certificate-validity reasons that made no sense. We now re-sync the guest clock on restore, resume and wake.

The audit consequence is worse than the TLS one. A guest-generated timestamp can be wrong by however long ago the template was baked, and wrong identically across every sandbox restored from that snapshot — the worst failure mode, because it looks consistent. The control plane's clock is the one to trust. Record both wall-clock and monotonic readings from the controller: wall clock so humans and subpoenas can use it, monotonic so a clock step cannot reorder two events whose order you know.

Non-repudiation on a budget

You need three properties and none of them require anything exotic. The log must be append-only, its order must be evident, and it must be outside the reach of the people whose actions it records — including you.

Append-only is a grant problem, not a storage problem: the role your application writes with has INSERT and no UPDATE and no DELETE. Order-evident is a hash chain, each event carrying the hash of the previous event for that tenant, so editing one breaks every hash after it. Per tenant rather than global, so that one noisy customer's volume does not gate another's verification and an export verifies on its own. Outside your reach is object storage with a write-once retention lock, plus the part people skip: periodically publish the chain head somewhere you cannot rewrite. A monthly digest in the customer's own inbox is unglamorous and does the job — once the head is in someone else's hands, rewriting history needs their cooperation.

And no, you do not need a blockchain. You have no consensus problem — one writer, nobody disputing which fork is canonical. You have a "someone with production access could edit this table" problem, and the fix is a hash chain plus an externally held checkpoint, which was always the part doing the work.

One more thing, almost always built wrong: approvals. If your human-in-the-loop approval record lives in your application database and the execution it authorised lives in your event store, you have two facts and no link. You cannot demonstrate the approval preceded that specific command, which is exactly what an auditor is asking. Put the approval in the same per-tenant chain and carry its hash on the exec event. Then the ordering is not a claim about two timestamps; it is arithmetic.

A complete audit trail is also a liability

Here is the tension nobody puts in the compliance deck. A genuinely complete audit trail of code execution contains customer data, because the code was processing customer data. It contains secrets, because somebody passed a token as a command-line argument. It contains PII, because the agent was asked to summarise an email thread. You have built one store that concentrates the most sensitive bytes in your system, indexed by tenant, retained for years, readable by support.

So treat it as such. Redact at emit time and never at read time — redaction on read means the plaintext was durable somewhere first, and "somewhere" is the only part that matters in a breach. Never log the command's full environment; the convenience-to-catastrophe ratio on `os.environ` in an audit record is the worst in this post. Record credentials by name rather than value, so the useful question ("what could it reach?") is answerable without the dangerous data. And pattern-matching argv for secrets is a mitigation, not a control; the control is not putting secrets in argv.

Retention is a security control, not a storage decision, and should be argued with the security team rather than finance. Our own ClickHouse event tables carry a 90-day TTL — a statement about how long a compromise of the event store stays painful. If your answer to "how long do you keep execution logs?" is "forever, disk is cheap", you have decided that every future breach includes every past execution, and you decided it by not deciding.

Snapshot before you kill: the evidence bag

Be clear about what is reconstructable from a log. The command stream, yes. Exit codes, durations, byte counts, egress destinations, who asked — yes, if you emitted them. Standard output and error, yes, for the sessions you chose to capture. The full filesystem state at an arbitrary moment: no. Not approximately, not with more logging, not ever. A log records transitions and you are asking for a state.

Unless you snapshot. This is the best idea available on a microVM platform and almost nobody uses it this way, because by the time anyone is thinking about evidence the sandbox has been killed. A Firecracker snapshot is a full guest memory image plus the disk. That is not a log of what happened; it is the machine, preserved — the process table, the heap of the interpreter that was running, the open sockets, the files written and deleted, the half-finished thing the agent was doing at 03:14. You can keep that, restore it three weeks later into a sandbox with no route out, and take it apart.

So wire it into the trigger path rather than into a runbook. When your detector fires — anomalous egress, a command matching a denylist, a credential used outside its blast radius — snapshot, record the snapshot id in the chain alongside the trigger, then kill. The chain entry is what makes it evidence rather than a large file in a bucket: it ties the artifact to the trigger and to the events on either side of it.

The bag is now one of the most sensitive artifacts you own. A guest memory image contains whatever was resident in RAM: every API token the agent was handed, every customer row it had read, every key the process had decrypted. It needs its own access control, its own custodian and its own retention clock — not the bucket your engineers already read. And restoring it re-animates the suspect code, so restore it with no egress, or carve the image offline and never boot it.

Two honest costs. Storage: these are multi-gigabyte artifacts, and a trigger that fires often is a budget line. Time: `snapshot()` is synchronous and writes the full guest memory image to disk — our own Python SDK budgets a 180-second timeout for the call for exactly that reason — and for that whole window the suspect is still running. If the trigger is "this thing is exfiltrating data", every second you spend bagging evidence is a second of exfiltration you chose. So: cut the network, then bag, then kill; and if you cannot cut the network, kill immediately and accept losing the memory image. Each sandbox has its own network namespace plus veth pair and tap device, so the cut is cheap for whoever owns the host — and unavailable to a controller holding only an API client.

def quarantine(chain, sbx, *, trigger, detail):
    """Security trigger: cut, bag, kill -- in that order, with a caveat.

    You cannot keep the machine. You can keep the machine's memory and disk.
    A Firecracker snapshot is a full guest memory image plus the disk, which
    means it is a complete machine you can restore weeks later, offline, and
    interrogate at your leisure. It is the highest-value forensic artifact a
    microVM platform can hand you, and nobody takes it, because by the time
    anyone is thinking about evidence the sandbox has been killed.
    """
    chain.emit("security.trigger", sandbox_id=sbx.id,
               trigger=trigger, detail=redact(detail))

    # 1. CUT. Be honest about what you can actually do here. Asking the guest
    #    to firewall itself is a request to the subject of the investigation;
    #    a compromised guest can decline. The cut you want is host-side, in
    #    the sandbox's own network namespace -- each sandbox has its own netns
    #    plus veth pair and tap device, so severing it is one `ip link set
    #    down` away for whoever owns the host. From a controller holding only
    #    an API client, you do not have that lever.
    #
    #    So the trade is explicit: if the trigger is "this thing is
    #    exfiltrating data", every second you spend bagging evidence is a
    #    second of exfiltration you chose. Kill now, lose the memory image.
    #    If the trigger is "this looks like a compromise but nothing is
    #    leaving", bag it.
    if trigger in {"egress_exfil", "crypto_mining_egress"}:
        chain.emit("evidence.declined", sandbox_id=sbx.id, trigger=trigger,
                   rationale="active egress; killed before snapshot")
        chain.emit("sandbox.destroy.requested", sandbox_id=sbx.id,
                   reason=f"quarantine:{trigger}")
        sbx.kill()
        chain.emit("sandbox.destroyed", sandbox_id=sbx.id,
                   reason=f"quarantine:{trigger}")
        return None

    # 2. BAG. snapshot() is synchronous and writes the full guest memory image
    #    to disk -- the Python SDK budgets a 180-second timeout for the call
    #    for exactly that reason -- and for all of that time the suspect is
    #    still running. That is the cost, stated plainly.
    snapshot_id = sbx.snapshot()

    # 3. Record the bag IN THE CHAIN, so the artifact is tied to the trigger
    #    that produced it and to the events on either side of it. A snapshot
    #    sitting in a bucket with no chain entry is a large file, not evidence.
    chain.emit(
        "evidence.snapshot",
        sandbox_id=sbx.id, snapshot_id=snapshot_id, trigger=trigger,
        # The bag contains whatever was in guest RAM: every API token the
        # agent was handed, every customer row it had read, every key the
        # process had decrypted. It is now a sensitive artifact with its own
        # ACL and its own retention clock, held by whoever holds evidence --
        # not by the engineer who happened to be on call.
        classification="sensitive:contains-guest-memory",
        custodian="security-evidence",
    )

    # 4. KILL, with a reason that survives the machine.
    chain.emit("sandbox.destroy.requested", sandbox_id=sbx.id,
               reason=f"quarantine:{trigger}")
    sbx.kill()
    chain.emit("sandbox.destroyed", sandbox_id=sbx.id,
               reason=f"quarantine:{trigger}")

    # Restoring the bag re-animates the suspect code, so restore it into a
    # sandbox with no egress, or carve the image offline and never boot it.
    # Note the forensic upside of the thing that normally bites people: a
    # restored guest comes back with the parent's RNG state and clock, so the
    # replay is deterministic in a way a re-run never is.
    #     Sandbox.create(template="forensics", from_snapshot=snapshot_id)
    return snapshot_id

Sampling, and what full capture actually costs

Full standard-output capture for every sandbox is expensive and mostly noise. An agent that runs a build prints a hundred thousand lines nobody will ever read, and you will pay to store, index and eventually leak all of them. Event metadata for everything; full output capture for the sessions that matter.

  • Untrusted code — anything the content of which was decided by a model or a customer rather than by your repository.
  • Privileged credentials — any session handed a token that can write to something you care about, which is a property of the session, not of the command.
  • Production-adjacent targets — the sandbox has a route to a real database, a real queue, a real payment API.
  • Anything already under suspicion: a tenant with a prior finding, an account inside its first week, a repo on a denylist.

The uncomfortable part is when you decide. You cannot retroactively turn on output capture — the output is gone, and it was gone before you knew you wanted it. The classification has to be computed at create time, from the request, before anything interesting has happened. That is an architectural constraint masquerading as a config flag, and it is why the capture decision is a parameter of the create call above.

-- Two stores, two queries. This split is not an accident of our
-- architecture; it is the architecture. Tenant identity is resolved by the
-- control plane and lives in its Postgres. The high-volume event stream is
-- keyed by the ids the agents actually know. An investigation is a join you
-- do by hand, and knowing that beforehand is the difference between twenty
-- minutes and an afternoon.

-- 1. ClickHouse: everything that happened to one sandbox, control-plane
--    clock, in order. The first query of every incident.
SELECT ts,
       type,
       code,
       message,
       JSONExtractString(metadata, 'reason')  AS reason,
       JSONExtractString(metadata, 'actor')   AS actor
FROM pandastack.sandbox_events
WHERE sandbox_id = {sandbox_id:String}
ORDER BY ts
FORMAT Vertical;

-- 2. Postgres (control plane): the tenant-scoped list for a date range --
--    which is the query a customer, an auditor or a lawyer actually asks
--    for. org_slug lives HERE, on the row the create proxy writes; the
--    ClickHouse rows are keyed by workspace and sandbox id.
SELECT sandbox_id,
       template,
       created_at,
       deleted_at,
       (deleted_at IS NULL) AS still_unaccounted_for
FROM sandbox_lifecycle
WHERE org_slug = $1
  AND created_at >= $2
  AND created_at <  $3
ORDER BY created_at;

-- Rows where deleted_at IS NULL and no destruction event exists in query 1
-- are the exact gap this post is about: a machine that stopped existing and
-- never said why. Treat a non-empty result there as a defect in the emitter,
-- not as a mystery about the sandbox.

What the platform gives you, and what it does not

Briefly and factually, because this is where blog posts turn into brochures. PandaStack emits lifecycle events to an in-process event bus with a ClickHouse sink; the `sandbox_events` table carries a millisecond timestamp, workspace, sandbox and agent ids, an event type and code, a message and a metadata JSON string, with a 90-day TTL. Each sandbox also exposes a live events stream over server-sent events and a logs endpoint — and that logs endpoint serves the host-side Firecracker log, the serial console, which is a different thing from your application's standard output and trips people up about once per integration.

The tenant-scoping detail matters more than it sounds. The ClickHouse rows are keyed by workspace and sandbox id; the org slug lives on a control-plane Postgres row written once per successful create. That is what makes a tenant-scoped investigation a join you can run rather than a guess assembled from ids. It is also the right place for it: tenant identity is resolved by the control plane during authorisation, so recording it elsewhere means recording a copy that can drift.

And what we do not give you: a hash-chained, tamper-evident audit log. Our ClickHouse schema has an `audit_log` table sitting in the file commented out, with a note saying there is no writer yet — an honest artefact of priorities. The chain in this post is one you build in your controller, and that is where it belongs, because a useful chain is anchored to your identity model, your approval flow and your tenant boundary, none of which I can see from here. What a platform owes you is the lifecycle truth, the stream, and the snapshot primitive.

The summary, if you implement one thing: emit intent before the call and a reason on destruction, from the controller, on the controller's clock. That is about forty lines, and it is the difference between an incident that takes twenty minutes and one that becomes a legal exercise in explaining why you cannot say.

Frequently asked questions

Can I not just run auditd or an eBPF agent inside the sandbox?

You can, and it is useful, but it cannot be your record of account for two separate reasons. The first is authority: a collector running inside a guest that the agent controls is reporting on its own subject, and the code you are trying to characterise is precisely the code with the means to stop, blind or feed it. The second reason kills it even with a perfectly honest guest, which is shipping. A line emitted but not yet forwarded when the microVM is torn down is gone — there is no graceful shutdown, no flush, no chance to drain a batching client's queue. So in-guest telemetry always has a missing tail of unknown length, and it goes missing exactly when the VM died abruptly, which is the same moment as every incident you care about. Use it for depth: syscall-level detail, file access, process trees. Anchor the record in your control plane, which both knows the intent and outlives the machine, and treat the in-guest stream as corroboration that may or may not be there.

How long should I keep execution audit records, and what about the snapshots?

Decide it as a security control with the security team, not as a storage question with finance, and then write down the reasoning so the next person does not quietly change it. The two forces pull opposite ways: investigations and auditors want a long window, while a longer window means every future breach of your log store includes more past executions, and that store is unusually concentrated — it holds command text, captured output and tenant identity in one indexed place. A common and defensible shape is a relatively short hot window for rich records including captured output, a longer window for event metadata only, and a short, explicit window for forensic snapshots with a named custodian. Our own ClickHouse event tables carry a 90-day TTL. Snapshots deserve their own, shorter clock and their own access control, because a guest memory image contains every token and customer row that was resident in RAM; if your evidence bag is more sensitive than your production database and lives in a bucket your engineers can already read, you have moved the risk rather than reduced it.

Does the snapshot-as-evidence-bag idea have a container equivalent?

Not a clean one, and the gap is the honest argument for a VM boundary here. Checkpointing a container means checkpointing processes that share the host's kernel, so you can capture process memory and the writable layer, but you cannot capture the kernel-side state those processes were interacting with, and that state lives in a kernel shared with everybody else's workloads. CRIU does real work and is genuinely impressive, but it has a long list of things it cannot restore, and it is not designed to be a forensic capture of a hostile workload. A microVM snapshot is a different kind of object: the guest has its own kernel, so the memory image plus the disk is the whole machine, and the artifact is self-contained in a way that can be restored on a different host weeks later. It is also worth saying what the snapshot does not give you, which is the host-side view — the network flows, the resource accounting, the scheduling decisions. Those come from the control plane. Read your container runtime's current checkpoint documentation before taking my word on any of it; the capability moves.

If a fork restores the parent's memory, can two sandboxes produce identical audit records?

Yes, and it catches people out. On PandaStack a fork is a disk-and-memory snapshot restore of the parent, so two siblings come back byte-identical: the same RNG state, the same clock, the same in-memory session identifiers, the same process ids. If any part of your audit identity is generated inside the guest — a UUID minted at startup, a random request id, a timestamp — then two machines doing different work will emit records that claim to be the same machine, and you will spend an afternoon deciding which of two contradictory event streams to believe. The fix is the same architectural point as the rest of this post: the identity that distinguishes sandboxes is the one the control plane assigned, and provenance has to be recorded as a tree that names the parent snapshot. Then two siblings are distinguishable by construction and their shared ancestry is a fact you can query rather than a confusion. Seed the guest's RNG and re-sync its clock after a fork as well, for the ordinary correctness reasons.

What is the smallest version of this I can actually ship this week?

Four events and one field. Emit sandbox.create.requested with the actor, the agent run id and the tool-call id; sandbox.created with the template and the parent snapshot; exec.requested with the redacted command before you make the call; and sandbox.destroyed with an enumerated reason, from a finally block so it cannot be skipped. The one field is the reason on destruction, because without it a TTL expiry and a deliberate deletion are the same log entry. All of it from your controller, on your controller's clock, into a table whose application role has no UPDATE and no DELETE grant. That is an afternoon, and it answers most of what you will be asked. Then add, in this order: the hash chain, because retrofitting order-evidence onto existing rows is miserable; the approval hash on exec events, because that is the one auditors push hardest on; output capture for your untrusted-code class; and the snapshot-on-trigger path. Do not start with the blockchain, the data lake or the schema working group.

Keep reading

Related posts

More in AI agent sandboxes · See AI agent sandboxes on PandaStack

Run code in a microVM in one API call.

49ms p50 cold start. Fork, snapshot, and scale to zero.

Start free
Written by Ajay Kumar, Founder, PandaStack.