Timeouts and Cancellation for AI Agent Tool Calls

Every agent that executes code eventually writes something that never finishes. Not maliciously — sincerely. It writes `while True: pass` because the loop condition seemed right. It calls `input("Continue? ")` on a stdin that will never, ever produce a byte, and then waits with the patience of a system that has nowhere else to be. It runs `pip install` against a mirror that accepted the TCP connection and then went quiet. It writes a retry loop with no backoff and no ceiling, which is technically an availability strategy. And the whole time, your task is "running," your sandbox is billed, and your agent loop is blocked on a future that will never resolve.

I'm Ajay, I built PandaStack, and I have watched an awful lot of hung tool calls. The thing I want to convince you of is narrow but load-bearing: a timeout is not a feature of your agent code, it is a property of the boundary you can destroy. If the thing you're timing out is running with the same privileges as the thing doing the timing, you do not have a timeout — you have a polite request. This post covers the four layers of timeout you actually need, why in-process timers are theater, and why killing a microVM is the only cancellation primitive that works every single time.

The specific ways agent-written code hangs

These aren't exotic. They're the same six failure modes over and over, and knowing them tells you where each layer of defense has to sit.

• The honest infinite loop — `while True: pass`, or a `for` over a generator that never exhausts. Burns 100% of a core, produces no output, and will do it with total sincerity, forever, on your bill.
• Blocking on stdin that will never come — `input()`, `read -p`, an interactive `apt` prompt, `git` asking for credentials, a package manager asking you to confirm. Zero CPU, zero output, infinite patience. This one is the worst because it looks like nothing is wrong.
• A network call with no timeout — a dead mirror, a hung DNS resolver, a socket that got a SYN-ACK and then silence. The default socket timeout in most languages is 'never', and the model rarely overrides it.
• The retry loop with no backoff and no cap — `while not ok: try again`. Technically progress-shaped, actually a denial-of-service against someone else's API and a permanent occupant of your task slot.
• The fork bomb — usually accidental (a recursive build script, a subprocess spawned in a loop), occasionally not. Fills the process table, and now even your cleanup code can't fork a shell to clean up.
• The process that catches SIGTERM and keeps going — a trap handler that logs 'shutting down gracefully' and then doesn't. Or a JVM in a shutdown hook, or a process ignoring signals during an uninterruptible disk wait. Your `kill` returned 0. Nothing died.

Why in-process timeouts are theater

The instinct is to reach for what's nearest: `signal.alarm`, `asyncio.wait_for`, a `context.WithTimeout`, a `Promise.race`. These are useful for bounding your own well-behaved code. They are not a containment mechanism for code you did not write, and the reason is structural: they run inside the same process, or the same trust domain, as the thing they're supposed to stop.

`signal.alarm` delivers SIGALRM to the process. If the untrusted code installed its own handler, or is sitting in a C extension that doesn't check for signals between instructions, or is blocked in an uninterruptible syscall, your alarm is a note left on the fridge. `asyncio.wait_for` is worse in a way that surprises people: it cancels the *await*, not the work. If the coroutine is blocked on a synchronous call — a `requests.get` with no timeout, a tight CPU loop, a blocking file read — the event loop never gets control back to deliver the cancellation at all. Your `wait_for` raises `TimeoutError` after the fact, or hangs right alongside the thing it was watching. Meanwhile the subprocess it spawned three frames down is still running, still holding a port, still writing to disk.

A timeout implemented inside the thing being timed out is a suggestion. A timeout implemented by something that can destroy the thing being timed out is a guarantee.

Even stepping out one level to `subprocess.kill()` only gets you partway. SIGKILL is unignorable — you can't trap it — but it applies to one PID. The child that already double-forked and reparented to init is now an orphan you have no handle on. The process group you tried to signal changed when someone called `setsid`. The thread blocked in an uninterruptible D-state disk wait won't die until the I/O completes. You end up writing process-group bookkeeping and a reaper loop, which is a real engineering project whose entire purpose is to approximate something you can get for free at a different layer.

The four layers of timeout you actually need

One timeout is never enough, because the hangs happen at different scales. A single command hanging is a different problem from a task that makes forty successful tool calls and never converges. You want four independent limits, each of which is enforced by something the layer below it cannot reach.

1. Per-exec command timeout

The tightest loop: every single command the model runs gets a wallclock bound. This is the one that catches `input()`, the dead mirror, and the tight loop. Set it per call-site, not globally — a `pytest` run legitimately takes five minutes, an `ls` legitimately takes 20 milliseconds, and a shared 300-second default means your fast tools spend five minutes hanging before you find out anything is wrong. Enforced by the sandbox platform, outside the guest, so guest code can't influence it.

2. Per-sandbox TTL

The backstop for your own code. Every sandbox gets a time-to-live at creation, and the platform reaps it when the clock runs out — whether or not your agent process is still alive, whether or not your `finally` block ran, whether or not your orchestrator got OOM-killed mid-task. This is the layer that saves you from the class of bug where the agent crashes and leaves a VM running for eleven days. Your cleanup code will eventually fail to run; the TTL is what makes that survivable instead of expensive.

3. Per-task wallclock budget across tool calls

Here's the failure mode that the first two layers completely miss: every individual command succeeds, well under its timeout, and the task still never ends. The agent tries a fix, runs the test, sees it fail, tries a slightly different fix, runs the test, sees it fail. Forty times. Each `exec` returns in nine seconds and exit code 1. No timeout fires, because nothing hung — the agent is just wrong in a loop, at a steady and confident pace. You need a budget that spans the whole task: a deadline computed once at task start, checked before every tool call, and enforced by stopping the loop. Pair it with a step count and a token cap so you're bounded on three axes.

4. A hard resource ceiling

The last layer isn't about time, it's about damage rate. A fork bomb, a memory balloon, or a disk-filling loop can make the machine unusable long before any wallclock deadline arrives — including making it unable to run your cleanup. In a microVM this comes for free: the guest gets a fixed vCPU count and a fixed RAM allocation baked into the snapshot, and a fork bomb inside it is a fork bomb inside a box with 2 vCPUs and 4 GiB that you are about to delete. The host never notices. That's the difference between a resource limit you have to configure correctly and one that's a property of the boundary.

Killing the VM is the only cancellation that always works

Everything above assumes you have a reliable way to actually stop the work. This is where most cancellation stories fall apart, because the usual primitives all have survivors.

• SIGTERM to a PID — What it kills: cooperative processes that handle it. What survives it: anything with a trap handler that logs 'shutting down' and continues, plus every child it spawned.
• SIGKILL to a PID — What it kills: that one process, unconditionally. What survives it: orphaned children that reparented to init, double-forked daemons, and anything stuck in an uninterruptible D-state wait until its I/O completes.
• SIGKILL to a process group — What it kills: the group as it existed when you looked it up. What survives it: anything that called `setsid` and left the group, plus races where a new child is forked between your enumeration and your signal.
• Closing the connection / cancelling the RPC — What it kills: your client's interest in the result. What survives it: literally all of the work, which continues running and billing on the far side of a socket nobody is reading.
• Container stop — What it kills: the namespace's PID 1 and, usually, its descendants. What survives it: whatever escaped to the host, mounted host paths that got written to, and anything sharing the host kernel that's now in a bad state.
• Destroying the microVM — What it kills: the entire guest — every descendant process, every open file descriptor, every leaked thread, every socket, the page cache, the guest kernel itself. What survives it: nothing. There is no process table left to have survivors in.

That last line is the whole argument. When you destroy a microVM you are not asking a process to stop; you are removing the machine it was running on. There's no signal to trap, no reparenting to escape to, no D-state to hide in, because the kernel that would have to honor any of that is gone too. Cancellation goes from a distributed-systems problem with a long tail of edge cases to a single unconditional operation. The runaway process didn't get killed — it stopped existing, along with the universe it inhabited.

The corollary matters just as much: because the microVM is the unit of cancellation, it should also be the unit of work. One task per sandbox, or one risky tool call per sandbox. If you're multiplexing six tenants' tool calls into one long-lived VM, you've re-created the exact problem — now cancelling one job means going back to signals and process groups, because you can't destroy the machine without taking down five innocent bystanders.

Make cancellation cheap enough to be liberal about it

There's a reason teams nurse wedged environments instead of killing them: replacing them is expensive. If a fresh environment costs 90 seconds of container build and dependency install, you'll write elaborate recovery logic — kill the stuck process, clear the port, reset the working tree, hope — because the alternative is worse. That recovery logic is bug-farmland, and it's the reason people build process reapers instead of just destroying things.

Invert the cost and the whole design changes. On PandaStack, a create restores a baked Firecracker snapshot on demand — about 49ms for the restore step, p50 179ms end to end, p99 around 203ms, with no warm pool of idle VMs (only a template's first-ever cold boot takes ~3 seconds). If you want to resume from a known-good post-setup state rather than a clean template, fork a configured sandbox instead: 400–750ms same-host, 1.2–3.5s cross-host. At those numbers, "kill it and start over" is faster than most recovery logic is at *deciding* whether to recover. So you stop writing the recovery logic.

That unlocks a bunch of tactics that are only sane when cancellation is cheap: kill aggressively on the first sign of a wedge rather than after three escalating attempts; speculatively run the same tool call in two sandboxes and kill the loser; give each retry a genuinely fresh machine instead of a scrubbed one, so retry #2 isn't inheriting the corpse of retry #1. Liberal cancellation is a strategy you can only afford if creation is milliseconds, which is the actual point of the boot path.

Preserve the evidence before you pull the trigger

The one genuine downside of destroying the machine is that the machine is where all your debugging information lives. A timeout with no output is the least actionable event in software: something took too long, we don't know what, it's gone now. So the order of operations is always the same — read first, then destroy. Grab the tail of the log file, the last few hundred lines of stdout, the process list if you can get it, any partial artifact the task wrote. Then kill it.

Do the reads with their own short timeouts, and treat every one of them as best-effort. You are, by definition, talking to a machine that has already demonstrated it can hang — if your evidence-collection step blocks, you've turned a bounded timeout into an unbounded one, which is a genuinely embarrassing way to lose an afternoon. Wrap each read in a two-second budget, swallow the failures, and proceed to the kill regardless. Evidence is a nice-to-have; the kill is not.

Return a useful timeout result to the model

Here's the part that's easy to skip and expensive to skip: the timeout is not just an operational event, it's a tool result that goes back into the model's context. And what you put there determines whether the agent adapts or repeats itself. If you return `{"error": "tool call failed"}`, the model has no idea what happened and its most likely next move is to run the identical command again. Congratulations, you've built a retry loop with no backoff — the exact thing you were trying to defend against, only now it's at the agent layer.

A good timeout result tells the model three things: that it was a timeout and not a crash, how long it actually got, and what the process managed to say before the clock ran out. That last one is what turns a dead end into a diagnosis: the tail of stdout usually shows `Proceed? [Y/n]` sitting there unanswered, or a pip resolver spinning on the same package. A model that sees that will pass `--yes`, or add `--timeout`, or pick a different mirror. A model that sees `error: failed` will try again, sincerely, forever.

• Say the word "timeout" and the number — "timed out after 30s" is unambiguous. `exit_code: -1` is not.
• Include the output tail, not the head — the last 2000 characters are where the hang is. The first 2000 are the banner.
• Truncate explicitly and say so — an unmarked truncation makes the model think the program ended where the text ends.
• Say what happened to the environment — "the sandbox was destroyed; a new one will be created" stops the model from writing commands that assume its half-finished state survived.
• Suggest the shape of a fix, not the fix — "if the command was waiting on input, use a non-interactive flag" is a hint the model can act on without you pretending to know the answer.

All four layers, in one wrapper

Here's the shape of it. A task-level deadline computed once, a TTL on the sandbox, a per-exec timeout on every command, evidence collection before the kill, and a model-facing result that explains itself.

import time
from pandastack import Sandbox
from pandastack.exceptions import SandboxTimeout

TASK_BUDGET_S = 600      # layer 3: whole-task wallclock
MAX_STEPS = 25           # layer 3: and a step cap, because time isn't enough
SANDBOX_TTL_S = 900      # layer 2: platform reaps it even if we crash
TAIL = 2000              # what we hand back to the model


class BudgetExhausted(Exception):
    pass


class AgentTask:
    """One task, one microVM, four independent timeout layers."""

    def __init__(self) -> None:
        self.deadline = time.monotonic() + TASK_BUDGET_S
        self.steps = 0
        # Layer 2 + layer 4: the TTL is enforced by the platform, and the
        # guest's vCPU/RAM ceiling is baked into the snapshot. A fork bomb
        # in here is a fork bomb in a box we are about to delete.
        self.sbx = Sandbox.create(
            template="code-interpreter",
            ttl_seconds=SANDBOX_TTL_S,
            metadata={"surface": "agent-task"},
        )

    def remaining(self) -> float:
        return self.deadline - time.monotonic()

    def run_tool_call(self, command: str, limit_s: int = 60) -> dict:
        # Layer 3: check the task budget BEFORE spending more of it.
        if self.steps >= MAX_STEPS or self.remaining() <= 0:
            raise BudgetExhausted("task budget exhausted")
        self.steps += 1

        # Never let one command outlive the task it belongs to.
        limit = int(min(limit_s, self.remaining()))

        # Layer 1: per-exec timeout, enforced outside the guest. Guest code
        # cannot trap it, ignore it, or outlive it. The SDK surfaces the
        # cutoff as SandboxTimeout rather than a normal non-zero exit.
        try:
            result = self.sbx.exec(command, timeout_seconds=limit)
        except SandboxTimeout:
            return self._on_timeout(command, limit)

        return {
            "exit_code": result.exit_code,
            "stdout": result.stdout[-TAIL:],
            "stderr": result.stderr[-TAIL:],
        }

    def _on_timeout(self, command: str, limit: int) -> dict:
        # Evidence FIRST, kill second. Best-effort, tightly bounded: we are
        # talking to a machine that has already proven it can hang.
        tail = ""
        try:
            probe = self.sbx.exec(
                "tail -c 4000 /tmp/tool.log 2>/dev/null || true",
                timeout_seconds=2,
            )
            tail = probe.stdout[-TAIL:]
        except Exception:
            pass  # evidence is nice-to-have; the kill is not

        # The only cancellation with no survivors: destroy the machine.
        # Every descendant process, FD, thread and socket goes with it.
        self.sbx.kill()
        self.sbx = None

        # A result the MODEL can act on, not just an error string.
        return {
            "exit_code": None,
            "status": "timeout",
            "error": f"Command timed out after {limit}s and was cancelled. "
                     f"The sandbox was destroyed; any partial state is gone. "
                     f"If it was waiting on input, re-run non-interactively.",
            "command": command,
            "stdout_tail": tail or "[no output captured before timeout]",
            "truncated": True,
        }

    def close(self) -> None:
        if self.sbx is not None:
            self.sbx.kill()

`exec` returns `stdout`, `stderr`, and `exit_code`; `kill()` destroys the microVM. The important structural detail is that `timeout_seconds` is enforced by the agent on the host, not by anything inside the guest — that's what makes it a real bound rather than a cooperative one. And note that `limit` is clamped by the task's remaining budget, so a 300-second `pytest` call made with 40 seconds left on the task clock gets 40 seconds, not 300.

To see why the outer boundary has to exist, here's the kind of thing that walks straight through the naive defenses. None of it is adversarial; it's all stuff a well-meaning model writes on a Tuesday.

#!/usr/bin/env bash
# Four ordinary hangs that defeat naive timeouts. Nothing here is malicious.

# 1. Traps SIGTERM and keeps going. Your graceful shutdown is a log line.
trap 'echo "shutting down gracefully..."' TERM INT

# 2. Double-forks so the child reparents to init. Killing $! kills the
#    launcher; the actual worker is now an orphan you have no handle on.
( ( while true; do :; done ) & ) &

# 3. Blocks on stdin that will never produce a byte. Zero CPU, zero output,
#    infinite patience — invisible to any CPU-based runaway detector.
read -r -p "Proceed? [Y/n] " answer

# 4. Retries forever against a mirror that accepted the connection and then
#    went quiet. No --timeout, no backoff, no ceiling. Technically resilient.
until pip install --index-url http://mirror.internal/simple somepkg; do
  echo "retrying..."
done

# `kill -TERM $pid` here returns 0 and changes nothing.
# `kill -9 $pid` gets the launcher and leaves the orphan spinning.
# Destroying the microVM gets all of it, unconditionally, in one operation.

Cancelling a half-finished side effect

Destroying the VM ends the compute cleanly. It does not un-send the email. If the tool call had external side effects — a payment charged, a row inserted, a webhook fired, a file uploaded to object storage — cancellation leaves you in the classic distributed-systems position of not knowing how far it got. The process died between "POST accepted" and "response received," and there is no log to tell you which, because the log was in the machine you deleted.

The answer isn't more careful killing, it's making the operations retry-safe so that not knowing is survivable:

1. Generate an idempotency key per logical operation, outside the sandbox, before the tool call runs — and pass it in. Then a retry after an ambiguous timeout is a no-op if the first attempt landed, and a fresh attempt if it didn't. Deriving the key inside the guest defeats the whole thing: the retry gets a new machine and a new key.
2. Write side effects through a durable outbox rather than firing them inline, so the effect is committed once by something that outlives the sandbox, not N times by however many attempts you made.
3. Prefer operations that are naturally idempotent — PUT over POST, upsert over insert, content-addressed uploads whose name is the hash of what's in them.
4. Record intent before acting and reconcile after. A row that says 'attempting charge X with key K' written before the call lets you resolve the ambiguity later even though the sandbox is gone.
5. Treat a timeout as UNKNOWN, never as failure. The most expensive bug in this whole area is code that treats a timed-out charge as 'didn't happen' and cheerfully charges again.

The takeaway

Agents write code that hangs, and they'll keep doing it, because "loop until the condition is met" is a reasonable thing to write and "the condition is never met" is not something you can see from inside the loop. You can't fix that at the model layer. What you can do is make hanging bounded and cancellation total: a per-exec timeout on every command, a TTL on every sandbox, a wallclock-and-step budget across the whole task, and a hard resource ceiling that comes free with the boundary. Then make the cancellation itself unconditional by destroying the machine instead of negotiating with the process — no traps, no orphans, no D-state, no survivors.

The last piece is economics. All of this only works if a replacement machine is nearly free, because otherwise you'll nurse the wedged one and write the reaper loop and slowly reinvent a worse version of the hypervisor. When a fresh isolated sandbox is a ~49ms snapshot restore, killing becomes the cheap default and the entire category of "recovering a stuck environment" disappears from your codebase. The model will still write `while True: pass`. It'll just do it for thirty bounded seconds, in a box you were going to throw away, and then tell you what it was thinking.