Designing a Guest-Agent Control Protocol Over vsock

Here is the situation a sandbox platform is always in. You have a Firecracker microVM running someone else's code — an AI agent, a build, a red-team payload, whatever. Inside it is a small init/agent process you put there. From the host, you need that process to do real work on your behalf: run a command and hand back stdout, stderr, and an exit code; write a file the workload needs; read a file the workload produced; and, before any of that, tell you the exact instant the guest is up and able to take orders. The tempting shortcut is to open an HTTP port inside the guest and call it. Don't. That hands the guest a network stack, a routable port, and an attack surface, all so you can run `ls`. The right answer is to design a tiny control protocol and carry it over vsock — a channel that reaches the guest without the guest having any network at all. This post is about designing that protocol: the transport choice, the framing, the streaming, the restore semantics, and the security posture that keeps a privileged inside-the-guest agent from becoming an escape hatch. PandaStack's own guest agent, pandastack-init, is the running example.

Why vsock, and not a port

The control channel has four requirements that quietly rule out TCP-over-network. It has to reach the guest before the guest's network exists — you want to ping for readiness the moment userspace is up, not after DHCP, routing, and a firewall have all converged. It has to keep working when the guest's network is clamped to default-deny egress or removed entirely, because for untrusted workloads that lockdown is the whole point. It must never widen the guest's attack surface with a port the workload's own code can see, bind, or collide with. And it must be structurally incapable of reaching a sibling sandbox. vsock satisfies all four because it is orthogonal to networking.

vsock is a socket address family — AF_VSOCK — for host-to-guest communication, used through the ordinary sockets API (socket, bind, listen, connect, read, write). The only thing that changes is the address: instead of (IP, port) you use (context ID, port). CID 2 always means the host; each guest gets its own CID (Firecracker conventionally uses 3). There is no IP layer, no routing, no ARP, no DNS — bytes ride the virtio transport, shared-memory rings between the guest and the VMM. The channel is up as soon as guest userspace is, whether or not the guest has a NIC. And because the transport is point-to-point, a guest can address only the host or itself, never a neighbor. (For the CID/port model in depth, see /blog/vsock-explained; for the head-to-head against a TCP channel, /blog/virtio-vsock-vs-tcp-guest.)

One convenience worth knowing before we design anything: on the host side, Firecracker does not make you speak AF_VSOCK at all. It bridges the guest's vsock onto a Unix domain socket (UDS) on the host filesystem. You configure the device with a guest CID and a host UDS path; to reach a guest port, a host process connects to that UDS and, as its very first line, writes `CONNECT <port>\n`. Firecracker replies with `OK <assigned_port>` and then forwards a transparent byte stream to whatever is listening in the guest. So the guest-side listener is real vsock, but the host-side dialer is plain Unix-socket code — trivial to wire into any language.

The framing: length-prefixed messages

vsock, like TCP, is a byte stream, not a message stream. Read a chunk and you might get half a request, one request, or two and a half. So the first design decision is framing — how the receiver knows where one message ends and the next begins. Resist the urge to reach for HTTP here; you'd be dragging a whole protocol across the boundary to move a few JSON objects. The boring, correct answer is a length prefix: write a fixed-width byte count, then that many bytes of payload. The reader reads the count, then reads exactly that many bytes, then loops. No delimiters to escape, no ambiguity, no re-implementing chunked encoding by hand.

Inside each frame, a small JSON object is plenty for a control protocol — it's self-describing, easy to version with an added field, and the message rate is low (you're issuing commands, not streaming video). Design the request/response pairs to be minimal and explicit. A workable core RPC looks like this:

• ping{} -> {ready: true} — the readiness probe. The single most important call, because it's how the control plane knows the guest is alive; keep it trivial and side-effect-free.
• exec{cmd, args, env?, cwd?, timeout_ms?} -> {stdout, stderr, exit_code, duration_ms} — run a command to completion and return the captured result. The unary (non-streaming) form for quick commands.
• fs.write{path, bytes, mode?} -> {bytes_written} — write a file inside the guest. bytes is base64 in the JSON, or better, carried in the frame body after the header so you're not inflating binary by a third.
• fs.read{path, offset?, len?} -> {bytes, size} — read a file back out. Support offset/len so you can page a large file rather than materializing it all in one frame.
• fs.stat{path} -> {exists, size, mode, is_dir} and fs.list{path} -> {entries} — the two read-only filesystem calls you always end up needing.
• exec.stream{cmd, ...} -> (stream of {kind: 'stdout'|'stderr'|'exit', ...}) — the streaming form of exec, covered below.

That is close to the entire surface. The discipline is to keep it that small on purpose (more on why under the security section). Every request carries an id so responses can be correlated if you multiplex, and every response carries either a result or a structured `{error: {code, message}}` — never a bare stack trace the caller has to string-match.

The host side: connect, frame, request

Here is an illustrative host-side sketch in Python — no AF_VSOCK on the host at all, because Firecracker bridged it to a Unix socket. Connect to the UDS, do the CONNECT handshake, then speak your length-prefixed protocol. This is deliberately stripped down to show the shape, not a production client:

import socket, struct, json

UDS = "/var/lib/pandastack/vms/<id>/vsock.sock"
GUEST_PORT = 1024  # where the guest agent listens

def connect():
    s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    s.connect(UDS)
    # Firecracker's handshake: wire us through to a guest vsock port.
    s.sendall(f"CONNECT {GUEST_PORT}\n".encode())
    ack = s.recv(64)                 # e.g. b"OK 12345\n"
    assert ack.startswith(b"OK"), ack
    return s

def _read_exact(s, n):
    buf = b""
    while len(buf) < n:
        chunk = s.recv(n - len(buf))
        if not chunk:
            raise ConnectionError("guest agent closed the stream")
        buf += chunk
    return buf

def request(s, msg: dict) -> dict:
    body = json.dumps(msg).encode()
    # 4-byte big-endian length prefix, then the JSON frame.
    s.sendall(struct.pack(">I", len(body)) + body)
    (n,) = struct.unpack(">I", _read_exact(s, 4))
    return json.loads(_read_exact(s, n))

s = connect()
print(request(s, {"op": "ping"}))                    # {'ready': True}
print(request(s, {"op": "exec", "cmd": "uname",       # unary exec
                  "args": ["-a"], "timeout_ms": 5000}))
s.close()

The guest side is the mirror image: it binds a real AF_VSOCK socket on VMADDR_CID_ANY at the same port, accepts, reads a length-prefixed frame, dispatches on the op, and writes a length-prefixed reply. The transport underneath is exactly that small; everything richer is built on top of the accept loop.

Streaming, backpressure, and timeouts

Unary exec is fine for `uname -a`. It's wrong for `npm install` or a ten-minute build, where you want output as it happens and you don't want to buffer megabytes of logs in the guest before returning. That's what exec.stream is for: instead of one response frame, the agent emits a sequence of frames — a `stdout` frame per chunk, `stderr` frames interleaved, and a final `exit` frame carrying the exit code and duration. The caller reads frames in a loop until it sees `exit`. Same length-prefixed framing, just many frames per request. In PandaStack this surfaces to users as a streaming exec: you subscribe and get stdout/stderr events live, then a terminal exit event.

Streaming forces you to think about backpressure, which unary requests let you ignore. If the guest produces output faster than the host consumes it, something has to give. The clean answer is to let the socket itself be the backpressure signal: the guest agent's write to the vsock socket blocks when the host isn't reading, which — if the agent reads the child process's pipe only as fast as it can forward — naturally throttles the child. Don't build an unbounded in-memory queue in the agent to 'smooth' this out; an unbounded queue inside an untrusted guest is just a slow-motion OOM (see /blog/firecracker-oom-killer-guest-memory). Bound every buffer, and let a full pipe slow the producer down. That's the correct behavior anyway.

Readiness and idempotency across snapshot-restore

The readiness ping is the load-bearing call, and it gets more interesting once you remember how these VMs actually boot. PandaStack has no warm pool of idle VMs — every create restores a baked Firecracker snapshot. The snapshot-restore step itself lands in roughly 49ms, and the full create path returns a usable sandbox at about 179ms p50 and 203ms p99 (a true first cold boot, before any snapshot exists, is around 3s). The control plane wants to hand the sandbox back the instant it is genuinely ready — and 'ready' is best answered by the guest agent itself over vsock, not guessed from the outside by probing a port and hoping.

Restore introduces two subtleties the protocol has to respect. First, idempotency: the ping (and ideally every read-only op) must be safe to call repeatedly, because the control plane will retry it while the guest finishes coming up. ping returning `{ready: true}` twice must mean the same thing as returning it once — no state changes, no side effects. Design the readiness op to be a pure liveness check, never a one-shot 'mark as started' that breaks on the second call.

Second, the socket path itself has to be per-restore. A single baked snapshot is restored into many different sandboxes, and the host-side UDS path frozen into the snapshot at bake time can't be shared across all of them. Firecracker's ability to override the vsock UDS path at snapshot restore is exactly the piece that makes this compose: each restore points the vsock bridge at a fresh per-sandbox socket, so the host does a clean per-sandbox readiness ping on the restored guest over its own path. Without that override you'd be multiplexing every restored sandbox onto one baked socket path — a correctness and isolation problem. With it, the readiness ping stays a simple, per-sandbox, idempotent call. (The snapshot-restore mechanics are in /blog/how-firecracker-boots-fast.)

The security posture: the agent is a privileged surface

Now the part that matters most. This guest agent runs inside the VM, usually as root, and does exactly the things an attacker escaping a sandbox would love to do: run arbitrary commands, read and write arbitrary files. It is, by construction, the most privileged surface reachable from outside the guest. Treat it that way. The channel is untrusted-adjacent by design, so the protocol and its implementation are load-bearing security, not plumbing you can be sloppy with.

• Keep the protocol minimal. Every op is attack surface. Resist the feature that lets exec take a raw shell string with unbounded interpolation, or fs.read take '..' traversal you never normalize, or an op that reconfigures the agent at runtime. The smaller the protocol, the smaller the blast radius. A control channel is a place to say no.
• Authenticate the caller — vsock is transport, not auth. Anything on the host that can open the UDS can reach the guest's port; the link proves nothing about who is calling. PandaStack layers authentication on top by validating injected ed25519 keys, so the agent only obeys a caller that holds the key baked in for that sandbox, not merely anyone who reached the socket.
• Don't let the agent become an escape hatch. The agent's job is to serve the host, not to give the guest a new way to reach the host or a neighbor. It should never proxy guest-initiated connections outward, never expose the host UDS to guest code, and never widen what the host can be tricked into doing. vsock's point-to-point nature helps — a guest can't address a sibling — but the agent must not undo that.
• Validate every argument at the boundary. Normalize and confine paths, cap sizes, enforce timeouts, and reject malformed frames instead of trying to 'recover' from them. A frame with a 4 GB length prefix is an attack, not a large file — bound the frame size and hang up.
• Fail closed. If a request is unrecognized, unauthenticated, or malformed, return a structured error and drop the connection; never guess at intent. An agent that tries to be helpful about ambiguous input is an agent that can be steered.

There is also a pragmatic fallback worth naming. Sometimes you want a raw interactive shell — a PTY for a terminal, an rsync, a tool that speaks SSH natively — and re-implementing all of that inside your framed RPC is a poor use of anyone's time. The tidy move is SSH-over-vsock: run an sshd inside the guest bound to the vsock port (or tunnel SSH through the same bridge) and let SSH's own well-audited auth and channel multiplexing carry the heavy interactive cases, while your minimal JSON RPC handles the fast structured ops. PandaStack does exactly this split — the framed vsock protocol for exec/fs/readiness, an SSH bridge (with injected ed25519 keys) as the fallback for the paths where a real shell is easier than a custom message.

What this looks like from the SDK

All of that framing, streaming, restore-override, and auth is plumbing the platform hides. From the SDK, the user never sees a vsock socket or a CONNECT handshake — they create a sandbox and call methods, and every one of those methods rides the guest-agent bridge underneath. The payoff of getting the protocol right is that this stays boring:

from pandastack import Sandbox

# create() blocks only until the guest agent answers its readiness ping
# over vsock — not until an arbitrary sleep elapses.
sbx = Sandbox.create(template="code-interpreter")

# exec rides the framed vsock RPC: cmd in, {stdout, stderr, exit_code} out.
# It works whether the sandbox has egress or is locked to default-deny,
# because the control channel never touched the guest's network.
r = sbx.exec("python -c 'print(2 ** 10)'")
print(r.stdout)       # "1024\n"
print(r.exit_code)    # 0

# fs.write / fs.read are the fs ops from the protocol, one call each.
sbx.filesystem.write("/tmp/note.txt", "hello from the host")
print(sbx.filesystem.read("/tmp/note.txt"))   # "hello from the host"

sbx.delete()

That `r.stdout` came back over a length-prefixed JSON frame on a vsock socket that Firecracker bridged to a Unix socket, authenticated by an injected ed25519 key, on a per-sandbox UDS that a snapshot restore pointed at moments earlier. The user got to type three lines. That gap — between the amount of plumbing underneath and the amount the user sees — is the whole job. PandaStack's core is open source under Apache-2.0, so you can read the pandastack-init accept loop, the host-side dialer, and the framing for yourself.

The protocol in one breath

A guest-agent control protocol is smaller than people expect and more security-sensitive than they treat it. Carry it over vsock so it needs no guest network and can't cross tenants. Frame it with a length prefix and small JSON messages — ping, exec, fs.read, fs.write, plus a streaming exec — and give every operation a deadline. Make the readiness ping idempotent so the control plane can retry it while a snapshot-restored guest finishes booting, and lean on Firecracker's vsock UDS override so each restore gets its own per-sandbox socket. Then treat the agent as the privileged surface it is: keep the protocol minimal, authenticate the caller, validate every argument, fail closed, and keep an SSH-over-vsock fallback for the interactive cases you shouldn't reinvent. Get that right and the sandbox does exec, files, and readiness over one clean channel while the guest's network stays whatever you decided it should be — which, for untrusted code, is exactly the separation of concerns that makes a sandbox a sandbox.