Firecracker's vsock-over-UDS Handoff, Explained

AF_VSOCK is the guest and the host passing notes under the door — no network required. That's the promise. But there's a delightful wrinkle in how Firecracker actually delivers it: on the host side, you never touch AF_VSOCK at all. Firecracker takes the guest's vsock device and projects it onto a plain Unix domain socket (UDS) sitting on the host filesystem. Your host code opens that file like it would any Unix socket, speaks a tiny one-line handshake, and from there has a transparent byte pipe into a process inside the guest — one that has no IP, no NIC, and no idea a network exists. This post is about that handoff specifically: the exact bytes on the wire, the two directions it runs in, why it's the right substrate for a guest-agent control channel, and the one v1.16 feature that keeps the whole thing from falling apart the moment you restore the same snapshot into two sandboxes at once. If you want the CID/port fundamentals first, /blog/vsock-explained covers the addressing model; here we're zoomed all the way in on the UDS bridge.

The UDS bridge: vsock inside, Unix socket outside

Inside the guest, vsock is real vsock. A process binds an AF_VSOCK socket on (CID, port) and listens; the guest kernel's virtio-vsock transport driver carries bytes to the VMM over shared-memory rings. Nothing unusual. The interesting design decision is entirely on the host side. Firecracker's vsock device is configured with two things: a guest CID and a host-side uds_path — a filesystem path where Firecracker creates (and listens on) a Unix domain socket. That single socket file is the whole host-facing surface of the guest's vsock device.

{
  "vsock": {
    "guest_cid": 3,
    "uds_path": "/var/lib/pandastack/vms/<id>/vsock.sock"
  }
}

So the host never has to know AF_VSOCK exists. It opens uds_path with ordinary AF_UNIX socket code — the same three lines you'd write to talk to any local daemon — and Firecracker does the translation between that Unix stream and the guest's vsock stream underneath. That's a bigger convenience than it sounds. AF_VSOCK support on the host is uneven across languages, container runtimes, and older kernels; AF_UNIX is universal and boring. By putting the boring, portable thing on the host and keeping the vsock semantics inside the guest, Firecracker makes the control channel trivially wireable from anywhere — socat, netcat, Go, Python stdlib, whatever you've got.

Host-initiated: connect to the UDS, send CONNECT <port>

Here is the exact protocol for the host reaching into the guest. A host process connects to the uds_path Unix socket. Before anything else, as its very first line, it writes an ASCII command: CONNECT <port>\n — where <port> is the guest vsock port you want to be wired through to. Firecracker parses that line, opens the corresponding vsock connection to the listener inside the guest, and replies on the same socket with OK <assigned_host_port>\n. After that OK line, the socket is a transparent, bidirectional byte stream straight to the guest process. Everything you write goes to the guest; everything the guest writes comes back. Firecracker is out of the way apart from copying bytes.

Because it's just a Unix socket plus one line of ASCII, you can drive it with tools you already have. Here's the handshake with socat and with netcat's Unix-socket mode, reaching a guest agent listening on vsock port 52:

# Reach guest vsock port 52 through Firecracker's host-side UDS.
# socat opens the Unix socket; we write the CONNECT line first, then talk.
( printf 'CONNECT 52\n'; cat ) | \
  socat - UNIX-CONNECT:/var/lib/pandastack/vms/<id>/vsock.sock
# On success Firecracker replies 'OK <assigned_host_port>' on the same
# stream, then forwards everything after it straight to the guest.

# Same idea with nc -U (netcat's AF_UNIX mode):
printf 'CONNECT 52\n' | nc -U /var/lib/pandastack/vms/<id>/vsock.sock

# The reply looks like:  OK 1073741824
# Anything you send AFTER that line is a raw byte pipe to guest port 52.

Guest-initiated: the uds_path_<port> companion socket

The bridge runs in the other direction too, and the naming convention is the part people miss. When a process inside the guest wants to call out to the host, it opens an AF_VSOCK socket and connects to (CID 2, port N) — CID 2 always being the host. Firecracker catches that outbound connection and surfaces it on the host as a connection to a Unix socket whose path is the base uds_path with an underscore and the port number appended: uds_path_N. For a guest dialing host port 52 with the config above, that's /var/lib/pandastack/vms/<id>/vsock.sock_52.

The crucial and easy-to-forget implication: for a guest-initiated connection to succeed, a host process must already be listening on that uds_path_N socket before the guest dials. Firecracker doesn't queue the guest's connection waiting for a host listener to show up — if nothing is accepting on uds_path_52, the guest's connect fails. So the pattern for host-bound guest traffic is: on the host, bind and listen on uds_path_<port> first; then let the guest connect to (CID 2, <port>). It's the mirror of the host-initiated flow, minus the CONNECT line — the port is encoded in the filename instead of in a handshake.

• Host -> guest — Host connects to uds_path, writes 'CONNECT <port>\n', reads 'OK <n>\n', then streams to the guest listener on that vsock port. The port lives in the handshake.
• Guest -> host — Guest connects to (CID 2, port N). Firecracker surfaces it as a connection to uds_path_N on the host, where a host process must already be listening. The port lives in the filename.

Why a UDS bridge beats a TCP port for a control channel

You could, instead, give the guest a virtio-net NIC, an IP, and a listening TCP port, and reach the agent over the network. It works, and for a user-facing service it's the right call. For a control channel into untrusted code it's the wrong one, and the vsock-over-UDS handoff is why. The control channel has requirements a TCP port fights against: it must be reachable before the guest's network converges (you want to ping for readiness the instant userspace is up, not after DHCP and routing settle); it must keep working when the guest's egress is clamped to default-deny or the NIC is removed entirely; it must never add a port the workload's own code can see, bind, or collide with; and it must be structurally incapable of reaching a sibling sandbox. The UDS bridge gives all four for free, because the channel lives on the host filesystem and the hypervisor transport, not on any network the guest participates in. We put vsock and TCP head to head in /blog/virtio-vsock-vs-tcp-guest; the short version:

• Boot-order independence — vsock-over-UDS: Up the instant guest userspace is, because it rides the virtio transport, not a NIC. TCP-over-tap: Unreachable until the interface, IP, and route are configured — a multi-step setup that can fail before the guest is reachable. SSH-over-tap: Same network prerequisites as TCP, plus sshd startup and host-key generation.
• Isolation — vsock-over-UDS: Point-to-point to the host; the guest can address only CID 2, never a neighbor, and the UDS is a host-filesystem path no guest code can see. TCP-over-tap: A port on a routable interface; isolation is only as good as your firewall/netns policy, and a misconfigured bridge can expose it to siblings. SSH-over-tap: Same network exposure as TCP; you're relying on SSH auth plus network policy.
• Setup cost on the host — vsock-over-UDS: Open a Unix socket, write one CONNECT line. Universal AF_UNIX code, no AF_VSOCK support needed on the host. TCP-over-tap: Assign IP, wire tap into a netns, set routes and firewall, then connect — plus re-map all of that on snapshot restore. SSH-over-tap: All of the TCP setup, plus key management and an SSH client.

That is exactly why PandaStack runs its guest agent's control channel over this bridge. The per-host agent dials Firecracker's uds_path, does the CONNECT dance, and drives exec, filesystem read/write, and the readiness ping over that one Unix socket — so management keeps working whether the sandbox has full egress or is locked to default-deny, and a hostile tenant can't use the channel to reach a neighbor. The workload's own ports ride the ordinary virtio-net tap over TCP, because those are meant to be network-reachable. Servants' staircase for the host, front door for the users.

The readiness-ping pattern

The single most valuable thing you send over this bridge is the smallest: a readiness ping. The control plane needs to know the exact instant the guest is up and able to take orders, and the honest way to know that is to ask the guest itself, not to guess from the outside by probing a port and hoping. So the guest agent binds its vsock port early in boot and answers a trivial ping with a ready reply; the host connects to the UDS, sends CONNECT <port>, and pings in a short retry loop until it gets that reply. The moment it does, the sandbox is genuinely usable — the agent proved it, rather than a TCP SYN-ACK from a kernel implying it.

This is where the UDS bridge earns its keep on boot ordering. Because the channel is up as soon as guest userspace is — no NIC, no IP, no route required — the readiness ping can start succeeding far earlier than any network probe could. In PandaStack there's no warm pool of idle VMs: every create restores a baked Firecracker snapshot. The restore step itself lands in roughly 49ms, and the full create path returns a usable sandbox at about 179ms p50 and 203ms p99 (a true first cold boot, before any snapshot exists, is around 3s). Hitting numbers like that depends on knowing the instant the guest is ready down to the millisecond — which is a vsock readiness ping over this exact bridge, and nothing slower.

Design the ping to be idempotent — a pure liveness check with no side effects — because the host will call it repeatedly while a restored guest finishes coming up. A ping that returns ready twice must mean the same as returning it once; never make readiness a one-shot 'mark as started' that breaks on the retry.

The guest side: a real vsock listener

For completeness, here's the inside half — real AF_VSOCK, because we're inside the guest. Bind to VMADDR_CID_ANY on the agreed port to accept from the host, then accept and serve. You can even prototype it with socat's VSOCK-LISTEN or ncat's --vsock before writing any real agent:

# Inside the guest: listen on vsock port 52 and echo a readiness reply.
# socat form (VSOCK-LISTEN takes the port; it binds CID_ANY):
socat VSOCK-LISTEN:52,fork EXEC:'/bin/echo ready'

# ncat form (nmap's ncat, --vsock mode):
ncat --vsock --listen 52 --keep-open --exec '/bin/echo ready'

# From the host, the matching probe (see the CONNECT handshake above):
#   printf 'CONNECT 52\n' | nc -U /var/lib/pandastack/vms/<id>/vsock.sock
# -> 'OK <n>' from Firecracker, then 'ready' from the guest.

A production agent does more than echo — it frames requests (a length prefix plus a small JSON message works well), dispatches exec and filesystem ops, and authenticates the caller, since vsock is transport and not auth. The framing and security design is its own topic, covered in /blog/firecracker-guest-agent-vsock-protocol. Here the point is just that the transport underneath all of it is exactly this accept loop, reached through exactly the UDS handoff above.

The v1.16 per-restore UDS override — and the collision it prevents

Now the gotcha that ties the whole thing to snapshots. The uds_path is part of the vsock device configuration, and when you take a Firecracker snapshot, that configuration is frozen into the snapshot along with everything else. A naive restore brings the baked uds_path back with it. That's fine if you restore a snapshot once. It is a disaster if you restore the same snapshot into many sandboxes — which is precisely how a snapshot-restore sandbox platform operates. Every restored clone would try to create and listen on the same host-side socket path baked at snapshot time.

Firecracker v1.16 fixes this by letting you override the vsock device's host-side UDS path at snapshot restore. Each restore points the vsock bridge at a fresh, per-sandbox uds_path — vms/<id-A>/vsock.sock for one, vms/<id-B>/vsock.sock for another — so every restored clone gets its own private socket file and the host does a clean, per-sandbox readiness ping and control session on the right guest. No shared path, no aliasing, no bind conflict. It's a small VMM feature with an outsized role: it's the single piece that makes 'restore one baked snapshot into thousands of sandboxes' and 'reach each one over its own vsock control channel' compose without stepping on each other. PandaStack relies on exactly this override so concurrent restores never collide on the control socket — it's a load-bearing detail of the boot path, not a nice-to-have. (The snapshot-restore side of the story is in /blog/how-firecracker-boots-fast.)

The handoff in one breath

Firecracker exposes the guest's vsock device to the host as a Unix socket at uds_path. To reach into the guest, the host connects to that socket and writes CONNECT <port>\n; Firecracker answers OK <n> and then pipes bytes to the guest's vsock listener. To let the guest call out, the host listens on uds_path_<port> and the guest connects to (CID 2, port). This beats a TCP port for a control channel on boot-order independence, isolation, and setup cost, which is why a guest agent's exec/filesystem/readiness traffic belongs here and the workload's own ports belong on the tap. Lean on the vsock readiness ping to know the instant the guest is up — and on Firecracker v1.16's per-restore UDS override so that when you restore one baked snapshot into many sandboxes, each gets its own control socket instead of colliding on a shared path. Get that handoff right and the guest gets a clean, network-independent control channel while its actual network stays whatever you decided it should be — which, for untrusted code, is the whole game. PandaStack's core is open source under Apache-2.0, so you can read the host-side dialer, the CONNECT handshake, and the guest listener for yourself.