vsock vs TCP: How Should the Host Talk to a Guest?

You have a microVM. Something inside it — a guest agent — needs to take orders from the host: run this command, read that file, write this one, and above all, tell me the instant you're actually ready to work. The question this post is about is narrower and more practical than "how do microVMs communicate": given that you're going to run an agent inside the guest, how should the host reach it? There are exactly two sane answers. One is virtio-vsock, a socket family that rides the hypervisor transport and needs no guest network at all. The other is a plain TCP socket over the guest's virtio-net TAP — the channel every HTTP client, curl, and load balancer on earth already understands. Both work. They make opposite trade-offs, and picking the wrong one for your control plane is the kind of decision that looks fine in the demo and hurts in production. Let's put them head to head.

The one-line intuition, before the mechanics: vsock is the servants' staircase. It's how the host talks to the guest without going through the front door the guest also uses — the network. TCP-over-TAP is the front door: universal, well-lit, understood by everyone, and also the door your untrusted tenant is standing in. Sometimes you want the staircase; sometimes the front door is genuinely simpler. Which is which depends on what the channel is for.

Two doors into the same guest

A TCP channel over the TAP is the obvious one. The guest gets a virtio-net NIC wired to a host TAP device, that NIC gets an IP (statically or via DHCP), the agent binds a TCP port, and the host connects to (guest-IP, port) like it would to any server. It's just networking. Every tool you own speaks it, and if the agent talks HTTP you get the entire ecosystem — clients, retries, TLS, proxies, health-check probes — for free. The cost is that all of that machinery has to exist and be correct before the channel works: an interface, an address, a route, a firewall posture, and a listening socket on a routable interface.

vsock is the other door. It's a socket address family — AF_VSOCK — for talking between a hypervisor host and its guests, used through the same sockets API you already know (socket, bind, listen, connect, read, write). The only difference is the address: instead of (IP, port) it's (context ID, port). No IP layer, no routing, no ARP, no DNS. CID 2 always means the host; each guest gets its own CID (Firecracker conventionally uses 3). Bytes move over the virtio transport — shared-memory rings between guest and VMM — not over any wire or virtual NIC. The channel is up the moment guest userspace is up, whether or not the guest has a network at all. We wrote the full CID/port primer in /blog/vsock-explained; here we're comparing it to its TCP alternative.

How Firecracker's vsock actually works

The elegant part of Firecracker's design is that on the host side, you don't speak AF_VSOCK at all. Inside the guest, vsock is real vsock. But Firecracker bridges the guest's vsock onto a Unix domain socket (UDS) on the host filesystem. You configure the device with a guest CID and a host UDS path, and Firecracker translates between the two — so host-side code is just plain Unix-socket code, trivially wired into any language.

{
  "vsock": {
    "guest_cid": 3,
    "uds_path": "/var/lib/pandastack/vms/<id>/vsock.sock"
  }
}

The connect convention is the piece worth memorizing. To reach a port inside the guest, a host process connects to that UDS and, as its very first line, writes CONNECT <port>\n to tell Firecracker which guest vsock port to wire it through to. Firecracker replies with an OK and its assigned host-side port, and from there the socket is a transparent byte stream to whatever is listening in the guest. (The reverse works too: when the guest dials the host on CID 2 port N, Firecracker surfaces it on a companion socket named uds_path_N.) With that one handshake you can reach a guest agent using tools you already have:

# Reach guest vsock port 1024 via Firecracker's host-side Unix socket.
# socat opens the UDS; we write the CONNECT handshake, then talk.
( printf 'CONNECT 1024\n'; cat ) | \
  socat - UNIX-CONNECT:/var/lib/pandastack/vms/<id>/vsock.sock
# Firecracker replies 'OK <assigned_host_port>' on success, then
# forwards the byte stream straight to the guest agent. No IP involved.

# The same idea with nc -U (netcat's Unix-socket mode):
printf 'CONNECT 1024\n' | nc -U /var/lib/pandastack/vms/<id>/vsock.sock

The two listeners, side by side

Inside the guest the two options are nearly identical code — the whole difference is one constant. A vsock listener binds AF_VSOCK and accepts from the host (CID 2); a TCP listener binds AF_INET on an IP the guest first had to be assigned. That single swap is the whole trade: the vsock socket needs no address configured on the guest; the TCP socket can't come up until the network does.

import socket

PORT = 1024

# --- Option A: vsock listener (no guest IP required) ---
vs = socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM)
vs.bind((socket.VMADDR_CID_ANY, PORT))  # accept from the host (CID 2)
vs.listen()
# Up the instant guest userspace is — before any network exists.

# --- Option B: TCP listener (needs an IP, a route, a firewall stance) ---
tc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tc.bind(("0.0.0.0", PORT))              # bound on a routable interface
tc.listen()
# Only reachable once the guest has an address and the path is open.

In practice you rarely hand-roll either side. In PandaStack, the guest agent (pandastack-init) runs the vsock accept loop, and the host-side agent dials Firecracker's UDS and does the CONNECT dance for you. From the SDK, you just create a sandbox and call exec — and every one of those calls rides the vsock control bridge under the hood, which is exactly why exec keeps working even on a sandbox whose network is locked to default-deny:

from pandastack import PandaStack

ps = PandaStack()  # reads PANDASTACK_API_KEY

sb = ps.sandboxes.create(template="code-interpreter")

# This exec rides the guest-agent bridge (vsock -> host UDS), not the
# guest's network. It works whether the sandbox has egress or none.
res = sb.exec("python -c 'print(2 ** 10)'")
print(res.stdout)   # 1024

sb.delete()

vsock vs TCP-over-TAP, dimension by dimension

Here's the comparison that actually drives the decision. Read each line as "for this concern, here's what each channel costs you."

• Needs a guest IP? — vsock: No. Addressed by (CID, port); the guest needs no address at all. TCP-over-TAP: Yes. The guest must be assigned an IP (static or DHCP) and have a route before it's reachable.
• Works pre-network / with networking off? — vsock: Yes. It rides the virtio transport, so it's up as soon as guest userspace is, even on a VM with no NIC. TCP-over-TAP: No. Nothing is reachable until the interface, address, and route are configured — a config step that can fail before the guest is ever reachable.
• Snapshot-restore behavior — vsock: The host-side UDS path can be overridden at restore (Firecracker v1.16), so each restored sandbox gets its own socket for a clean per-sandbox readiness ping. TCP-over-TAP: The baked IP/MAC must be re-mapped to the restored instance's netns and TAP, and the port re-probed over the network before it's usable.
• Tooling / HTTP compatibility — vsock: Foreign to most tools; you speak a Unix socket plus a CONNECT handshake, or run an HTTP-over-vsock shim. TCP-over-TAP: Universal — curl, HTTP clients, health-check probes, proxies, and TLS all work unchanged.
• Isolation of the control plane — vsock: Structurally point-to-point to the host (CID 2); a guest can never address a sibling, so the channel can't become a cross-tenant path. TCP-over-TAP: Only as isolated as your network policy makes it; a listening port lives on a routable interface and must be firewalled off from the workload and from neighbors.
• Host-side multiplexing — vsock: Firecracker exposes one UDS per guest and multiplexes ports over the CONNECT handshake, so the host fans out many logical connections over a single per-VM socket path. TCP-over-TAP: Multiplexed the ordinary way — many TCP connections to (guest-IP, port), managed by the host's network stack and whatever connection pooling your client does.

When vsock wins

vsock is the right primitive when the channel is a control plane — management traffic that must work regardless of the guest's network state, and that you never want the guest's own code to be able to see, bind, or reach. That describes almost every sandbox platform's exec / filesystem / readiness path. You want to be able to run a sandbox with its network clamped to default-deny egress — or with no NIC at all — and still exec into it, push and pull files, and know the instant it's ready. Because vsock is orthogonal to networking, all of that keeps working while the network is locked down, and because it's point-to-point to the host, a hostile tenant can't use it to reach a neighbor. The control plane and the network become two independent decisions, which is exactly the separation you want when the workload is hostile by assumption.

The readiness ping is where this pays off most concretely. PandaStack has no warm pool of idle VMs — every create restores a baked Firecracker snapshot, landing a fresh sandbox at roughly 179ms p50 and 203ms p99 (a true first cold boot, before a snapshot exists, is around 3s). The control plane wants to hand the sandbox back the instant it's genuinely ready, and "ready" is best answered by the guest itself. Firecracker v1.16's vsock UDS override at snapshot restore is the piece that makes this compose: a single baked snapshot is restored into many sandboxes, and each restore points the vsock bridge at a fresh per-sandbox UDS, so the host gets a clean per-sandbox readiness ping over the guest's own socket rather than guessing from the network side.

When TCP-over-TAP is simpler

TCP earns its keep the moment the thing you're reaching is already an HTTP service, or is a service you also expose to users. If the guest runs a web app, an API, a database, a Jupyter server — anything a human or another service connects to over the network anyway — then it already has an IP, a port, and a network path. Reaching it over TCP means reusing the ecosystem you didn't have to build: curl for a quick probe, an HTTP client with real retry and timeout semantics, a load balancer for health checks, TLS end to end. Bridging that same traffic over vsock would mean inventing an HTTP-over-vsock shim to re-implement machinery that TCP hands you for free — pure friction with no isolation benefit, because the service is user-facing by design.

So the honest rule of thumb is about intent, not dogma. If the channel is internal management traffic that must survive a locked-down or absent network and must never be reachable by the tenant or a neighbor, use vsock. If the channel is (or is co-located with) a service you're deliberately exposing over the network, use TCP — you already paid for the network, so use it. Many real systems run both at once: vsock for the private control bridge, TCP-over-TAP for the workload's own public ports, cleanly separated. That's precisely PandaStack's shape.

What PandaStack does

PandaStack puts its control plane on vsock and leaves TCP for the workload. The guest agent, pandastack-init, listens on a vsock port; the per-host agent dials Firecracker's UDS to reach it and drives exec, filesystem operations, and the readiness ping over that one link, with an SSH fallback for paths where a raw shell is easier than the framed protocol. That keeps management working no matter what the sandbox's network policy is, and keeps it structurally unreachable from a sibling sandbox. Meanwhile the workload's own ports — the web app you deployed, the preview URL on port 3000 — ride the ordinary virtio-net TAP and are reached over TCP/HTTP like anything else, because those are meant to be reachable. Servants' staircase for the host; front door for the users. PandaStack's core is open source under Apache-2.0, so you can read the pandastack-init vsock loop and the host-side dialer and watch the CONNECT handshake go by yourself.

The takeaway is short. Don't reflexively reach for a TCP port just because it's familiar — ask what the channel is for. A control plane wants a channel that's independent of the guest's network, invisible to the guest's code, and incapable of crossing tenants: that's vsock. A user-facing service wants the universally understood network door: that's TCP. Get that split right and the plumbing disappears, which — for a sandbox — is exactly what good plumbing is supposed to do. For the vsock addressing model in depth see /blog/vsock-explained; for how the whole minimal device model fits together, /blog/firecracker-virtio-devices.