Firecracker Guest DNS Resolution Explained

"It's always DNS" is a cliché precisely because it keeps being true, and microVMs give the cliché fresh material. The report arrives looking like a network outage: the sandbox boots, the route table is right, it pings its gateway, it pings 1.1.1.1, and then every single thing the workload tries to do fails with something in the family of "temporary failure in name resolution". Nothing is down. There is simply nobody in that guest whose job it is to turn a name into an address.

I'm Ajay, I built PandaStack. This post is about how DNS actually works inside a Firecracker guest — which is to say, how it works in the guest's userspace, because Firecracker itself has no opinion on the matter whatsoever — and about the snapshot-specific failure modes that make DNS in a restore-based platform weirder than DNS in a container.

Firecracker does not do DNS. It does not do DHCP either.

Start by deleting the mental model you brought from Docker. When you run a container, the runtime writes a resolv.conf into the container's filesystem and usually points it at a resolver the daemon runs for you. That's a runtime convenience, not a property of Linux, and Firecracker offers no equivalent.

What Firecracker gives a guest is a virtio-net device exposed over MMIO, backed by a TAP interface on the host. That is the whole contract. Frames go in, frames come out. There is no DHCP server built into the VMM, no NAT helper, no resolver stub, no /etc/hosts injection, no "magic" address that answers queries. If the guest wants an IP address it either does DHCP against something you ran, or it comes up with a static configuration you baked or passed on the kernel command line. If the guest wants to resolve names, everything about that is guest userspace plus whatever you put on the host end of the TAP.

So "DNS is broken in my microVM" almost never means DNS is broken. It means one of three things: the guest has no resolver configured, the resolver it is configured with is not reachable from inside its network namespace, or the resolver is reachable but the library doing the asking behaves differently than the tool you tested with. Those are three different bugs with three different fixes, and telling them apart takes about ninety seconds if you go in the right order.

Who actually resolves the name

Inside the guest, a call to `getaddrinfo()` walks a chain that most people have never had to look at directly. It is worth looking at, because each link fails differently.

1. The application calls getaddrinfo() — or doesn't. Go's pure-Go resolver, the JVM, and various runtimes implement their own resolution paths and cache results in-process. This is where "curl works but my app doesn't" is born.
2. NSS decides the order. /etc/nsswitch.conf lists sources for the `hosts` database — typically `files dns`, meaning /etc/hosts is consulted before any nameserver. A stale hosts entry beats a perfectly healthy resolver, silently.
3. The stub resolver reads /etc/resolv.conf. glibc parses `nameserver`, `search`, and `options` lines, tries each nameserver in order, and applies its own timeout and retry policy. There is no daemon involved here — this is library code running in your process.
4. The query leaves on UDP/53 (usually). If the response is truncated, the resolver retries over TCP/53. A firewall that allows UDP/53 but blocks TCP/53 produces the delightful failure mode where small answers work and large ones don't.
5. Something on the other end answers. That something is either a resolver you run on the host side of the TAP, or a public resolver out on the internet, or nothing at all — in which case glibc waits out its timeout, retries, and eventually gives up.

Then there are the two complications that account for most of the confusion.

systemd-resolved and the 127.0.0.53 stub

On a modern Ubuntu or Debian guest image, /etc/resolv.conf is frequently a symlink into /run, and its contents are a single `nameserver 127.0.0.53` line. That address is systemd-resolved's local stub listener. Queries go to a daemon inside the guest, which applies its own per-link configuration, its own search domains, its own DNSSEC and DoT settings, and its own cache, before forwarding upstream to whatever it was told about.

This is fine when something is feeding resolved a real upstream — normally a DHCP client or a network manager. In a microVM with static baked networking, frequently nothing is. You then get a guest whose resolv.conf looks populated, whose stub listener is up and answering, and which cannot resolve anything, because the daemon behind the stub has no upstream server configured. `cat /etc/resolv.conf` looks healthy. `resolvectl status` tells you the truth.

musl is not glibc

If your guest image is Alpine-based, the resolver is musl's, and it is a different implementation with deliberately different behaviour. It reads the same /etc/resolv.conf, but historically it has differed from glibc in how it handles search domains and `options ndots`, it issues A and AAAA queries in parallel rather than sequentially, and its timeout and retry policy is its own. It also has no equivalent of glibc's NSS plugin system, so anything that expects a custom `hosts` source will not work.

None of that is a bug in musl. It matters because a resolver setup validated on a glibc image can behave differently on a musl one — particularly around single-label names and short search-domain suffixes — and the symptom shows up in the application rather than in `dig`, since `dig` bypasses the libc resolver entirely and speaks DNS itself.

Three ways to wire it, and what each costs you

There are essentially three designs. Platforms tend to pick one and then discover the trade-off later, so here they are up front.

1. Bake a public resolver into the image

Put `nameserver 1.1.1.1` (or your provider's) into the template's /etc/resolv.conf and be done. It is one line, it works on the first boot, and every guest that ever comes from that template inherits it.

The cost is that you have hardcoded an absolute address into a frozen image. You now cannot see, filter, log, or cache your tenants' DNS traffic without intercepting it, you have made every sandbox depend on an external service you do not run, and you have to allow egress to that address on UDP/53 from every sandbox — which, as the security section below argues, is a wider hole than it looks. Fine for a dev template. Weak for a multi-tenant platform.

2. Run a resolver on the host end of the veth and point the guest at the gateway

Bind a forwarding resolver — dnsmasq, unbound, CoreDNS, whatever you like — to the host-side address of each sandbox's veth pair, and have the guest's resolv.conf name the gateway address rather than a public IP. The guest asks its gateway; the gateway asks upstream; you sit in the middle.

This is the design worth building toward. It gives you a cache close to the guest, a single place to allowlist or block names, a natural audit log, and — the underrated part on a snapshot platform — a resolver address that is a property of the *topology* rather than of the *internet*. With per-sandbox /30 links, the gateway address is structurally stable, so a guest baked pointing at its gateway keeps pointing at something meaningful after a restore. A guest baked pointing at a resolver that lived on some other network does not.

3. Inject resolver config at boot

Three mechanisms, in rough order of how well they survive contact with snapshots:

• Guest agent write — a small agent inside the guest writes /etc/resolv.conf from data it fetches at start-of-life. The most flexible option, and the only one that also works on resume rather than only on boot, which is why it's the one that matters most below.
• MMDS — the host publishes resolver config as guest metadata and a guest-side agent or boot script reads it over HTTP from the link-local endpoint. Clean separation, no image rebuild to change a nameserver. Still requires something in the guest to actually fetch and apply it.
• Kernel command line — the classic `ip=` boot parameter has fields for two DNS servers, and the kernel exposes what it was given via /proc/net/pnp; some images symlink /etc/resolv.conf there. It works, but only on a cold boot, and only if the guest's userspace consults it. A snapshot restore does not re-parse the command line — the kernel already booted, once, at bake time.

Per-namespace networking makes DNS policy per-sandbox

Because each sandbox lives in its own Linux network namespace — its own interfaces, routes, iptables rules, and conntrack — DNS policy is not a global setting. It is a per-slot decision. One sandbox can point at a caching resolver on its gateway; the next can point at an internal resolver that only knows about your private zones; the third can have no nameserver at all.

That third option deserves more respect than it usually gets. "No DNS" is a legitimate and strong isolation primitive. Untrusted code that cannot resolve names cannot reach a hardcoded C2 domain, cannot fetch a payload from a paste site, and cannot use DNS as a covert channel — and it degrades gracefully, because a resolution failure is a normal, well-handled error in every language, unlike a mysteriously dropped TCP connection. If a workload genuinely needs one host, give it an /etc/hosts entry and no resolver at all. The blast radius of a hosts file is exactly one name.

The namespace model underneath this is covered properly in /blog/firecracker-network-namespace-isolation-explained; the short version is that pre-allocating the plumbing (PandaStack keeps 16,384 /30 subnets ready per agent) means each guest's entire network reality is a point-to-point link to its host, and the resolver policy is one more thing hanging off that link.

The snapshot gotchas, which are the real reason this post exists

Everything above is true of any VM. This part is specific to platforms where a create is a restore rather than a boot, and it is where DNS gets genuinely strange.

A Firecracker snapshot captures guest memory and device state. Restoring resumes that memory image. The guest does not boot; it wakes up mid-thought, holding every belief it held at bake time. Its resolv.conf is on the CoW rootfs. Its resolver cache is in RAM. Its open sockets are in RAM. Its systemd-resolved daemon, if it has one, is a running process whose in-memory state — upstream servers, per-link config, cached answers, negative-cache entries — is restored exactly as it was. On PandaStack that restore is roughly 49ms of the ~179ms p50 create, which means the wake is fast enough that nothing in the guest ever notices it happened.

Four concrete failure modes fall out of that:

The baked gateway that no longer exists

A guest baked in one network wakes in another. If its resolv.conf names an absolute address that was reachable at bake time — a resolver on a build network, a colleague's dnsmasq, a private IP from a different VPC — the restored guest sends queries into the void and times out. This is the argument for naming the gateway rather than a specific server: the gateway address is part of the per-sandbox topology, which the platform reconstructs on restore, so it stays meaningful. Absolute addresses are a bet that the world hasn't moved.

The resolver cache that is older than it looks

TTLs are counted against the clock, and a restored guest's clock is itself a restored artifact — it resumes at bake time until something corrects it, which is its own class of bug (see /blog/firecracker-guest-clock-and-time-drift-explained). So a cached A record can be simultaneously expired in reality and valid according to the guest, or the reverse. Negative cache entries are worse: a name that failed to resolve during template bake can stay NXDOMAIN in the guest's cache across every clone made from that snapshot. The fix is unglamorous — flush the caches on resume, before the workload starts.

The long-lived connections that survived the freeze

Anything the guest had open at bake time is still in its socket table on restore: a database pool, a keepalive HTTP connection, a resolver's TCP connection upstream. From the guest's point of view those are established. From the network's point of view they are ancient, on the wrong side of a NAT table that has been reset, aimed at peers that timed them out long ago. Connection pools that never re-resolve their target hostname are the specific hazard — they will keep trying an address that DNS would now answer differently. Bake templates with pools cold, and treat re-resolution on resume as a feature your workload needs rather than an optimization it can skip.

The search domains that leak your infrastructure

A `search corp.internal prod.us-east.example.com` line baked into a template is an information disclosure to anyone who runs code in that sandbox. It names your environments, your regions, and your internal naming convention, and — worse — it makes the guest append those suffixes to failed lookups, so a single mistyped or attacker-chosen name emits queries that carry your internal domain structure to whatever resolver is listening. Bake the minimum. Ideally bake none, and set `options ndots:1` so single-label names don't trigger a search-list walk in the first place.

The debugging recipe, in order

Work outward from the wire. Every step is only meaningful if the previous one passed, and the step where it breaks names the bug. Run this inside the guest.

# 1. Is the link up and addressed at all?
ip -br addr show
ip -br link show

# 2. Is there a default route, and where does it point?
ip route
ip route get 1.1.1.1

# 3. Reach the gateway by IP. No names involved yet.
GW=$(ip route | awk '/^default/ {print $3}')
ping -c 2 -W 2 "$GW"

# 4. Reach the internet by IP. Still no names. If this fails, it is not DNS.
ping -c 2 -W 2 1.1.1.1

# 5. What does the guest THINK its resolver is?
cat /etc/resolv.conf
readlink -f /etc/resolv.conf      # a link into /run means something manages it
command -v resolvectl >/dev/null && resolvectl status   # the truth, if resolved
cat /etc/nsswitch.conf            # 'files' before 'dns' -- check /etc/hosts too

# 6. Ask the resolver directly, bypassing every library.
dig +short +time=2 +tries=1 @"$GW" example.com
dig +short +time=2 +tries=1 +tcp @"$GW" example.com   # TCP/53 blocked separately?

# 7. NOW ask through the C library, the way an application does.
#    dig passing while getent fails is a resolv.conf/nsswitch problem, not a
#    network problem.
getent hosts example.com
getent ahostsv4 example.com

# 8. Only at this point is it fair to blame the application.
curl -sS -o /dev/null -w 'http=%{http_code} dns=%{time_namelookup}s\n' https://example.com/

The read-off is mechanical. Fails at 3 or 4: it is routing or egress policy, not DNS. Passes 6 but fails 7: the network and the resolver are fine and the guest's *configuration* is wrong — resolv.conf, a symlink into a managed file, an nsswitch line, or a stale /etc/hosts. Passes 7 but the application still fails: it is the application's own resolver.

That last case is the one worth internalizing. "curl works but my app doesn't" is almost always a resolver-library difference rather than a network fault. curl uses the system resolver (or c-ares, depending on build). A Go binary may use the pure-Go resolver, which reads resolv.conf but implements its own semantics. A JVM caches lookups in-process under `networkaddress.cache.ttl`. Node caches nothing by default but many of its HTTP clients keep sockets alive past a DNS change. If step 7 passes, stop tcpdumping the wire and go read what your runtime does with names.

DNS is an egress hole and an exfiltration channel

Now the part that gets skipped. A very common egress posture for untrusted code is "block outbound TCP/443, allowlist a few endpoints" — and then, because nothing works without name resolution, UDP/53 is left open to the world. That is a data path out, and it is a well-understood one.

DNS tunnelling works by encoding data into the labels of queries for a domain whose authoritative nameserver the attacker controls. The query walks the normal resolution path, reaches the attacker's server, and the payload arrives. Responses come back in TXT, NULL, or CNAME records. It is slow and it is loud if anyone is looking, but if your policy blocks every other protocol and leaves 53 open to any destination, DNS is the only door — which makes it the door that gets used. The same channel does dual duty as command-and-control.

The fix is not to block DNS. It is to make DNS go to exactly one place you control, and to look at it.

# Root namespace. vh-<id> is the host side of this sandbox's veth pair, so
# traffic arriving on it is traffic leaving the guest.
SBX=8f3a1c02
RESOLVER=10.200.0.1        # your forwarder, on the gateway address

# 1. Allow DNS to the sanctioned resolver only -- both transports. Allowing UDP
#    but not TCP gives you the classic "short answers work, long ones hang".
iptables -A FORWARD -i "vh-$SBX" -p udp --dport 53 -d "$RESOLVER" -j ACCEPT
iptables -A FORWARD -i "vh-$SBX" -p tcp --dport 53 -d "$RESOLVER" -j ACCEPT

# 2. Log what gets refused, rate-limited, BEFORE dropping it. A silent drop
#    tells you nothing; this is how tunnelling attempts become visible.
iptables -A FORWARD -i "vh-$SBX" -p udp --dport 53 \
  -m limit --limit 10/min -j LOG --log-prefix "dns-denied: "

# 3. Drop every other attempt to speak DNS, wherever it is aimed.
iptables -A FORWARD -i "vh-$SBX" -p udp --dport 53 -j DROP
iptables -A FORWARD -i "vh-$SBX" -p tcp --dport 53 -j DROP

# 4. DoT has its own port. DoH hides inside 443 and is NOT stoppable by port
#    matching -- only a real egress allowlist closes that one.
iptables -A FORWARD -i "vh-$SBX" -p tcp --dport 853 -j DROP

# 5. Alternative to dropping: transparently redirect ALL DNS to your resolver,
#    so someone's hardcoded 8.8.8.8 is answered -- and logged -- by you.
iptables -t nat -A PREROUTING -i "vh-$SBX" -p udp --dport 53 \
  -j DNAT --to-destination "$RESOLVER:53"

# 6. Verify from outside the guest, not inside it. A guest can lie about its
#    own config; the FORWARD counters cannot.
iptables -L FORWARD -v -n | grep -E "dpt:53|dpt:853"

Three things to keep in mind about that ruleset. Redirecting all port-53 traffic (rule 5) catches hardcoded resolvers but not DNS-over-HTTPS, which is indistinguishable from ordinary HTTPS at the packet level — if your threat model includes deliberate evasion, port-based DNS policy is a supplement to an egress allowlist, never a replacement for one. Second, once every query passes through your forwarder you can allowlist by name, cache aggressively, and rate-limit per sandbox, all of which are cheap there and impossible anywhere else. Third, log volume is a real cost: rate-limit, or the logging becomes the outage. Broader egress design is in /blog/controlling-network-egress-untrusted-code.

Re-asserting resolver config after a restore

Here is the pattern that makes all of the above durable on a snapshot platform: after the sandbox wakes, write the resolver config, flush whatever cache came along for the ride, and verify resolution through the C library before handing the sandbox to a workload. It costs a few tens of milliseconds and converts an intermittent, environment-dependent bug into a loud, immediate one.

from pandastack import Sandbox

RESOLV = """# re-asserted on restore -- do not bake absolute resolver addresses
nameserver 10.200.0.1
options timeout:2 attempts:2 ndots:1
"""

CHECK = r"""#!/bin/sh
# /etc/resolv.conf is often a symlink into /run on systemd images. Replace it
# with a real file so nothing reclaims it after we wake up.
if [ -L /etc/resolv.conf ]; then
  rm -f /etc/resolv.conf
  cp /work/resolv.conf /etc/resolv.conf
fi

# Flush anything cached before the freeze -- including negative entries, which
# are the ones that outlive their welcome.
command -v resolvectl >/dev/null 2>&1 && resolvectl flush-caches 2>/dev/null
[ -x /usr/sbin/nscd ] && /usr/sbin/nscd -i hosts 2>/dev/null

# Verify through the C library, not through dig -- dig bypasses resolv.conf
# semantics and would pass even when applications cannot resolve.
if getent hosts example.com >/dev/null 2>&1; then
  echo RESOLVE_OK
else
  echo RESOLVE_FAIL
  exit 1
fi
"""

sbx = Sandbox.create(template="base", ttl_seconds=600)
try:
    # The restored guest resumes holding whatever resolver config it had at
    # bake time. Overwrite it before anything in the guest tries to use it.
    sbx.filesystem.write("/work/resolv.conf", RESOLV)
    sbx.filesystem.write("/etc/resolv.conf", RESOLV)
    sbx.filesystem.write("/work/dns-check.sh", CHECK)

    r = sbx.exec("sh /work/dns-check.sh", timeout_seconds=60)
    print(r.stdout, r.stderr, r.exit_code)

    if "RESOLVE_OK" not in r.stdout:
        # Dump the three things that explain ~every failure here.
        print(sbx.filesystem.read("/etc/resolv.conf"))
        print(sbx.exec("ip route", timeout_seconds=15).stdout)
        print(sbx.exec("cat /etc/nsswitch.conf", timeout_seconds=15).stdout)
        raise RuntimeError("guest cannot resolve names after restore")
finally:
    sbx.kill()

The same reasoning applies with more force to forks, because a fork inherits its parent's memory wholesale — cache, sockets, resolver daemon state and all. A same-host fork lands in 400-750ms and a cross-host fork in 1.2-3.5s, and in the cross-host case the child may well wake on an agent whose network topology differs from where the parent was baked. Re-assert, then verify. It is the cheapest step in the whole pipeline.

The short version

• Firecracker gives the guest virtio-net and nothing else — no DHCP, no resolver, no metadata magic beyond MMDS, which serves data rather than answers queries.
• Resolution is guest userspace: nsswitch, then resolv.conf, then glibc's or musl's stub, then possibly a systemd-resolved daemon behind 127.0.0.53 that has its own upstream config and its own cache.
• Point the guest at its gateway and run the resolver there. Baked absolute addresses are a bet that the network hasn't changed since bake time; on a snapshot platform, it has.
• Re-assert resolver config and flush caches on every restore and fork. Cache state, negative entries, and open connections all survive the freeze.
• Bake no search domains you wouldn't publish. They leak your internal naming to anyone running code in the sandbox.
• Debug outward from the wire: link, route, gateway by IP, internet by IP, config, dig, getent, then the app. If getent passes and the app fails, it's the runtime's resolver.
• Treat UDP/53 as egress. Allowlist one resolver, log the refusals, and remember DoH is invisible to port-based policy.

DNS is the layer everyone assumes someone else configured, which is exactly why it keeps being the answer. In a microVM there is no someone else. For the layer beneath this one — TAP, virtio-net, NAT, and baked network identity across restores — see /blog/firecracker-networking-explained.