Honeypots on microVMs: a machine you expect to lose

There is one category of machine whose job is to be broken into. Not hardened into irrelevance, not patched on a Tuesday cadence, not defended — broken into, deliberately, by whoever is walking your address space at three in the morning. A honeypot is infrastructure that exists to lose, and its entire value comes from somebody attacking it while you watch.

That inverts every instinct you have from production. The design question stops being "how do I prevent compromise" and becomes "when this is compromised — and it will be, that's the acceptance criterion — what does the attacker get, where can they go next, and how fast can I hand them a factory-fresh one?" Most deployments answer badly, usually by running the decoy as a container on a host that also does something real.

I'm Ajay; I build PandaStack, which runs Firecracker microVMs as a service, so read this knowing I have a side. I'll be specific about the shape that works for high-interaction decoys, and equally specific about the three ways honeypot projects actually die in practice: the attacker notices, the abuse desk emails you, or nobody ever reads the logs.

Low-interaction and high-interaction are different products

The word "honeypot" covers two things that share almost no engineering. A low-interaction honeypot emulates a service rather than a machine. Cowrie speaks the SSH protocol and presents a fake shell; Dionaea impersonates SMB and swallows whatever gets pushed at it; Conpot answers Modbus like a PLC that isn't there. Behind the socket there's no operating system — just a program that knows what the protocol looks like. Cheap, safe, deployable in bulk, and excellent at volume telemetry: which credentials are being sprayed this week, which CVE the scanners picked up.

Their limit is structural. An emulator only answers questions its author anticipated. Type something slightly off-script into a fake shell and it falls apart — a command that isn't implemented, an error message with the wrong wording, a `uname` that doesn't match the `/proc` the attacker just read. Anyone doing more than running a scanner notices quickly, and the interesting attackers are exactly the ones who do more than run a scanner.

A high-interaction honeypot is the other thing entirely: a real operating system, real services, a real shell. The visitor gets to run their actual tooling, drop their actual second stage, configure their actual persistence, and try their actual lateral movement — and you get to record all of it. That's where the intelligence people want lives: working payloads, live C2 infrastructure, real tradecraft against your specific attack surface rather than the industry average.

A container is the wrong container for this

The reflex is Docker, and the reasons are good: it's cheap, it's packaged, and resetting is `docker rm` then `docker run`. For low-interaction emulators that's genuinely fine — the attacker never gets code execution, so the boundary is barely load-bearing. For high-interaction decoys it's the wrong shape, and the reason fits in a sentence: the attacker who just rooted your honeypot and every other workload on that host are negotiating with the same kernel.

Namespaces and cgroups constrain what a process can see and consume, but every syscall from inside the container lands in the host kernel — the same one running your log shipper, your monitoring agent, and whatever else you told yourself was safely elsewhere on the box. Linux kernel privilege-escalation bugs are a recurring genre rather than a freak event. Here you've installed a machine whose success criterion is that a motivated human obtains root on it, and then put it on a shared kernel. It's a strange thing to do on purpose.

Containers also lose on the deception itself. From inside, a container announces what it is: `/.dockerenv` sitting in the root, container paths in `/proc/1/cgroup`, a PID 1 that is your service rather than an init system, no block devices in `/proc/partitions`, an overlay filesystem in `/proc/mounts`, a suspiciously short `dmesg`. Ten seconds of orienting tells a competent visitor they're in a container, and their next move is either to leave — you learn nothing — or to start looking for the host, which is worse. Neither outcome is the one you funded.

And the reset is shallower than it looks: "fresh container from the same image" resets the process tree and the writable layer, not the kernel, not a shared sysctl someone poked, not a volume you forgot about. If you mounted the Docker socket in to make the box look realistic, or ran it privileged so `iptables` works, you've handed host root to whoever wins. The long form of that argument lives in /blog/why-docker-is-not-a-sandbox.

The shape that fits: one disposable guest kernel per decoy

Give each decoy a Firecracker microVM: its own guest kernel, its own memory, its own virtual disk, under KVM with hardware virtualization. When the visitor gets root, they get root in a kernel that exists only for this decoy and is scheduled for deletion. Reaching your host means finding a VMM bug rather than a Linux kernel bug — a much smaller, better-audited target, and Firecracker's device model is deliberately tiny (virtio-net, virtio-block, vsock, a serial console) precisely so the classic device-emulation escape surface mostly doesn't exist. Four properties fall out:

• A kernel you can afford to lose. Privilege escalation inside the guest is expected, not an incident. The visitor's kernel exploit works, gets them root, and gets them root on a machine whose whole purpose was to be given away.
• Reset that isn't reimaging. The decoy is baked once into a snapshot and restored per deployment, so rotating a burned honeypot is a create, not a rebuild — on PandaStack, p50 179ms and p99 203ms end to end, with the restore step itself around 49ms.
• A network namespace per decoy. Each VM gets its own namespace, veth pair, and tap device, so "this box cannot reach production" is a routing fact rather than a firewall rule you hope is ordered correctly in a shared table.
• Observation from outside the guest. Packet capture on the host side of the veth, and full-machine snapshots that include guest memory, are collected by a machine the visitor doesn't have root on — which is the only kind of evidence worth having here.

The point of a honeypot isn't that the attacker can't win. They're supposed to win. The point is that winning gets them a fake machine, and gets you a copy of their tooling.

Burning a decoy should be cheaper than thinking about it

Here's the operational tax nobody budgets for. The honeypot gets rooted at 02:14. Someone notices on Tuesday. The runbook says reimage, which is half an hour of somebody's afternoon, so it gets scheduled, so it doesn't happen, so the decoy keeps running with an implant on it. That gap — between compromise and rebuild — is where a honeypot stops being a sensor and becomes an asset on someone else's balance sheet.

Snapshot-restore collapses that window. You bake the decoy once — install the services, plant the fake data, create the persona, let it settle — then freeze the running machine to a memory file plus a state file. Every deployment after that restores the frozen machine instead of booting one; the first-ever cold boot of a template is around 3 seconds and you pay it once. When a clean decoy costs a fifth of a second, aggressive rotation stops being a policy you enforce and becomes the laziest available option.

Copy-on-write forking is the other useful primitive. Same-host forks land in 400 to 750ms (1.2 to 3.5s across hosts), which lets you branch a live intrusion: freeze a copy of the machine exactly as it is — implant resident, C2 socket open — while the original session continues uninterrupted. The visitor sees nothing; you get a forensic image of the interesting moment.

from pandastack import Sandbox

# The bait. Baked in at deploy time so that anyone who lands a shell finds a
# machine with a past, not a machine that booted during their port scan.
PERSONA = """#!/bin/bash
set -eu
useradd -m -s /bin/bash deploy
echo 'deploy:Summer2024!' | chpasswd          # weak on purpose; that's the door
mkdir -p /home/deploy/.ssh /srv/backups /opt/billing

# Breadcrumbs. Every one of these leads to another decoy and nowhere else.
cat > /home/deploy/.bash_history <<'H'
ssh deploy@fileserver.internal
mysql -h db01.internal -u billing -p
sudo systemctl restart billing-api
H
cat >> /etc/hosts <<'H'
10.77.4.11  fileserver.internal
10.77.4.12  db01.internal
H

# Backdate everything. Nothing says "sandbox" like a home directory whose
# files were all created four minutes ago.
touch -d '2024-11-03' /home/deploy/.bash_history /srv/backups /opt/billing
chown -R deploy:deploy /home/deploy
"""


def deploy_decoy(decoy_id: str) -> Sandbox:
    """Stand up one high-interaction decoy. Assume it will be rooted."""
    sbx = Sandbox.create(
        template="base",
        ttl_seconds=86_400,        # backstop: no decoy outlives a day unattended
        metadata={"role": "decoy", "decoy_id": decoy_id, "trust": "none"},
    )
    sbx.filesystem.write("/opt/persona.sh", PERSONA)
    sbx.exec("bash /opt/persona.sh", timeout_seconds=120)

    # Mark time. Anything newer than this file is the visitor's doing.
    sbx.exec("touch /opt/.deployed", timeout_seconds=15)
    return sbx


def burn_and_rotate(sbx: Sandbox, decoy_id: str, reason: str) -> dict:
    """Something tripped. Freeze the evidence, then hand out a clean machine."""
    # 1. Snapshot FIRST, before anything else touches the box. This captures
    #    guest memory and disk as they are right now -- implant resident,
    #    sockets open, whatever got decrypted still decrypted.
    snap = sbx.snapshot()

    # 2. Best-effort in-guest collection. Treat every line as a hint, never as
    #    evidence: root disabled HISTFILE hours ago if they were any good, and
    #    root is precisely what we handed them.
    changed = sbx.exec(
        "find / -xdev -newer /opt/.deployed -not -path '/proc/*' "
        "-not -path '/sys/*' -not -path '/run/*' 2>/dev/null | head -500",
        timeout_seconds=90,
    )
    procs = sbx.exec("ps -eo pid,ppid,user,etimes,args --forest", timeout_seconds=15)
    conns = sbx.exec("ss -tunap || netstat -tunap", timeout_seconds=15)
    hist = sbx.exec("cat /home/*/.bash_history /root/.bash_history 2>/dev/null",
                    timeout_seconds=15)

    report = {
        "decoy_id": decoy_id,
        "reason": reason,
        "snapshot": snap,
        "files_changed": changed.stdout.splitlines(),
        "process_tree": procs.stdout,
        "connections": conns.stdout,
        "shell_history": hist.stdout,
        "collection_errors": procs.stderr[-2000:],
    }

    # 3. Destroy. Not "clean" -- destroy. Every rootkit, cron entry, systemd
    #    unit and LD_PRELOAD shim dies with the guest kernel it installed
    #    itself into, because that kernel is a file we are about to unlink.
    sbx.kill()

    # 4. Put a virgin decoy back at the same identity. The address stays live;
    #    the machine behind it has no memory of the previous visitor.
    deploy_decoy(decoy_id)
    return report

Lateral movement that goes nowhere

One honeypot is a tripwire. A deception network is a story. The reason the second is worth building is behavioural: the first thing anyone does after getting a shell is orient and pivot — read `~/.bash_history`, check `/etc/hosts`, try the SSH key they just found. If everything they find leads to another decoy, you get to watch the pivot, which is the most informative phase of an intrusion, and none of it touches anything real.

Per-VM network namespaces make that affordable. Each decoy lives in its own namespace with its own veth pair and tap device, so the "internal network" the attacker discovers is a routed collection of those namespaces rather than a slice of your VPC. There's no rule to misorder and no VLAN to misconfigure, because there's no route from that namespace to production at all. PandaStack pre-allocates 16,384 /30 subnets per agent, which is what decides how large your fake network can get. Mechanics in /blog/firecracker-network-namespace-isolation-explained.

The highest signal-to-noise part of deception isn't the machines, though — it's honeytokens. An AWS access key that does nothing except page you when somebody uses it. A customer row with a unique email address, so a dump tells you which system leaked. A file named `passwords-final-v3.xlsx` wired to a beacon. They cost nothing, need no maintenance, and produce alerts that are true by construction, because no legitimate process ever touches them.

Don't let your decoy become somebody else's problem

Here's the failure mode nobody plans for, because it only happens when the project succeeds. Your honeypot works. Someone roots it, installs their loader, and the box starts doing what it was built to do — scanning, brute-forcing, mining, joining a DDoS — from your IP, your ASN, your abuse contact. You'll find out from your provider, in an email with a ticket number, and the tone will not be congratulatory. In the worse version nobody tells you, and your netblock quietly acquires a reputation that takes months to shed.

Cutting egress entirely is safe and boring: most implants check in, fail, and exit, so you get a sample and no behaviour. The workable middle is default-deny plus a narrow, rate-limited, heavily logged allowance — enough for a loader to fetch its second stage and phone home once, nowhere near enough to be useful as attack infrastructure. Log the drops especially: the destinations a decoy tried and failed to reach are often the best indicator the session produces. The general version is in /blog/controlling-network-egress-untrusted-code; below is the honeypot-shaped one.

# All of this runs on the HOST, inside the decoy's own network namespace.
# The guest cannot see these rules, read them, or argue with them -- which
# matters, because we are about to give a stranger root inside that guest.
NS=ns-decoy-$ID
UPLINK=vh-decoy-$ID              # host side of this decoy's veth pair

# 1. Default deny in all three directions, before anything else exists.
#    A honeypot that fails open is not a honeypot, it is a donation.
ip netns exec "$NS" nft -f - <<'EOF'
table inet decoy {
  chain input   { type filter hook input   priority 0; policy drop; }
  chain output  { type filter hook output  priority 0; policy drop; }
  chain forward { type filter hook forward priority 0; policy drop; }
}
EOF

# 2. Inbound: exactly the bait and nothing else. This is the only door,
#    and we want it to be the door they came through.
ip netns exec "$NS" nft add rule inet decoy input tcp dport { 22, 445, 3306 } accept
ip netns exec "$NS" nft add rule inet decoy input ct state established,related accept

# 3. Outbound: the actual decision. Let DNS out (log every query -- the
#    domains are frequently the best IOC in the run), let a trickle of
#    HTTP(S) out so a loader proceeds past check-in, and slam the door on
#    everything honeypots get abused for.
ip netns exec "$NS" nft add rule inet decoy output udp dport 53 \
    log prefix "decoy-dns " accept
ip netns exec "$NS" nft add rule inet decoy output tcp dport { 80, 443 } \
    limit rate 20/second burst 40 packets accept
ip netns exec "$NS" nft add rule inet decoy output tcp dport { 25, 465, 587 } drop
ip netns exec "$NS" nft add rule inet decoy output tcp dport { 22, 23, 3389, 445 } drop
ip netns exec "$NS" nft add rule inet decoy output tcp dport { 3333, 4444, 14433 } drop
ip netns exec "$NS" nft add rule inet decoy output log prefix "decoy-drop " drop

# 4. A hard ceiling on top of the ruleset, because rulesets have bugs and
#    "our nftables config was subtly wrong" is not an argument an abuse desk
#    has ever accepted. 2 Mbit out, per decoy, no exceptions.
tc qdisc add dev "$UPLINK" root tbf rate 2mbit burst 32kbit latency 400ms

# 5. Capture on the HOST side of the veth. A process root-inside-the-guest
#    cannot see, kill, or even detect. This is your evidence; anything you
#    collect inside the VM is a hint.
tcpdump -i "$UPLINK" -s0 -U -w "/var/log/decoy/$ID/wire.pcap" &

# 6. Alert on the shape of the traffic, not just its content. A decoy that
#    opens thousands of outbound connections a minute is not "interesting
#    telemetry", it is a machine that needs to be burned in the next second.
conntrack -E -e NEW -o timestamp | awk -v id="$ID" '{ print id, $0 }' \
  >> "/var/log/decoy/$ID/conns.log" &

Capture the session, ship it where the attacker can't reach

A disposable machine means every question you'll want answered later has to be answered before teardown. Four artifact families matter: the wire (host-side pcap on the veth, plus DNS query logs), the session (commands, keystrokes, uploaded files), the disk (what changed since deployment), and memory (a full-machine snapshot, which is where packed implants helpfully unpack themselves).

The discipline is about where collection happens. Anything running inside the guest runs on a machine the visitor owns: shell history dies to `unset HISTFILE` or `exec bash --norc`, auditd rules can be removed by root, and an in-guest log agent can be stopped, or — much worse — fed. Collect it anyway, because most intruders are lazier than the threat model, but never build a pipeline that depends on it. Host-side pcap and VM snapshots are evidence; `/root/.bash_history` is an anecdote.

Ship it out-of-band — over a path the guest doesn't control and can't see: the host side of the interface, a vsock channel, the serial console, or the snapshot itself. And watch what the shipping path is authorized to do. A honeypot running a log forwarder that holds a write credential for your central SIEM has just handed an attacker a write credential for your central SIEM, which, since a SIEM's entire job is to be believed, is a bad thing to give away.

import { Sandbox } from "@pandastack/sdk";

// Tripwires. None of these would be alarming on a real server. On a decoy
// they are all definitive, because nothing legitimate ever logs into this
// machine -- that asymmetry is the entire product.
const TRIPWIRES = [
  { name: "login", cmd: "grep -c 'Accepted password' /var/log/auth.log || true" },
  { name: "listener", cmd: "ss -ltn | grep -vc ':22\\|:445\\|:3306' || true" },
  { name: "cron", cmd: "ls -1 /var/spool/cron/crontabs 2>/dev/null | wc -l" },
];

// NOTE: polling from inside the guest is convenient and visible -- a root
// user can watch these commands appear in the process table. Host-side
// signals (the egress drop counters, new-connection rate, pcap) are quieter
// and more trustworthy. Use in-guest checks as a backstop, not as the alarm.
async function watchDecoy(sbx: Sandbox, decoyId: string) {
  for (;;) {
    const findings: Record<string, string> = {};

    for (const t of TRIPWIRES) {
      const r = await sbx.exec(t.cmd, { timeoutSeconds: 15 });
      if (r.exitCode === 0 && Number(r.stdout.trim()) > 0) {
        findings[t.name] = r.stdout.trim();
      }
    }

    if (Object.keys(findings).length > 0) {
      // Freeze the machine as it is, page a human, then destroy it. Note
      // what we do NOT do: attempt remediation. You cannot clean a machine
      // you gave away. You can only replace it, and replacing it is cheap.
      const snapshot = await sbx.snapshot();
      await alertOncall(`decoy ${decoyId} touched`, { findings, snapshot });
      await sbx.kill();
      return { decoyId, findings, snapshot };
    }

    await new Promise((r) => setTimeout(r, 30_000));
  }
}

// Fan out: a deception network is many decoys, each in its own microVM and
// its own network namespace. They share a host and nothing else.
const decoys = await Promise.all(
  ["web-03", "fileserver", "db01", "jenkins"].map((id) =>
    Sandbox.create({
      template: "base",
      ttlSeconds: 86_400,
      metadata: { role: "decoy", decoyId: id, trust: "none" },
    }),
  ),
);

await Promise.all(decoys.map((sbx, i) => watchDecoy(sbx, ["web-03", "fileserver", "db01", "jenkins"][i])));

Five ways to host a decoy

Same workload, five topologies. The PandaStack timings are our measured numbers; treat anything about other platforms as "check their docs," because these things change.

• Bare-metal honeypot — isolation: total, no shared kernel and no hypervisor between the decoy and anything else, which is the strongest containment on this list; reset cost: a full reimage, realistically tens of minutes plus a human's attention, so burned decoys stay live far longer than anyone intends; caveat: firmware, BMC, and NIC firmware are persistent state a serious visitor can target, and one physical machine per decoy makes a deception network with any depth to it absurdly expensive.
• Full VM honeypot on ESXi or KVM from a golden image — isolation: hardware virtualization, a genuine boundary, with a rich emulated device model that also happens to look convincingly like ordinary corporate infrastructure; reset cost: revert-to-snapshot, seconds to minutes depending on platform and disk backing — fast enough to be routine, slow enough that teams batch it; caveat: the fat device model is a materially larger hypervisor attack surface than a minimal VMM, and per-VM overhead caps how many decoys you can keep running to make a story believable.
• Container honeypot — isolation: namespaces and cgroups over the same host kernel your other workloads use, so a kernel privilege-escalation bug is a host compromise; reset cost: near-instant, which is honestly its best property; caveat: trivially fingerprinted from inside via /.dockerenv, /proc/1/cgroup, an overlay root and an empty /proc/partitions, so a competent visitor either leaves or starts hunting the host — and the "reset" gives you a new process tree, not a new machine.
• microVM honeypot — isolation: a dedicated guest kernel per decoy behind KVM with a minimal virtio-only device model, so escaping means a VMM bug rather than a Linux kernel bug, and per-VM network namespaces make lateral movement a routing dead end; reset cost: snapshot-restore at p50 179ms / p99 203ms on PandaStack (restore step around 49ms), which makes rotating a burned decoy cheaper than deciding whether to; caveat: that same minimal device model is distinctive from inside, so without a baked persona and randomized identity every decoy in your fleet shares one fingerprint.
• Honeypot- or canary-as-a-service — isolation: someone else's problem, running on someone else's address space, which also moves the abuse-desk exposure off your ASN; reset cost: a click or an API call, operationally trivial and genuinely maintenance-free; caveat: you get the vendor's interaction depth rather than yours — most are low or medium interaction by design — so you learn about attacks on their footprint and their generic services, not about tradecraft aimed at your specific application.

The honest part: three ways this goes wrong

Sophisticated visitors check, and a minimal VM is distinctive

Everything that makes Firecracker a good boundary makes it a recognizable one. There's no BIOS or SMBIOS tables full of plausible vendor strings, because there's no firmware. No PCI bus to enumerate, no GPU, no USB controller, no SATA disk; the device list is short and unusual enough to be a signature by itself, and the CPUID hypervisor bit is set. Snapshot-restore adds tells of its own: implausibly low uptime, an empty process history, and — a specific one we hit in production — a wall clock frozen at bake time until something resyncs it. "Uptime four seconds, system date three weeks stale" is not subtle.

The good news is that honeypots face an easier bar than detonation sandboxes, because real servers are virtual — nobody is surprised to find a VM. What burns you is looking like an analysis box: no user data, no logs predating the current boot, one open port with nothing plausible behind it, a package set that doesn't match the advertised service, a hostname like `honeypot-01`. Bake a persona into the snapshot, randomize MAC, hostname, and machine-id per restore, resync the clock before exposing the box, and vary uptime across the fleet. It's an arms race; budget for it as one, and see /blog/microvm-malware-detonation-sandbox for the deeper anti-analysis version.

The legal and abuse questions are real, and I'm not your lawyer

I'm an engineer, so treat this as a list of things to raise with counsel rather than advice. Recording attacker traffic is interception, and interception rules vary by jurisdiction. The data you collect will contain other people's personal data — stolen credentials, victim files, somebody's compromised mailbox — which is now personal data you hold, with retention and access obligations attached; "we captured it from a criminal" is not an exemption. If your decoy attacks a third party because your egress controls had a gap, that's your outbound traffic. And never hack back.

Practically: get written sign-off before deployment, document a retention period and actually enforce it, keep decoys on address space you're prepared to explain to an abuse desk, and tell your own SOC the thing exists — otherwise the first real detection your honeypot generates will be your incident response team responding to your honeypot, which is funny exactly once.

A honeypot you don't read is just an expensive way to get on a blocklist

This is the one that actually kills projects. Honeypots are enormous fun to build and profoundly boring to operate. Six months in, the alerts route to a channel nobody opens, the pcaps roll off a disk nobody has looked at since the demo, and the only time anyone thinks about the deployment is when the provider emails about outbound scanning. At that point you're running unpatched infrastructure on the internet as a hobby, and paying for it.

So answer three questions before deploying anything: who reads this, on what cadence, and what specifically changes as a result? "We'd get early warning of an intrusion" is only true if the alert reaches someone with the authority and the time to act on it. If you can't name that person, don't build a deception network — deploy honeytokens instead. They fire rarely, fire loudly, need no maintenance, and don't require anyone to read anything on a Tuesday.

When this is worth building

Internal decoys are where high-interaction honeypots pay off most reliably, and it's an argument about base rates. An internet-facing honeypot drowns in scanner noise — you'll learn that the internet is hostile, which you knew. A decoy on an internal segment, with a plausible hostname and nothing legitimate ever connecting to it, produces alerts with a near-zero false positive rate: every authentication attempt against it is, by construction, something that shouldn't be happening.

The external, high-interaction version earns its place when you need tradecraft aimed at your specific surface — your protocol, your customers' credentials — or when the honeypot is the product. Either way, what microVMs change is which parts of the job are expensive. The containment is solved by the shape — a disposable guest kernel per decoy, a network namespace per decoy, egress enforced from outside the guest. The reset is solved by snapshot-restore: rotating a burned honeypot costs less than the round trip that noticed it was burned. What's left is the part that was always the hard part — deciding what story your fake network tells, and having someone who reads what happens when a stranger believes it.