eBPF for sandbox observability: what it can and cannot see

There is one fact about eBPF that decides every argument you will ever have about sandbox observability, and it fits in six words: eBPF observes the kernel it is loaded into. Not the machine. Not the workload. The kernel. Everything else — why Falco sees your containers in glorious detail, why the same probe goes silent the moment you put a microVM in front of it, why "just run Tetragon on the host" is excellent advice for one architecture and a category error for the other — falls out of that sentence.

I'm Ajay; I build PandaStack, a Firecracker microVM platform, so I spend an unreasonable amount of time explaining to security reviewers why I cannot show them guest syscalls from the host, and then explaining to the same reviewers why that is the feature they asked for. This post is that explanation with the hand-waving removed: how eBPF actually attaches and gets data out, what each attachment point can observe across the container and microVM boundaries, what the host can still see about a guest it cannot introspect (which is more than most people assume), and where to put your instrumentation if you run the microVM shape.

What eBPF actually is, precisely

eBPF is a small register-based virtual instruction set with an in-kernel verifier and a JIT. You compile a restricted subset of C (or Rust, or Go via a generator) to eBPF bytecode, load it with the bpf() syscall, and the kernel proves — statically, before it will run a single instruction — that the program terminates, never reads uninitialised memory, never dereferences a pointer it has not bounds-checked, and only calls helper functions permitted for its program type. If the proof succeeds, the JIT compiles it to native code and attaches it to a hook. If the proof fails, you get a verifier log that reads like a hostile code review and no program.

The attachment points are what determine what a program can observe, and they are not interchangeable:

• kprobe / kretprobe — attach to (almost) any kernel function entry or return. Maximum reach, zero stability guarantee: the function you probed is an implementation detail and can be renamed, inlined away, or restructured between kernel releases.
• Tracepoints — static instrumentation points the kernel maintainers placed deliberately, with a documented argument struct. Fewer of them, but they are the ones you should build production tooling on. sys_enter_execve, sched_process_exit, net_dev_xmit, block_rq_issue, kvm_entry all live here.
• fentry / fexit — the modern, cheaper replacement for kprobes on BTF-enabled kernels, using BPF trampolines instead of breakpoint traps.
• uprobe / uretprobe — user-space function entry and return, by binary path and symbol or offset. This is how you trace a TLS library's plaintext before it is encrypted, or an interpreter's function-call path.
• BPF-LSM — programs attached to Linux Security Module hooks, which run at the same authorisation points SELinux and AppArmor use and can return a non-zero errno to deny the operation. Requires CONFIG_BPF_LSM and bpf in the kernel's lsm= boot parameter list.
• XDP and tc (traffic control) — packet processing at the earliest driver hook (XDP) or on the qdisc ingress/egress path (tc), where you can count, mirror, rewrite, redirect, or drop.
• cgroup hooks — attach to a cgroup subtree for socket operations, connect()/bind() interception, sysctl access, and device permission checks. Naturally scoped to a set of processes.

Getting data out is the other half. eBPF programs write into maps: hashes, arrays, per-CPU variants, LRU variants, stack-trace maps, and — for streaming events to user space — the ring buffer (BPF_MAP_TYPE_RINGBUF, available since Linux 5.8), which superseded the older per-CPU perf buffer for most uses because it preserves event ordering and wastes far less memory. A user-space agent polls the ring buffer and does the expensive work: symbolisation, enrichment, shipping. The kernel-side program stays small on purpose.

The verifier is the reason any of this is safe to run in kernel context, and also the reason writing eBPF is occasionally maddening. It enforces a verification complexity budget, so a program whose branch explosion is too large is rejected even though it would terminate. Loops were flatly banned until bounded loops arrived in 5.3, and anything genuinely unbounded still needs the bpf_loop() helper. Tail calls have a depth limit. Helper functions are allowlisted per program type — a tc program cannot call the tracing helpers, and a tracing program cannot rewrite a packet. And crucially, CO-RE (compile once, run everywhere) portability depends on BTF type information being present in the running kernel, which is a build-time decision someone else made. Hold that last point; it comes back.

Containers: one kernel, total visibility, and the reason why

Containers share the host kernel. A container is a process (or process tree) with namespaces restricting what it can name and cgroups restricting what it can consume — a polite suggestion to the kernel about which parts of the machine it should pretend do not exist. When a process inside a container calls execve, the host kernel executes that syscall. The tracepoint fires in the host kernel. A single eBPF program attached once on the host therefore observes every container on the box, with full argument fidelity, attributed per-cgroup for free.

# Every exec on the box, all tenants, one probe, no per-container agent.
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_execve {
  printf("cgid=%-14d uid=%-6d comm=%-16s file=%s\n",
         cgroup, uid, comm, str(args->filename));
}'

# Map a cgroup id back to a container. On cgroup v2 with 64-bit kernfs ids,
# the cgroup id is the cgroup directory's inode number.
find /sys/fs/cgroup -xdev -type d -printf '%i %p\n' | grep -w "$CGID"

# The same idea, but for outbound connections, which is where the
# interesting behaviour usually is:
sudo bpftrace -e '
kprobe:tcp_connect { @conns[cgroup, comm] = count(); }'

This is exactly why Falco, Tetragon, and Cilium are architected the way they are: one privileged host-side loader, probes on shared-kernel events, per-cgroup or per-pod attribution derived from the kernel's own accounting. It is a genuinely excellent design and the visibility is superb. Execs with full argv, file opens with resolved paths, socket connects with the remote tuple, capability checks, namespace transitions, ptrace attempts — all of it, in one stream, with microsecond timestamps.

Now say the quiet part. The reason that visibility exists is that there is exactly one kernel, and that same single kernel is the shared attack surface that makes containers the weaker boundary. Both properties have the identical cause. If you are running hostile code on a shared kernel with host eBPF watching, what you have bought is an excellent, high-resolution recording of the escape — timestamped, argv-complete, and delivered to your SIEM approximately four seconds after it stopped mattering. Detection is not containment. It is very good evidence.

On a shared kernel, observability and vulnerability are the same property viewed from different sides of the incident.

MicroVMs: the same fact, inverted

A Firecracker microVM boots its own Linux kernel inside a hardware-virtualized guest. When a process inside that guest calls execve, the guest kernel executes it. The host kernel is not involved and never learns about it. There is no host-side tracepoint that fires, because from the host's point of view nothing happened except that a vCPU thread of the firecracker process ran some instructions and eventually exited to the hypervisor for a reason like an I/O port access or an interrupt.

Attaching a host kprobe to sys_execve and pointing it at a microVM host is not a partially-working setup. It is a correctly-configured probe on a kernel where the event does not occur. Same for openat, connect, ptrace, and every uprobe you might want: the guest's process memory is, to the host, anonymous memory belonging to the VMM, with no symbols, no ELF mappings the host can resolve, and no cooperation from the guest loader. What the host does see is the machinery of virtualization itself.

# 1. Is this microVM burning CPU? Count VM entries per vCPU thread.
#    You learn the guest is busy. You do not learn what it is doing.
sudo bpftrace -e 'tracepoint:kvm:kvm_entry { @vmentries[pid, tid] = count(); }'

# 2. What is it sending? Bytes out, keyed by network device name.
#    Each sandbox owns a host-side veth (vh-<id>), so the keys are tenants.
sudo bpftrace -e '
tracepoint:net:net_dev_xmit {
  @tx_bytes[str(args->name)] = sum(args->len);
}'

# 3. What is it reading off disk? Block I/O issued by the VMM process.
sudo bpftrace -e '
tracepoint:block:block_rq_issue /comm == "firecracker"/ {
  @io_bytes[pid] = sum(args->bytes);
}'

# What you will NOT see, however long you leave these running:
#   one execve, one openat, one connect, one ptrace from inside the guest.

The boundary that blinds your host probe is the boundary. That is not a consolation prize, it is the entire product. The honest way to frame the trade is not "which stack has better observability" but "what do you want to be true when the sandbox turns out to be hostile." On a shared kernel you get the recording. Behind a hypervisor you get a guest kernel that has to be exploited first, and a VMM that is itself seccomp-filtered down to a small syscall vocabulary — at the cost of a much smaller keyhole to watch through.

What the host can still see, which is more than people assume

"eBPF can't see into the guest" gets repeated until people conclude the host is blind. It is not. The host owns the guest's entire physical world — its network path, its block devices, its memory backing, its CPU time — and every one of those is instrumentable with the ordinary toolkit:

• Per-VM network flows. tc or XDP on the host-side veth, or inside the sandbox's netns on the tap, gives you full 5-tuple flow records, byte and packet counts, DNS queries, TLS SNI, and connection fan-out — everything except which guest process opened the socket.
• Block I/O. block_rq_issue and block_rq_complete filtered to the VMM process give read/write volume, latency distribution, and the working-set behaviour of the guest's disk, per sandbox.
• VMM process lifecycle. The firecracker process is an ordinary host process: fork, exec, exit, signals, OOM kills, thread counts, and file descriptors are all plainly visible.
• KVM exit behaviour. kvm_entry/kvm_exit tracepoints and exit-reason histograms tell you whether a guest is compute-bound, I/O-bound, or spinning on something pathological.
• Memory faulting. If you restore guests via userfaultfd, the page-fault stream is host-side by construction — page-in rate, fault locality, and working-set growth are yours without touching the guest.
• cgroup accounting. CPU time, memory usage, PSI pressure, and throttling for the VMM's cgroup are exact, and they are per-sandbox because the cgroup is per-sandbox.
• Egress enforcement points. Because these hooks are on the host side of the boundary, a compromised guest cannot unload, blind, or lie to them — the strongest property any of this has.

The attribution question — "which tenant is this?" — is where architecture earns its keep. If every sandbox gets its own network namespace with its own veth pair and tap, then the netns is the tenant boundary, and every packet counter, flow record, and conntrack entry inside it is unambiguously one tenant's, with no correlation heuristics and no cgroup-id-to-container-id lookup table that goes stale during churn. PandaStack pre-allocates 16,384 /30 subnets per agent for exactly this reason; the isolation design and the telemetry design turn out to be the same design.

# A per-sandbox egress counter in tc/eBPF. Attach on the host side of the
# veth pair so a hostile guest is on the far side of the instrument.
cat > egress_bytes.bpf.c <<'EOF'
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <linux/pkt_cls.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

struct {
    __uint(type, BPF_MAP_TYPE_LRU_HASH);
    __uint(max_entries, 65536);
    __type(key, __u32);    /* remote IPv4, network byte order */
    __type(value, __u64);  /* bytes */
} egress_bytes SEC(".maps");

SEC("tc")
int count_egress(struct __sk_buff *skb)
{
    void *data     = (void *)(long)skb->data;
    void *data_end = (void *)(long)skb->data_end;

    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)      /* the verifier insists */
        return TC_ACT_OK;
    if (eth->h_proto != bpf_htons(ETH_P_IP))
        return TC_ACT_OK;

    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end)
        return TC_ACT_OK;

    __u32 dst = ip->daddr;
    __u64 len = skb->len;

    __u64 *slot = bpf_map_lookup_elem(&egress_bytes, &dst);
    if (slot)
        __sync_fetch_and_add(slot, len);
    else
        bpf_map_update_elem(&egress_bytes, &dst, &len, BPF_ANY);

    return TC_ACT_OK;
}

char _license[] SEC("license") = "GPL";
EOF

clang -O2 -g -target bpf -c egress_bytes.bpf.c -o egress_bytes.bpf.o

VETH=vh-9f3c1b2a                      # this sandbox's host-side veth
sudo tc qdisc add dev "$VETH" clsact
sudo tc filter add dev "$VETH" ingress bpf da obj egress_bytes.bpf.o sec tc
#            ^ ingress on the host side == egress from the guest

sudo bpftool map dump name egress_bytes

And if you want the same signal without writing any eBPF at all, the netns gives you a great deal for free. Interface counters and conntrack are per-namespace by definition:

NS=ns-9f3c1b2a          # the sandbox's network namespace
VETH=vh-9f3c1b2a        # its host-side veth, in the root netns

# Cheapest possible egress signal, already per-tenant.
ip -s -j link show "$VETH" | jq '{rx: .[0].stats64.rx, tx: .[0].stats64.tx}'

# Live flow table for this sandbox only, without entering the guest.
sudo ip netns exec "$NS" conntrack -L 2>/dev/null | head -20

# Destination-port histogram. Mining pools, exfil, and port scanning all
# have distinct shapes here long before you could name the guest process.
sudo ip netns exec "$NS" conntrack -L 2>/dev/null \
  | grep -oE 'dport=[0-9]+' | sort | uniq -c | sort -rn | head

# Fan-out: one destination is a download, three thousand is a scan.
sudo ip netns exec "$NS" conntrack -L 2>/dev/null \
  | grep -oE 'dst=[0-9.]+' | sort -u | wc -l

In-guest eBPF, and the guest kernel gotcha nobody warns you about

If you want guest syscalls, you load eBPF in the guest. That is not a workaround; it is the only thing consistent with the six-word rule. Run your Falco or Tetragon or custom loader inside the microVM, ship events out over vsock or your guest agent's channel, and you have full in-guest visibility. It works. It costs a bit of guest memory and CPU, and it costs you the property that made host-side probes trustworthy — see the next section.

The practical gotcha is that a stock microVM guest kernel very often cannot run eBPF at all. Firecracker guest kernels are deliberately minimal — the whole point is a small boot surface and a fast boot — and the config options eBPF tooling needs are exactly the kind of thing a minimal config drops. CONFIG_BPF_SYSCALL may be off. CONFIG_DEBUG_INFO_BTF is frequently off, which means no /sys/kernel/btf/vmlinux, which means every CO-RE skeleton you compiled fails to load with an error that does not obviously say "your kernel has no BTF." CONFIG_BPF_LSM needs both the config and the lsm= boot parameter. And on older guest kernels — 5.10 is still a very common Firecracker guest — a good deal of modern tooling simply assumes helpers that do not exist yet.

# Run this INSIDE the guest before you plan any in-guest eBPF work.

# 1. Is the config even readable? (needs CONFIG_IKCONFIG_PROC, often off)
zcat /proc/config.gz 2>/dev/null \
  | grep -E 'CONFIG_(BPF_SYSCALL|BPF_JIT|DEBUG_INFO_BTF|BPF_LSM|KPROBES|FTRACE_SYSCALLS|PERF_EVENTS)=' \
  || echo 'no /proc/config.gz - check the kernel config you built with'

# 2. The single most common silent failure: no BTF, so no CO-RE.
ls -l /sys/kernel/btf/vmlinux 2>/dev/null \
  || echo 'no BTF: CO-RE skeletons will not load on this guest kernel'

# 3. Ground truth on helpers and program types actually available.
bpftool feature probe kernel 2>/dev/null | head -40

# 4. Are tracepoints compiled in at all?
ls /sys/kernel/debug/tracing/events/syscalls 2>/dev/null | head

uname -r

eBPF is not a sandbox, and tenants do not get to load it

eBPF can enforce, not just observe. BPF-LSM programs return an errno from an LSM hook and the operation fails. cgroup/connect4 hooks can refuse an outbound connection. tc programs can drop packets. This is real policy enforcement and it is genuinely useful. It is still not an isolation boundary, and the distinction matters more than the capability does.

A filter over a shared kernel is a filter. It sits at chosen hook points and evaluates chosen conditions, which means its security value is bounded by the completeness of the hook coverage and the correctness of the policy — and it does nothing at all about a bug in kernel code that runs before or beside the hook. A privilege-escalation bug in a netfilter path or a filesystem driver does not care that you attached an LSM program to bprm_check_security. Contrast that with a hypervisor boundary, which is not a decision made at a hook point but a different address space enforced by the CPU. Seccomp-bpf has the same shape, incidentally, and is worth naming precisely: seccomp filters are classic BPF, not eBPF, and they constrain which syscalls a process may issue. Firecracker uses seccomp on its own VMM threads to shrink the VMM's host attack surface — a filter used to harden a boundary, not to be one.

There is also a blunt operational fact that gets skipped: loading eBPF is privileged. It needs CAP_BPF plus a capability appropriate to the program type (CAP_PERFMON for tracing, CAP_NET_ADMIN for tc/XDP), or plain CAP_SYS_ADMIN on older kernels, and unprivileged eBPF is disabled by default on essentially every modern distribution. So "let tenants load eBPF for their own observability" is not a product feature you ship on a shared kernel; it is a privilege grant. In a microVM it is fine and even pleasant — the tenant is root in their own kernel and the blast radius of their verifier-approved program is their own guest — which is a neat inversion of the usual story.

The four observability surfaces, compared

• Host eBPF over containers — Sees: everything, at full syscall fidelity, per-cgroup, from a single privileged loader. Costs: probe overhead on the shared kernel's hot paths, plus kernel-version coupling if you use kprobes. Evasion: cannot be unloaded by the workload, but the workload runs in the same kernel doing the observing, so a successful kernel exploit ends the argument.
• Host eBPF over microVMs — Sees: network flows, block I/O, VMM process lifecycle, KVM exits, memory faults, cgroup accounting. Not a single guest syscall. Costs: near zero on the guest, and no guest cooperation required. Evasion: none — the guest is on the wrong side of the instrument and cannot disable, blind, or falsify it.
• In-guest eBPF — Sees: full guest syscall detail, exactly like the container case, because it is now the same case one level down. Costs: guest RAM and CPU, plus a guest kernel built with BPF_SYSCALL, BPF_JIT and ideally BTF — a real constraint on minimal Firecracker guest kernels. Evasion: a guest-root attacker can unload the program, kill the collector, or feed it nonsense; it observes the environment it is inside.
• Guest-agent telemetry — Sees: whatever the agent chooses to report — process lists, resource usage, application logs, exit codes. Costs: trivially cheap and works on any guest kernel, no BPF config needed. Evasion: it is a self-report, and a compromised guest reports whatever it likes. Excellent for debugging, weak as a security control on its own.

Where to put your instrumentation for a microVM platform

The design that has held up for us is a split by trust, not by convenience. Anything you will use to make a security decision goes on the host, on the far side of the boundary from the workload. Anything you need for depth and debugging comes from inside the guest, and is treated as a report rather than as evidence.

1. Host, per-netns: flow records, byte counters, destination fan-out, DNS and SNI. This is your abuse-detection substrate and it is unforgeable by the tenant.
2. Host, per-VMM-process: cgroup CPU and memory, block I/O volume and latency, KVM exit histograms, process lifecycle and OOM events. This is your capacity, noisy-neighbour and liveness substrate.
3. Host, at the enforcement points: cgroup/connect4 or nftables egress policy, rate limits on the veth, and a kill switch that terminates the VM. Detection without a lever attached is a chart.
4. Guest agent: process table snapshots, application logs, exit codes, in-guest resource usage. Cheap, universally available, and enormously useful for the 99% of incidents that are a customer's build script rather than an attacker.
5. In-guest eBPF: only if you have committed to a guest kernel that supports it, and only for workloads where in-guest syscall detail justifies the kernel-config and boot-size cost.

The reason egress-flow telemetry sits at the top of that list is that it is where the abuse actually is. You do not need syscall visibility to detect the three things that actually happen on a public sandbox platform. Cryptomining looks like a small number of long-lived connections to a stable set of endpoints with a very characteristic small-packet, high-frequency request/response pattern, on a VM pinned at 100% CPU. Exfiltration looks like an outbound byte volume that has no relationship to the inbound volume that preceded it. Scanning looks like connection fan-out to hundreds or thousands of distinct destinations in a short window. All three are visible in flow records alone. None of them require knowing which guest process did it — and in practice, on a per-sandbox network namespace, "which tenant" is the only attribution question you actually need answered.

from pandastack import Sandbox

# In-guest detail is a report from the guest agent, not a host kernel event.
# Useful for debugging; never the sole basis for a security decision.
sbx = Sandbox.create(template="code-interpreter", ttl_seconds=1800)

r = sbx.exec(
    "ps -eo pid,ppid,etimes,pcpu,pmem,comm --sort=-pcpu | head -15",
    timeout_seconds=20,
)
print(r.exit_code)
print(r.stdout)

# The host-side signal for the same sandbox is collected out-of-band from
# its veth and cgroup, on the other side of the boundary, and the guest
# has no way to reach it. Correlate the two; trust only one of them.
#
#   guest says:  "one python process, 3% CPU, nothing to see here"
#   host says:   "1.9 GB out to 41 destinations in the last four minutes"
#
# When they disagree, the host is right.

sbx.kill()

The honest summary

Containers give you a cheap panopticon over a shared kernel. MicroVMs give you a real boundary and a smaller keyhole. Neither is free, and the trade is not a gradient — it is the same architectural fact read from two directions. If your threat model is "my own code, misbehaving," the container panopticon is superb and you should use it. If your threat model is "code I did not write, actively trying to leave," you want the boundary, and you should design your telemetry around what survives on the correct side of it: flows, I/O, resource accounting, lifecycle — plus a guest agent for the depth, held at arm's length.

One last note on cost, since it comes up. Instrumenting the host side costs you nothing in the guest, which matters when the guest is a snapshot-restored microVM that reaches a running state in about 179ms at p50 and may exist for four seconds. There is no warm pool to attach agents to and no long-lived process to bootstrap; every create restores a baked Firecracker snapshot, and the netns, veth and cgroup that your telemetry hangs off already exist before the VM does. Which is a fairly good argument, on its own, for hanging it there.