How to profile code running in a sandbox

Someone tells you the job is slow in the sandbox. Your hands already know what to do: shell in, attach a profiler, port-forward the UI, stare at a flamegraph. Inside a Firecracker microVM about half of that works exactly as you expect, and the other half fails in ways that are annoying rather than obvious — the profiler UI you cannot reach, the kernel event that is not compiled in, the run that looked catastrophic because it was the first one.

I'm Ajay; I run PandaStack, a Firecracker microVM platform, which means I spend a lot of time reading other people's profiles and a fair amount of time being told the hypervisor is at fault. It usually isn't. Blaming the hypervisor is the senior-engineer version of blaming the compiler: emotionally satisfying, occasionally correct, and almost never where you should look second. Here is the order I actually work in.

Decide what you're chasing before you attach anything

There are three different questions people mean by "it's slow", and the tools that answer them barely overlap.

• Wall-clock time — how long the whole thing took. This is what the user complained about, and it is the only number that is definitionally true.
• CPU time — how much of that wall clock was your code actually executing instructions. This is what a sampling profiler shows you.
• Blocked time — the gap between the two. Network, disk, locks, a subprocess, a DNS lookup that is quietly taking five seconds twice.

In my experience the large majority of "slow in the sandbox" reports are the third category or a cold cache, not the first two. Somebody profiles CPU, finds nothing interesting, concludes the platform is throttling them, and files a ticket. The one-line triage that would have saved the ticket is `time`, read properly: if real is much larger than user plus sys, your CPU profiler is going to spend thirty seconds telling you your process was idle.

# Is it CPU, or is it waiting? real vs user+sys is the whole question.
/usr/bin/time -v python job.py 2>&1 \
  | grep -E 'wall clock|User time|System time|Maximum resident|context switches'

# Never trust one run. The first one pays costs the others don't.
for i in $(seq 1 6); do
  /usr/bin/time -f "run $i: real=%e user=%U sys=%S" python job.py > /dev/null
done

Run that first. If user plus sys is close to real, you have a CPU problem and a sampling profiler is the right next tool. If it isn't, skip to the syscall section further down — you will find your answer there in a fraction of the time.

Start in-language, in-guest — it needs nothing special from the kernel

Language-level profilers are the first tool for a reason that is specific to this environment: they run entirely in userland. They don't need perf events, don't need PMU counters, don't need a kernel built with anything in particular. Whatever your guest kernel was configured with, `cProfile` works. That property alone makes it the right starting point in a microVM, and it happens to also be where the answer usually is.

The other reason is more mundane. You are already inside the VM. There is no pod boundary, no PID namespace to cross, no sidecar to inject. A sampling profiler that needs to see the target process — py-spy is the obvious one — sees it, because it is running as root in the same guest, on the same process tree. The thing that is genuinely fiddly on a container platform is trivial here.

Python: cProfile for determinism, py-spy for reality

# Deterministic, whole-process, zero extra tooling. Every call is instrumented,
# which is exactly why the numbers shift for call-heavy code — check the
# cProfile docs for what it does and doesn't count before you draw conclusions.
mkdir -p /root/prof
python -m cProfile -o /root/prof/job.pstats job.py

# Sampling instead: attach to something already running, no restart, no code change.
pip install py-spy
PID=$(pgrep -f 'python job.py' | head -1)
py-spy record --pid "$PID" --subprocesses --duration 30 \
  --format speedscope --output /root/prof/job.speedscope.json

# Not slow, but hung? One snapshot of every thread's stack answers it immediately.
py-spy dump --pid "$PID"

Reach for `cProfile` when you can rerun the workload and want exact call counts. Reach for py-spy when the process is already running, when the slow thing only happens in production shape, or when the instrumentation cost of a deterministic profiler is distorting the thing you're measuring. py-spy reads the target's memory via ptrace, so it needs privileges and visibility — inside your own sandbox you have both, which is the nicest thing I can say about profiling in a microVM.

Node: --cpu-prof writes a file, which is the point

# Node writes CPU.<date>.<pid>.<seq>.cpuprofile into the prof dir ON CLEAN EXIT.
node --cpu-prof --cpu-prof-dir=/root/prof --cpu-prof-interval=200 server.js

# Memory shape rather than time shape
node --heap-prof --heap-prof-dir=/root/prof build.js

# For a long-running server, exit it politely or you get no file at all
kill -SIGINT "$(pgrep -f 'node server.js' | head -1)"
ls -la /root/prof

Go: pprof over loopback, or straight from the test binary

// Add this to a long-running Go service and the profile becomes a fetch.
// Bind loopback only — inside the guest that is all you need, and it keeps
// the endpoint off the sandbox's network entirely.
import (
	"net/http"
	_ "net/http/pprof"
)

func init() {
	go func() {
		_ = http.ListenAndServe("127.0.0.1:6060", nil)
	}()
}

# 30 seconds of CPU samples, written to a file inside the guest
curl -o /root/prof/cpu.pprof 'http://127.0.0.1:6060/debug/pprof/profile?seconds=30'
curl -o /root/prof/heap.pprof 'http://127.0.0.1:6060/debug/pprof/heap'

# Blocked-on-something, which is the case a CPU profile will never show you
curl -o /root/prof/block.pprof 'http://127.0.0.1:6060/debug/pprof/block?seconds=30'

# No server involved: profile a benchmark directly
go test -run '^$' -bench BenchmarkParse -benchtime 10x \
  -cpuprofile /root/prof/cpu.pprof ./internal/parser

Notice what all three of those have in common: the output is a file on the guest filesystem. That is deliberate, and it is the single most important habit for profiling anything ephemeral.

The artifact is what survives the sandbox

The instinct from long-lived infrastructure is to run the profiler's viewer where the code is and connect a browser to it. Don't. A sandbox is ephemeral by design, port-forwarding a profiler UI is a tunnel you have to build and then tear down, and the moment the sandbox goes away so does everything you were looking at.

Write the profile to a file in the guest, then pull it out with the filesystem API. Analyse it on your laptop, where you already have the tooling, where the file can sit in a directory next to last week's profile for comparison, and where nothing expires. The sandbox's job is to produce the artifact; your machine's job is to read it.

from pandastack import Sandbox

# 1. A sandbox with enough TTL to actually finish the work.
sbx = Sandbox.create(template="base", ttl_seconds=900)
sbx.exec("mkdir -p /root/prof && pip install py-spy flameprof")
sbx.filesystem.write("/root/job.py", open("job.py").read())

# 2. Freeze the ready state. Every future profiling session starts here
#    instead of re-running pip install and wondering why run 1 is slow.
snap = sbx.snapshot()

# 3. Warm-up run, discarded. Then N measured runs.
sbx.exec("python /root/job.py")
for i in range(5):
    sbx.exec(f"python -m cProfile -o /root/prof/run{i}.pstats /root/job.py")

# 4. Pull the artifacts out before anything is torn down.
for i in range(5):
    data = sbx.filesystem.read(f"/root/prof/run{i}.pstats")
    open(f"run{i}.pstats", "wb").write(data)

# Want to profile three variants of the same code without three setups?
# Fork the ready sandbox — a same-host fork lands in 400-750ms.
variant = sbx.fork()
variant.exec("python -m cProfile -o /root/prof/variant.pstats /root/job.py")
variant.destroy()

sbx.destroy()

That snapshot in step 2 is the part worth stealing even if you never touch PandaStack. Installing a profiler is setup, not measurement, and paying for it on every session is how you end up with a first run that is slower than the rest for reasons that have nothing to do with your code.

Turn the artifact into a flamegraph locally

# Python: pstats -> SVG
pip install flameprof
flameprof run3.pstats > run3.svg

# Or open the speedscope JSON py-spy produced. speedscope runs locally;
# the profile never leaves your machine.
npx speedscope job.speedscope.json

# Node: a .cpuprofile loads straight into Chrome DevTools
# (Performance -> Load profile), or the same viewer as above
npx speedscope CPU.20260830.114500.312.0.cpuprofile

# Go: pprof ships its own web UI, flame graph included
go tool pprof -http=:8081 cpu.pprof

Read the flamegraph width-first and ignore the depth. Width is time; depth is just how many frames deep the call chain went. And when it tells you 98% of your time is in `main`, that isn't an answer, it's the profiler being pedantically correct — collapse or re-root the graph on the frame below it until you find the widest box that names something you actually wrote.

When it isn't CPU: strace, and the three patterns

If your triage said real is much bigger than user plus sys, the time is going somewhere outside your process, and `strace` will show you exactly where. Start with the summary — `-c` gives you a table of syscalls by total time, which usually names the culprit in one line — and only then read the transcript.

# Summary first: which syscall is eating the wall clock
strace -f -c -o /root/prof/summary.txt python job.py

# Then the transcript, with timestamps (-tt) and per-call duration (-T)
strace -f -T -tt -o /root/prof/trace.txt python job.py

# Pattern 1: a retry storm — the same connect() over and over
grep -c 'connect(' /root/prof/trace.txt
grep -E 'ETIMEDOUT|ECONNREFUSED|EAGAIN' /root/prof/trace.txt | head -20

# Pattern 2: a DNS timeout — UDP traffic to :53 with multi-second -T values
grep -E 'sendto|recvfrom|poll' /root/prof/trace.txt | grep -E '<[0-9]{1,2}\.' | head

# Pattern 3: a file opened in a loop — same path, hundreds of times
grep 'openat(' /root/prof/trace.txt | sed 's/.*"\(.*\)".*/\1/' \
  | sort | uniq -c | sort -rn | head -20

Those three cover most of what I see. A retry storm looks like a slow service and is actually a client with an aggressive retry policy pointed at something that is refusing fast. A DNS timeout has a very distinctive shape — several seconds of nothing, twice, always the same amount — and is often a resolver config issue rather than a network one. And a file opened in a loop is the classic accidental-quadratic: a config or a certificate bundle being re-read on every iteration, invisible in a CPU profile because it's all syscall time.

`ltrace` is the same idea one layer up, for library calls rather than syscalls. It's useful when you suspect a specific C library is the hot path, it is considerably more invasive than strace, and it is not installed by default on most images. I reach for it maybe once a year.

perf inside a microVM: verify, don't assume

This is the section where I have to be honest rather than helpful. Kernel-level profiling in a microVM is the one place where the environment genuinely does behave differently from a bare-metal box or a container on a general-purpose host, and no amount of confident blogging changes that.

Two things are going on. First, a microVM guest kernel is built for boot speed and a small attack surface. Ours is Linux 5.10 under an Ubuntu 24.04 userland, and like most minimal guest kernels it is not compiled with every tracing and event subsystem a bare-metal profiler assumes is present. Second, and more fundamentally, hardware performance-monitoring counters are generally not exposed to the guest. The PMU is host hardware; the hypervisor does not hand it through. So the events that a hardware profiler wants — cycles, instructions, cache misses, branch mispredictions — are the ones most likely to be unavailable.

What that leaves you is software events and timer-based sampling, which are the reliable path. `cpu-clock` and `task-clock` are software events driven by a timer rather than the PMU, and they are enough to get a sampled call-graph profile. You just have to ask for them explicitly instead of letting perf default to `cycles` and reporting that it found nothing.

# Check what this guest kernel actually offers before you plan around it.
perf list | grep -iE 'hardware|software|cpu-clock|task-clock'

# Loosen the sampling restriction (root, inside your own sandbox)
sysctl -w kernel.perf_event_paranoid=1

# Software event, not the PMU default — this is the one that tends to work
perf record -e cpu-clock -F 99 -g -o /root/prof/perf.data -- python job.py
perf report -i /root/prof/perf.data --stdio | head -40

# Fold to a stack-collapsed text file you can turn into a flamegraph at home
perf script -i /root/prof/perf.data > /root/prof/perf.folded

Run the `perf list` line first, on your actual guest kernel, and believe what it prints. Kernel configs vary between platforms and between kernel versions on the same platform; anything I tell you about what is available would be a claim about my build, not yours.

And here is the practical conclusion, which took me longer to accept than it should have: if you find yourself recompiling a guest kernel to get a profiler working, you have almost certainly lost the plot. The in-language profiler would have given you the answer an hour ago. Kernel profiling is the right tool when the question is genuinely about kernel time — and when it is, the honest move is usually to reproduce that workload on a machine where you control the kernel, rather than fighting the microVM.

The measurement traps that are specific to this environment

Four things will make a profile lie to you inside a sandbox, and three of them are about the first run.

Memory is faulted in on demand. A restored microVM does not copy its whole memory image up front; pages arrive as the guest touches them. That means the first execution of a code path pays a page-in cost that the fifth one does not. If you profile invocation one and compare it against invocation five, you are measuring the memory system, not your code. Always discard at least one warm-up run, and say in your notes how many you discarded.

A snapshot-restored VM starts warm-ish, not cold. The page cache in the guest is whatever it was at the moment the snapshot was baked. That is usually a gift — it is a large part of why restore-on-create lands around 179ms at p50 rather than the roughly 3 seconds a genuine cold boot takes — but it also means your "cold start" measurement is not the cold start you think it is. If you want a real cold-cache number, drop the caches explicitly in the guest before the run and say that you did.

Boot time is not run time. If you are timing from the API call that creates the sandbox, you are including create latency in whatever number you report. Time the workload inside the guest, with the guest's own clock, and keep the create measurement as a separate figure.

And watch the guest clock across a suspend. A VM that was paused and resumed can have a clock that needs to catch up, and a monotonic-clock reading taken across that boundary is not something you want to build a benchmark on. Take your timings within one continuous run of the guest, and be suspicious of any duration that spans a pause.

Which tool for which question

• cProfile (Python) — Best for: exact call counts on a workload you can rerun from the start. Cost: deterministic instrumentation on every call, so call-heavy code shifts shape; check the cProfile docs before quoting its numbers as absolute.
• py-spy (Python) — Best for: a process that is already running, or one you can't modify. Cost: needs ptrace visibility of the target, which you have inside your own sandbox and rarely have across a container boundary.
• node --cpu-prof / --heap-prof — Best for: a Node service or build step, with zero dependencies added. Cost: the file is only written on a clean exit, so a killed process gives you nothing.
• Go pprof — Best for: anything in Go, including the blocked-on-something case that CPU profiles miss entirely. Cost: a few lines of import in the service, or a test binary.
• strace — Best for: time spent outside your process — retries, timeouts, files reopened in a loop. Cost: substantial slowdown; good for patterns, useless for durations.
• perf — Best for: genuine kernel-time questions. Cost: depends on the guest kernel config and generally cannot use hardware PMU counters in a VM; verify with `perf list` before you build a plan around it.

The workflow I'd standardise on

1. Triage with `time` first. Real versus user plus sys decides whether you want a CPU profiler or strace, and it costs you one run.
2. Create a sandbox with a TTL long enough for the whole session — profiling with a two-minute TTL is a way to lose an artifact.
3. Install the profiler once and snapshot the ready state, so setup cost never lands inside a measurement again.
4. Run the workload N times and throw the first one away. It paid page-in costs the others didn't.
5. Write every profile to a file in the guest, then pull them out with the filesystem API before anything is destroyed.
6. Analyse locally. Flamegraph, diff against last week's run, keep the artifact in the repo next to the fix.
7. Destroy the sandbox. The snapshot is the thing you keep, not the VM.

None of this is exotic. It's the ordinary profiling loop with one adjustment: the environment is disposable, so the artifact has to be the deliverable rather than the session. Get that right and profiling in a microVM stops being a special skill and goes back to being the boring, effective thing it is everywhere else — and you can go back to blaming the compiler.