How to Debug a Hung Process in a Sandbox

A process that has stopped responding is a specific kind of frustrating, because it isn't giving you anything. There's no stack trace, no error line, no exit code. Just a command that hasn't printed since 14:02, and a growing suspicion that restarting it will make the problem go away until it happens again next Tuesday.

I'm Ajay, I build PandaStack — a platform where a lot of the running processes are other people's builds and agent-generated code, so 'why is this stuck' is a question I've had to get systematic about. This is the order of operations that works, from cheapest to most invasive. It applies to any Linux environment; the sandbox-specific parts are called out where they differ.

Step 1: What state is it in?

The Linux process state is the single highest-information character available, and it splits your investigation into three completely different paths. Get it first:

# STAT is the column that matters. WCHAN says what the kernel is waiting on.
ps -eo pid,ppid,stat,wchan:24,pcpu,etime,cmd --sort=-pcpu | head -20

• R (running) with high CPU — it isn't hung, it's busy. You have an infinite loop or genuinely slow work. Go to step 2.
• S (interruptible sleep) with 0% CPU — it's waiting on something: a socket, a lock, a pipe, a child. This is the most common case. Go to step 3.
• D (uninterruptible sleep) — it's blocked in the kernel, almost always on I/O. Go to the D-state section; nothing you send it will help.
• Z (zombie) — it already exited and its parent never reaped it. The bug is in the parent.
• T (stopped) — something sent it SIGSTOP. Usually a debugger that went away.

The WCHAN column tells you the kernel function it's sleeping in, which is often the answer by itself. futex_wait means a userspace lock. pipe_read means it's waiting for a pipe nobody is writing to. wait_on_page_bit means it's waiting on disk.

Step 2: If it's burning CPU, get a stack

A process at 100% CPU is executing something. You want to know what, and the cheapest way is to sample where it is a few times and look for the line that keeps appearing.

Most runtimes have a built-in way to dump a stack on a signal, and it's much friendlier than a native debugger:

# Python 3.3+: faulthandler dumps tracebacks for every thread on SIGABRT
# if the process enabled it. Otherwise attach with py-spy, which needs
# no cooperation from the process at all.
py-spy dump --pid 1234

# Go: SIGQUIT prints goroutine stacks for every goroutine and exits.
kill -QUIT 1234

# Java: thread dump without stopping the process.
jstack 1234

# Node: send SIGUSR1 to open the inspector, then attach a debugger.
kill -USR1 1234

py-spy is worth special mention because it reads the target process's memory from outside — no import, no signal handler, no restart. For a Python process that's stuck right now and can't be modified, it is the single most useful tool available.

If none of those apply, gdb gives you a native backtrace for any process:

# One-shot backtrace of every thread, then detach and leave it running.
gdb -p 1234 -batch -ex "thread apply all bt" 2>/dev/null | head -60

Step 3: If it's asleep, find out what it's waiting for

A process in S state with no CPU is blocked on a syscall. strace attaches and shows you which one — and critically, whether it's making any at all.

# Attach to a running process and every thread of it.
strace -f -p 1234

# Nothing at all after a few seconds means it is blocked in a single
# syscall. The last line printed before the silence IS your answer.

Read the silence, not the noise. These are the patterns worth recognising immediately:

• futex(...) with no return — waiting on a mutex or condition variable. A deadlock, or a lock held by a thread that's itself blocked.
• read(0, ...) — reading stdin. This is the single most common cause of a stuck build: an installer asked an interactive question and nobody is there to answer it.
• connect(...) or recvfrom(...) — waiting on a network peer that isn't answering. Check whether it's DNS by looking at the socket's destination port.
• wait4(...) — waiting for a child that never exits. Move your attention to the child.
• flock(...) or fcntl(..., F_SETLKW, ...) — waiting on a file lock, usually a package manager lock file held by a dead process.

If strace isn't available, the kernel exposes the same information without attaching:

# Which syscall it's in right now (first field is the syscall number).
cat /proc/1234/syscall

# Kernel stack -- the WCHAN column with more detail.
cat /proc/1234/stack

# For Python/Go/Rust binaries, the userspace stack sometimes shows here.
cat /proc/1234/wchan; echo

Step 4: Look at what it has open

File descriptors tell you the shape of the problem when the syscall alone is ambiguous. A hung connect is unhelpful; a hung connect to port 5432 on a host that doesn't resolve is a diagnosis.

# Sockets, files and pipes this process holds open.
ls -l /proc/1234/fd | head -40

# Socket state -- SYN-SENT means the peer never answered the handshake.
ss -tanp | grep 1234

# Is it out of file descriptors? A process at its limit blocks in
# accept() or open() in a way that looks exactly like a hang.
cat /proc/1234/limits | grep 'open files'
ls /proc/1234/fd | wc -l

The file-descriptor exhaustion case is worth checking early because it's common and the symptom is so misleading. A server that leaks sockets doesn't crash — it stops accepting connections and looks frozen.

The D-state case

Uninterruptible sleep means the process is blocked inside the kernel and cannot be signalled. kill -9 will not work, and this surprises people. The process is not ignoring your signal; the kernel will not deliver it until the operation completes.

In practice D state means storage. A slow or stalled disk, a network filesystem whose server went away, or an overloaded host where I/O is queued behind other work. Check whether it's you or the machine:

# Everything currently in D state -- if it's several processes,
# the problem is the device, not your program.
ps -eo pid,stat,wchan:32,cmd | awk '$2 ~ /D/'

# Per-process I/O counters. If these are climbing, it's working,
# just slowly. If they're frozen, something below it is stuck.
cat /proc/1234/io

There's no clever fix from userspace. Either the I/O completes or you restart the machine. What you can do is stop it recurring: put a timeout on the operation, and if it's a network filesystem, mount it with an interruptible option so a stalled server produces an error rather than an unkillable process.

Doing this inside a sandbox

The tooling above needs two things that constrained runtimes often don't give you: a real /proc, and the ptrace capability that strace and gdb depend on. In a Firecracker microVM you get both, because it's a real kernel with a real userspace rather than a restricted process container. On PandaStack that means you can exec into a live sandbox and run the same commands you'd run on any Linux box:

# Find the stuck process from outside.
pandastack exec $SANDBOX_ID -- \
  ps -eo pid,stat,wchan:24,pcpu,etime,cmd --sort=-pcpu

# Attach strace to it. Nothing to enable, no privileged mode needed.
pandastack exec $SANDBOX_ID -- timeout 10 strace -f -p 1234

Two practical notes. Wrap strace in timeout so an investigation command doesn't itself hang your terminal. And if you need to look at a hung state repeatedly — comparing before and after a change, or handing it to someone else — snapshot the sandbox first. A snapshot captures the memory and disk of the VM as it is, so you can fork the hung state and poke at a copy while the original sits untouched.

Making it not happen again

1. Put a timeout on every external call. Not a generous one — a specific one. The default in most HTTP clients is no timeout at all, which is how a single unresponsive dependency freezes an entire service.
2. Redirect stdin from /dev/null in any automated context, so an unexpected prompt fails immediately instead of waiting forever.
3. Give long-running jobs an overall deadline, so a hang becomes a failed job with logs rather than a slot occupied indefinitely.
4. Log a heartbeat from inside long operations. 'Processing item 400 of 10,000' converts a mystery hang into a known position.
5. Alert on duration, not just on errors. A job whose p95 runtime is creeping toward its interval is the early warning that a hang is coming.

The summary

Get the process state first — it splits the problem three ways and costs nothing. Burning CPU means take a stack; py-spy, SIGQUIT and jstack are friendlier than gdb. Asleep means attach strace and read what it's blocked on, remembering that the silence after the last line is the signal. D state means storage, and no signal will help. Check file descriptors when the syscall is ambiguous. And resist the restart until you've looked, because the hung process is a free reproduction of a bug you'd otherwise have to chase.