How to debug a failed deployment

A deploy fails and the natural instinct is to redeploy. Sometimes that works, which is the worst outcome, because it teaches you that deploys are flaky when actually you have a real bug that shows up under a specific ordering.

Deployments fail in a small number of distinguishable ways, and the first job is figuring out which stage broke. The stages are: clone the repo, install runtimes, install dependencies, build, start the process, health-check the port, flip traffic. The status tells you roughly where you are; the logs tell you exactly.

First: which log are you reading?

This trips up more people than any actual bug. There are three separate log streams and they contain completely different things.

• Build logs — everything from clone through build. This is where dependency and compilation failures live.
• Runtime logs — your app's own stdout and stderr after it starts. This is where a crash on boot lives.
• Machine logs — the microVM's console output. Useful roughly never, unless the kernel or the VM itself failed to come up.

A build that fails leaves nothing in the runtime logs, because the app never started. An app that starts and then dies leaves a perfectly clean build log. Reading the wrong one produces the distinct feeling that there's no information anywhere.

# Which deployment, and how far did it get?
pandastack app deploys $APP_ID

# The build stream for that deployment (clone → install → build)
pandastack app logs $APP_ID $DEPLOY_ID

# Your app's own stdout/stderr, once it started
curl -sN "https://api.pandastack.ai/v1/apps/$APP_ID/runtime-logs?follow=1" \
  -H "Authorization: Bearer $PANDASTACK_API_KEY"

"App never responded on port 3000"

The most common failure by a wide margin, and it has three causes in descending order of frequency.

You bound to localhost. An app listening on 127.0.0.1 starts perfectly, logs that it's listening, and is unreachable from outside the process. It is the single most common cause of a deploy that looks healthy in the logs and fails its health check anyway. Bind to 0.0.0.0.

You hard-coded the port. The platform exports PORT into the launch environment and probes that port. If your start command runs a server on 8080 while the platform is probing 3000, the health check fails against a perfectly healthy app.

// Wrong: unreachable, and ignores the platform's port
app.listen(3000, "127.0.0.1");

// Right
app.listen(process.env.PORT || 3000, "0.0.0.0");

# The same rule, in the shape most Python apps hit it
# Wrong:
#   uvicorn main:app
# Right:
uvicorn main:app --host 0.0.0.0 --port $PORT

Or the app crashed on boot. It started, threw, and exited before the probe ever connected. The build log won't show this — the runtime log will, and it's usually a missing environment variable.

The build dies with no error

A build that stops mid-step with no traceback, or with a bare 'Killed', is almost always the OOM killer. JavaScript builds are the usual suspects — tsc and bundlers on a large project can exceed 2 GB comfortably, and the failure looks like a hang followed by nothing.

# The tell: no stack trace, just a dead process
==> npm run build
> tsc -p tsconfig.json && vite build
Killed

# Confirm and work around it
NODE_OPTIONS=--max-old-space-size=3072 npm run build

Note that raising Node's heap limit only helps if there's actually more memory available — it's a ceiling, not an allocation. If the build VM has 4 GB, setting the limit to 8 GB just moves where it dies. The real fixes are building fewer things at once, splitting the type-check from the bundle, or building on a larger machine.

"It builds locally"

It does, and the reasons it doesn't remotely are boring and finite.

• A different runtime version. Your machine has Node 22 from a version manager; the build image defaults to something else. Pin it in the repo — .nvmrc, .python-version, .tool-versions — so both environments agree.
• A file that isn't committed. A .env, a generated type file, a local config. The build machine clones the repo and gets only what's in it.
• Case sensitivity. macOS filesystems are case-insensitive by default; Linux isn't. An import of ./Components/Button that's actually components/Button works on your laptop and fails in CI, and the error names a file you can plainly see exists.
• Dev dependencies pruned. A production install skips devDependencies, and then the build fails because your bundler lives there. Either install everything for the build, or move the build-time packages into dependencies.

Missing environment variables

There's a distinction worth internalising here: build-time and runtime environment variables are not the same thing, and the failure mode differs.

Frameworks that inline variables at build time — anything with a NEXT_PUBLIC_ or VITE_ prefix — need them present during the build. Set them afterwards and the built bundle contains undefined, forever, until you rebuild. The app starts fine and misbehaves in the browser, which sends you looking in entirely the wrong place.

Server-side variables are needed at start. A missing DATABASE_URL usually produces a crash in the runtime log within a second of boot.

# What the app has now
pandastack app get $APP_ID

# Add one, then redeploy — build-time vars only take effect on the next build
pandastack app create --name web --git-url https://github.com/acme/web \
  --env NEXT_PUBLIC_API_URL=https://api.acme.com,DATABASE_URL="$DATABASE_URL"

The wrong build commands were guessed

Platforms detect your framework from the repository, and detection is a heuristic. A monorepo, an unusual project layout, or a stack outside the common set produces plausible-but-wrong commands, and the failure reads as a strange error deep inside a tool you didn't expect to be running.

The build log says which framework was detected in its first few lines. If that line is wrong, stop debugging the error and set the commands explicitly — you'll never fix a detection problem by fixing the symptom.

pandastack app create --name api \
  --git-url https://github.com/acme/monorepo \
  --root-directory services/api \
  --install-command 'npm ci' \
  --build-command 'npm run build' \
  --start-command 'node dist/main.js' \
  --port 3000

It deployed, and then broke

Different problem, different tools. The deploy succeeded, so the app passed its health check with the new code — meaning what broke is either something that only happens under real traffic, or something that happens later.

Roll back first, investigate second. A rollback returns you to the previous deployment in seconds; debugging with production down is a choice, not a requirement.

pandastack app rollback $APP_ID       # back to the previous deployment
pandastack app deploys $APP_ID        # then read what the bad one did

Then look for the three usual causes: a migration that ran and changed behaviour the old code didn't expect, a dependency that resolved to a new version because the lockfile wasn't committed, or an environment variable that exists in staging and not in production.

The order to work through

1. Check the deployment status — it tells you which stage failed, and therefore which log to open.
2. Failed during build: read the build log from the top, not the bottom. The first error causes the rest.
3. Failed at health check: check the bind address and the port before anything else, then read the runtime log for a crash on boot.
4. No error at all, just a dead build: assume OOM and check memory.
5. Deployed but broken: roll back, then compare the two deployments' commits and environments.
6. Reproduce in a sandbox on the same template before changing code — a five-minute exec session beats four speculative redeploys.

That last point is the one worth building a habit around. Being able to open a shell in an environment identical to the build environment turns most of this from guesswork into a two-minute check.