How to Roll Back a Bad Deployment

You shipped, error rates went up, and someone is asking whether to roll back. The reason this question causes arguments at 11pm is that everyone is answering a different one. 'Can we revert the code' is nearly always yes. 'Can we revert the system to the state it was in' is a much harder question, and the difference is usually a database migration.

I'm Ajay, I build PandaStack. Here's the decision, in the order you should make it, and the preparation that turns rollback from a judgement call into a button.

Step 1: decide, in under two minutes

Roll back when the fix isn't obvious, when user-facing impact is ongoing, or when you're not certain what broke. Rolling back is a way of buying time to think, and time to think is the scarcest thing in an incident.

Fix forward when the cause is unambiguous and one-line, when the deploy included a migration that can't be undone, or when rolling back would itself be risky — a rollback that hasn't been rehearsed is a change too, and changes during incidents are how a bad hour becomes a bad day.

Step 2: the mechanics

On a platform with immutable deployments, rollback should be a single command that promotes a previous build. The important property is that it redeploys the exact artifact — the same commit, the same dependency versions — rather than rebuilding from a git revert, because a rebuild resolves dependencies afresh and may not produce what ran before.

# List deployments and find the last known-good one
pandastack app deploys <app-id>

# Roll back to the previous live deployment
pandastack app rollback <app-id>

# Or target a specific deployment via the API
curl -X POST https://api.pandastack.ai/v1/apps/<app-id>/rollback \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"deployment_id": "<known-good-deployment-id>"}'

Under a blue-green model, this rebuilds the old version alongside the broken one, health-checks it, and only then moves traffic — so a failed rollback leaves you where you already were rather than somewhere new. That property matters more than rollback speed. A rollback that can itself fail open is worse than no rollback.

If you're on plain containers or VMs, the equivalent is redeploying the previous image tag or the previous release directory. The requirement is the same either way: previous versions must still exist and be startable. A platform that keeps only the current build has no rollback, whatever the dashboard says.

Step 3: the database, which is the actual problem

Application code is stateless and reverts cleanly. Your database does not. If the deploy ran a migration, rolling code back leaves new code's schema under old code, and old code will hit columns it doesn't know about, missing columns it expects, or constraints it can't satisfy.

The two safe positions are: the migration is backwards-compatible, so old code runs fine against the new schema; or you have a down migration you've actually tested against production-shaped data. 'The ORM can generate a down migration' is not the second thing.

• Additive changes — a new nullable column, a new table, a new index — are safe to roll back over. Old code ignores what it doesn't know about.
• Destructive changes — dropped or renamed columns, tightened constraints, changed types — are not. Rolling back over these needs a restore, not a revert.
• Data migrations that transform rows in place are the worst case: even a perfect down migration can't recover information the up migration discarded.

This is why the expand-and-contract pattern is worth the discipline. Deploy the schema change first and separately, in a form both old and new code tolerate. Deploy the code. Remove the old column in a later, separate deploy, once you're confident. Each step is independently reversible, which is the whole point — rollback stops being a special operation and becomes a normal one.

Step 4: verify, then write it down

1. Confirm the rollback actually took effect. Check the deployed commit, not the dashboard's status badge. A rollback that silently no-oped is a real failure mode.
2. Watch the metric that told you something was wrong. If error rate was the signal, error rate is the confirmation — not 'the page loads for me'.
3. Check for partial state. Jobs that started under the new version, cache entries written in a new format, webhook deliveries already sent. These outlive the rollback.
4. Tell people it's done, in the same channel where you said it was broken.
5. Only then look at what happened. The temptation to debug while the site is down is strong and wrong.

Making rollback boring

Everything above is easier if you did these things in advance. None of them are exotic.

1. Keep previous deployments startable, not just recorded. Check how many your platform retains and whether promoting one is a single action.
2. Separate schema changes from code changes into different deploys. This is the single highest-leverage habit in this whole list.
3. Make migrations backwards-compatible by default, and treat a destructive migration as a deliberate, reviewed exception.
4. Rehearse a rollback on staging quarterly. An untested rollback path is a plan, not a capability.
5. Have a database restore point you trust, and know its age. For per-environment databases, a clone from a point in time before the deploy is often faster and safer than a down migration.
6. Write the health check so it actually checks something. An endpoint that returns 200 because the process is alive will happily certify a broken deployment.

# When a migration can't be reversed, restore is the honest path.
# Clone to a point in time before the deploy, verify against the clone,
# and only then decide what to do with production.
pandastack db clone prod-db \
  --label pre-deploy-check \
  --target-time 2026-08-19T14:05:00Z

# The source database is untouched; you get an independent copy to inspect.

Short version

Roll back when you're unsure — it buys thinking time. Fix forward when the cause is obvious and the migration isn't reversible. Promote a previous artifact rather than rebuilding a revert. And the real work isn't the rollback command, it's the habit of shipping schema changes separately from code, so that reverting one never requires reverting the other.