Blue-Green Deploys on MicroVMs: Isolation You Can Actually Roll Back

Blue-green is the deploy pattern everyone can describe and almost nobody implements honestly. The description is simple: you have a live environment (blue), you build the next version somewhere else (green), you prove green is healthy, and then you move traffic. If green turns out bad, you move traffic back. The word doing all the load-bearing work in that sentence is "back" — and in a lot of real pipelines it's a lie, because by the time you want to go back, blue no longer exists in the state it was in. You restarted it in place, or you redeployed over it, or the migration it depended on already ran. What you're calling rollback is a second forward deploy wearing a hat, carrying all the risk of the deploy that just broke you, plus the added excitement of being performed by someone who is currently panicking.

I'm Ajay; I built PandaStack, and its git-driven app hosting does blue-green with a Firecracker microVM as the unit of color. Not a container on a shared kernel, not a process you SIGHUP — a whole machine, with its own guest kernel, its own filesystem, its own network namespace. That choice is what makes rollback structurally real instead of aspirational: the old machine is still sitting there, unmutated, exactly as it was when it was serving traffic fine. Rolling back is pointing the router at it. This post is the anatomy of that flip, why VM-level isolation is what makes it atomic, what in-place restarts actually leave behind, and the one honest constraint the snapshot model imposes on you.

What blue-green actually promises

Strip the color metaphor and blue-green makes exactly two promises. First: at no point does a user hit a half-deployed system — the switch from old to new is a single discrete event, not a rolling window where some requests get version N and some get a version N+1 that's still unpacking a tarball. Second: if the new version is bad, you can return to the previous known-good state without rebuilding it. Both promises depend on the same underlying property, which is that the old environment stays intact and untouched while the new one is being constructed.

That property is easy to say and easy to accidentally violate. The instant your deploy touches the running environment — stops a process, overwrites a directory, mutates a config file, warms a cache in place — you have converted "switch between two environments" into "transform one environment." A transformation has an inverse only if you wrote one, tested it, and it holds under the exact failure conditions you're in. Most teams have not, which is how you end up with a rollback procedure that is really just a revert commit plus twelve minutes of hoping.

Anatomy of a flip

Here's the sequence PandaStack's deploy pipeline runs, which is also just what a correct blue-green flip looks like generically. The order matters more than the tooling, and specifically: nothing touches the router until the second-to-last step, and nothing touches blue until after the router has moved.

1. Provision green as a brand-new microVM restored from a baked template snapshot. Blue is untouched and still serving every request.
2. Get the code in — a shallow clone of the exact commit, not 'whatever HEAD is now'. Pinning to a SHA is what makes a redeploy of the same input produce the same output.
3. Resolve the runtime and install dependencies inside green. Version resolution comes from the repo's own files, so the runtime is a property of the commit rather than a property of the machine.
4. Build. Stream logs to the deployment record, because a build that fails silently at 3am is functionally identical to a build that never ran.
5. Start the process detached, with stdout and stderr captured to a file inside the guest. Green is now running and receiving exactly zero traffic.
6. Health-check green's actual app port over HTTP, with a bounded budget. This is the gate. Not 'did the process start' — 'does it answer'.
7. Atomically flip the router to point at green. One write, one discrete event, no window where both colors serve.
8. Mark blue superseded and tear it down — after the flip, never before.

Step 6 is the one people skip, and it's the only step that is actually a safety property. Everything else is plumbing. If you flip before health-checking, you have not built blue-green — you've built a more elaborate way to deploy a broken build to production, with the added cost of provisioning a second machine to do it. A process can exit(0) out of its launcher, bind nothing, and sit there dead while your orchestrator marks the deploy successful. Ask the port, not the process.

Step 8 is the one people get wrong in the opposite direction: tearing down blue too eagerly. The window between "flipped to green" and "tore down blue" is your entire rollback budget. Kill blue in the same breath as the flip and you've spent the budget before you know whether you need it. Keep the old machine at least until green has survived a few minutes of real traffic — the failures a health check misses are precisely the ones that need actual users to surface.

Why the unit of color should be a whole machine

You can do blue-green at several granularities, and they are not equally honest. At the process level, blue and green share a filesystem, a kernel, a page cache, and usually a set of open file descriptors — so "green" is really "blue after some edits." At the container level you get a fresh filesystem view but still a shared kernel, shared sysctls, shared page-cache pressure, and shared exposure to whatever a neighbor does. At the microVM level, green is a machine that did not exist five seconds ago and shares nothing with blue except the physical host and a router pointing at one of them.

That last one is the version where the promises hold. When green is a separate VM, there is no code path by which building green corrupts blue, because building green happens inside a different kernel. A build script that fills the disk fills green's disk. A test that fork-bombs fork-bombs green. A dependency install that scribbles into a global cache scribbles into green's global cache. Blue isn't "probably fine" — it's provably untouched, because nothing that ran had a handle to it. And when green fails, you delete it whole. There is no partially-mutated machine to un-mutate, because you never mutated a machine; you built one and threw it away.

• Rollback mechanism — In-place restart: re-run the deploy with the old artifact and hope the old artifact still builds. MicroVM blue-green: point the router back at a machine that is still running.
• Rollback duration — In-place restart: a full build and install cycle, during which you are down. MicroVM blue-green: one router write, bounded by proxy propagation rather than by npm.
• State left behind by a failure — In-place restart: leaked FDs, half-written caches, stale env, a half-extracted node_modules. MicroVM blue-green: none — the failed green is deleted whole.
• Isolation during build — In-place restart: the build runs on the machine currently serving traffic. MicroVM blue-green: the build runs on a machine with its own guest kernel that no user can reach.
• Blast radius of a bad build — In-place restart: the live environment, because the live environment is the thing being mutated. MicroVM blue-green: one disposable VM.
• Cost of a fresh environment — In-place restart: high, which is exactly why teams reuse and mutate instead. MicroVM blue-green: a snapshot restore of roughly 49ms, so a fresh machine per deploy is the cheap option.

The economics used to be the objection here. "A whole VM per deploy" sounded expensive back when a VM meant a multi-second boot and gigabytes reserved up front. Firecracker plus snapshot-restore changes the arithmetic. PandaStack creates a sandbox by restoring a baked snapshot on demand rather than cold-booting: the restore step lands around 49ms, end-to-end create is p50 179ms and p99 ~203ms, and only the first-ever cold boot of a template is around 3 seconds. Memory restores copy-on-write and the rootfs is a reflink clone, so the Nth green machine doesn't copy gigabytes off disk. A fresh, hardware-isolated machine per deploy stops being a luxury and becomes the default.

The residue in-place restarts leave behind

"Just restart the service with the new code" works right up until it doesn't, and the way it fails is rarely dramatic. It's a slow accumulation of small differences between what the machine is and what your deploy script believes the machine is. A short, non-exhaustive list of what survives a process restart on a long-lived box:

• File descriptors and sockets a supervisor held open across the restart, so the new process inherits a connection the old version negotiated.
• Half-written caches: a build-artifact directory, a compiled-template cache, a node_modules containing a package that was mid-extraction when the install got SIGKILLed at your 30-second timeout.
• Stale environment — the shell that launched the new process inherited the env of whatever launched the old one, so your config change is applied everywhere except on the box.
• Files from a version you deployed four releases ago, still on disk, still readable, still being globbed by a startup script nobody has read since it was written.
• A schema migration that ran halfway, applied three of five statements, and failed — so the database now matches neither the old code nor the new code.
• The tuning someone did by hand at 2am during an incident, which is load-bearing, undocumented, and will vanish the first time the box is replaced.

The last two are the genuinely dangerous ones. A partially-applied migration is the case where rollback is hardest, because your data no longer matches either version of your code, and no deploy strategy fixes that for you. Blue-green with real machines doesn't magically make schema changes reversible; it makes the code side reversible, which narrows the problem down to the part you actually have to think about. That's the honest framing: expand-and-contract your schema so both colors can run against it simultaneously, and let the VM boundary handle everything else.

Building the flip yourself

If you want the pattern without a platform's opinion baked in, the SDK gives you the primitives directly. The shape is: create green, build it, start it detached, poll its port until it answers, flip your router, and only then kill blue. Note the `except` branch — a green that fails its health check must be deleted, or you accumulate a graveyard of half-built machines that cost money and confuse whoever is on call next week.

from pandastack import Sandbox

COMMIT = "a1b2c3d4"          # pin the exact SHA, never 'whatever HEAD is'
REPO   = "https://github.com/acme/web"
PORT   = 3000

def deploy(blue: Sandbox | None) -> Sandbox:
    # 1. Provision GREEN from a baked snapshot. Blue keeps serving.
    green = Sandbox.create(template="base", ttl_seconds=86400)
    try:
        # 2. Clone, then check out the pinned commit.
        green.exec(f"git clone {REPO} /app", timeout_seconds=120)
        green.exec(f"cd /app && git checkout {COMMIT}", timeout_seconds=60)

        # 3-4. Install + build. Anything this does is confined to green:
        # its own kernel, its own disk, its own network namespace.
        r = green.exec("cd /app && npm ci && npm run build",
                       timeout_seconds=900)
        if r.exit_code != 0:
            raise RuntimeError(f"build failed:\n{r.stderr[-4000:]}")

        # 5. Start detached; capture stdout/stderr inside the guest.
        green.exec(
            f"cd /app && export PORT={PORT} && "
            f"setsid sh -c 'npm start > /var/log/app.log 2>&1' &",
            timeout_seconds=10,
        )

        # 6. THE GATE. Ask the port, not the process.
        if not wait_healthy(green, PORT, budget_seconds=60):
            tail = green.exec("tail -n 80 /var/log/app.log",
                              timeout_seconds=10)
            raise RuntimeError(f"green never got healthy:\n{tail.stdout}")
    except Exception:
        green.kill()   # deleted whole — no residue anywhere to clean up
        raise

    # 7. Flip. One discrete write; blue is still alive behind us.
    router.point_at(green.id)

    # 8. Only NOW is blue expendable. Keep it around for a warm rollback.
    if blue is not None:
        blue.kill()
    return green

`exec` gives you `stdout`, `stderr`, and `exit_code`, and the exit code is what you branch on — a build tool that prints scary red text and exits 0 has succeeded, and one that prints nothing and exits 1 has not. Always pass `timeout_seconds`: a dependency install hanging on a registry that's having a bad day should fail your deploy in ten minutes, not wedge it until someone notices. And set `ttl_seconds` on create so a green you orphaned by crashing your own deploy script still reaps itself.

The health check is the whole safety property

Everything else in the pipeline is logistics; the health check is the part that decides whether you ship a working thing. Write it to be boring and strict. Poll the real port over HTTP. Give it a bounded budget, because "still starting" and "never starting" look identical from the outside and only a deadline can tell them apart. And make the endpoint mean something — a handler that returns 200 unconditionally is a health check in the same sense that a smoke detector with the battery removed is a smoke detector.

import time
from pandastack import Sandbox

# A /healthz that means something: it checks the dependencies the app
# cannot serve traffic without. Written into green before it starts.
HEALTHZ = """
export default async function healthz(_req, res) {
  try {
    await db.query('select 1');   // can we reach our own database?
    await cache.ping();           // is the cache client actually wired up?
    res.status(200).json({ ok: true });
  } catch (e) {
    res.status(503).json({ ok: false, err: String(e) });
  }
}
"""

def install_healthz(sbx: Sandbox) -> None:
    sbx.filesystem.write("/app/pages/api/healthz.js", HEALTHZ)


def wait_healthy(sbx: Sandbox, port: int, budget_seconds: int = 60) -> bool:
    """Poll green's real port from inside the VM until it answers 200."""
    deadline = time.monotonic() + budget_seconds
    while time.monotonic() < deadline:
        r = sbx.exec(
            f"curl -fsS -o /dev/null -w '%{{http_code}}' "
            f"http://127.0.0.1:{port}/healthz",
            timeout_seconds=10,
        )
        if r.exit_code == 0 and r.stdout.strip() == "200":
            return True
        time.sleep(1)
    return False   # caller deletes green and never touches the router

One subtlety worth internalizing: a health check that only proves the process is listening will happily pass a build that shipped with a broken database URL. The app boots, binds, answers 200 on `/`, and then 500s on every real request — and you flipped to it, because your gate asked the wrong question. Check the dependencies the app cannot function without, since the entire point of the gate is to catch the failures that would otherwise be caught by your users, loudly, on a Friday.

Rollback that is actually rollback

Here's where the VM-as-color choice pays. If blue is still running and the router is a pointer, rollback is a write to that pointer. No build, no install, no dependency resolution against a registry that may have yanked a version since Tuesday, no waiting on an image pull. You are not re-producing the old version; you are ceasing to point away from it. That distinction is the difference between a rollback measured in seconds and one measured in however long a deploy takes — which, notably, is the same amount of time as the thing that just failed you.

# Rollback with git-driven app hosting: flip back to the previous
# deployment. This does NOT rebuild anything — it points the app's
# stable URL at the prior deployment's machine.
curl -sS -X POST https://api.pandastack.ai/v1/apps/$APP_ID/rollback \
  -H "Authorization: Bearer $PANDASTACK_API_KEY"

# Or roll back to a specific known-good deployment by id.
curl -sS -X POST https://api.pandastack.ai/v1/apps/$APP_ID/rollback \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"deployment_id\": \"$LAST_GOOD\"}"

# Confirm what the app is pointing at now.
curl -sS https://api.pandastack.ai/v1/apps/$APP_ID \
  -H "Authorization: Bearer $PANDASTACK_API_KEY" \
  | jq '{url, status, active_deployment_id}'

# Tail the live process logs from the machine actually serving traffic.
curl -sS "https://api.pandastack.ai/v1/apps/$APP_ID/runtime-logs?follow=1" \
  -H "Authorization: Bearer $PANDASTACK_API_KEY"

There's a related trick if you don't want to pay to keep every previous color running: snapshot blue instead of killing it. A same-host fork of a snapshot is 400–750ms, cross-host 1.2–3.5s, so "restore the last known-good machine" is still fast enough to be an incident response rather than a project. It's a slightly weaker guarantee than a still-running blue — you're restoring a machine rather than pointing at one — but it's dramatically stronger than rebuilding from source, and it's the right default once you have more than a couple of apps.

A rollback plan you have never executed is a hypothesis. Practice it on a Tuesday afternoon, deliberately, while nothing is on fire — that's the only way to find out that your "previous version" needs an environment variable that was deleted three deploys ago.

The honest constraint: green must match the baked size

Every architecture has a bill, and here's this one's. A Firecracker snapshot captures the machine configuration along with the memory — vCPU count and RAM size are frozen at bake time and cannot be changed at restore. That's a property of the hypervisor, not a platform shortcut. So when green is restored from a baked template snapshot, it comes up with the template's baked vCPU and RAM, not whatever number you'd like to pass in the deploy request. On PandaStack the `base` apps template is baked at 2 vCPU and 4 GiB, and the agent overrides an incoming create request's cpu/memory to match the snapshot rather than silently producing a VM that doesn't match its own memory image.

The practical consequences are worth stating plainly, because they surprise people who are used to containers where memory limits are a flag:

• You cannot resize an app by editing a number in the deploy config. Changing green's RAM means re-baking the template snapshot at the new size.
• Blue and green are the same size by construction, which is actually what you want for a fair health comparison — a green that passed only because it got more memory is a green that will fail later, at scale, on a weekend.
• Sizing is a template-level decision made ahead of time, so treat it like a schema: version it, change it deliberately, and expect a re-bake rather than a hot edit.
• The tradeoff you get in exchange is the ~49ms restore. Boot-time-configurable memory means cold-booting a kernel; frozen memory is what makes the snapshot restorable in milliseconds.

When a microVM per color is overkill

Be honest about the threshold. If you ship a static site to a CDN, you already have atomic deploys and instant rollback — the CDN gives you an immutable build addressed by hash, and there is nothing a microVM adds to that. If your app is a single stateless process behind a load balancer you already drain properly, and your deploys have never left residue you had to clean up by hand, a rolling restart is fine and the operational cost of managing snapshots isn't worth paying.

The model earns its keep when the deploy runs real work — an install, a build, a codegen step — on the same machine that serves traffic; when your rollback story is a rebuild; when you've been burned by state a restart left behind; or when the thing being deployed can execute code you don't fully control. Other platforms offer their own flavors of atomic swap and instant rollback, and you should verify the specifics against their docs rather than my summary — but the structural question is the same regardless of vendor. Ask what the unit of color is. If it's a process or a mutated directory, your rollback is a forward deploy in a costume. If it's a whole machine that's still sitting there untouched, rollback is just a pointer, and the worst deploy of your quarter becomes a thing you fix in seconds rather than a thing you write a postmortem about.