GitOps for Ephemeral Compute: Where Reconciliation Stops

The reconcile loop is one of the few ideas the Kubernetes ecosystem exported that deserved to travel. Git holds the desired state, a controller observes the actual state, it diffs the two, it acts, it repeats. The loop is level-triggered rather than edge-triggered, which is the whole trick: it does not matter that the controller missed an event, crashed halfway through, or was restarted during the incident, because every pass re-derives the world from the declaration rather than from a history of things that happened. That property is why GitOps survived contact with production when "run the deploy script from someone's laptop" did not.

Underneath it sits an assumption that nobody states because it is nearly always true: every object under management is supposed to keep existing. A Deployment, a Service, an IAM policy, a DNS record, a bucket — these are all things whose absence is a defect. Any difference between declared and observed is therefore drift, and drift is something to correct. Change the workload to compute that is deliberately temporary and that assumption inverts, and the loop starts doing confidently wrong things at high frequency.

The forty-second object

Take a sandbox created to run one agent's tool call. It is created, it executes something, it is killed. On PandaStack a create is a snapshot restore with a p50 around 179ms, and the whole lifespan might be under a minute. Now express it as desired state. You write a manifest that says this sandbox exists. The controller creates it. Your code kills it. On the next pass the controller observes that a declared object is missing, classifies the absence as drift, and creates it again. You have built a machine that fights your own application, and it is not confused — it is doing precisely what you told it to.

This is not a hypothetical failure mode dreamed up to make a point. It is the single most common GitOps support question there is, phrased as "I deleted the pod and it came back". The usual answer is that the reconciler is working correctly and you should have changed the declaration instead. That answer is right when the object is a long-lived service. It is useless when deletion is the correct end state of a normal, successful operation, because there is no declaration to change — the desired state of that sandbox was "exist, briefly, some time ago".

The deeper problem is that a level-triggered reconciler cannot distinguish deliberate deletion from failure. Both present identically: a declared object that is not there. Kubernetes' own answer to this for Jobs is instructive — a Job is not merely a Pod spec, it carries completion semantics, a backoff limit, and a TTL for cleaning up after itself, precisely because "this should have run once" is not expressible as "this should exist". Everything downstream inherits the awkwardness. Argo CD grew hooks, sync waves and a Replace strategy largely to cope with resources whose spec is immutable once created and whose lifecycle is one-shot.

Desired state has no tense

The clean way to say it: a declaration is a statement about the present, indefinitely. It has no past tense and no perfective aspect. "Run this migration once" is a fact about a moment, not a property of the world, and every attempt to encode it as desired state ends up smuggling the moment back in through a side channel — a hash in the resource name, an annotation with a timestamp, a hook that fires on sync rather than on state, a completed marker that the reconciler is told to ignore.

Those workarounds are fine. They are also an admission. If your resource name has to contain a content hash so that a new object appears whenever the input changes, you have implemented an event log in the naming scheme of a state store. It works, and it is worth knowing that it works, but it is not a reason to push the next imperative thing into the same shape.

The half that is legitimately declarative

None of this makes declarative configuration a bad fit for ephemeral platforms. It makes it a bad fit for ephemeral instances. The fleet around them is still overwhelmingly declarative, and it is worth being precise about which parts.

PandaStack's app-hosting surface is the clearest example, because it is already GitOps-shaped without ever having been marketed that way. An app row is a specification: git URL, git branch, framework, runtime and runtime version, install command, build command, start command, root directory, port, an env map, the microVM template, CPU and memory, an auto-deploy flag, an auto-hibernate flag with an idle timeout, and a maximum instance count with a concurrency target. Every one of those fields is a statement about how the world should be, indefinitely, and every one is safe to reconcile. Nothing bad happens if a controller re-asserts them a thousand times.

The repo can carry part of that spec itself. A `pandastack.json` at the root of the deployed repository declares the framework type, the static output directory, and the install, build and start commands, and the deploy pipeline reads it with a precedence chain of explicit pin, then manifest, then auto-detection. That is configuration living beside the code it configures, which is most of what people actually want from GitOps in the first place.

{
  "type": "static",
  "outputDir": "out",
  "installCommand": "npm ci",
  "buildCommand": "npm run build"
}

The same holds for schedules — a cron expression, a target function, and a paused flag are pure desired state. So are quotas and tier limits: concurrent sandbox ceilings, total CPU and memory, a maximum TTL, an hourly create limit. So are templates, which are Dockerfiles built into microVM root filesystems and are about as declarative as an artefact gets. Templates, app definitions, schedules and limits are the fleet. Sandboxes, deployments, forks and function runs are the instances. The line falls exactly there.

Push-to-deploy is a reconcile trigger, and only a trigger

The auto-deploy path is worth reading as a reconciliation primitive because that is what it is. An HMAC-verified push webhook arrives, the handler strips `refs/heads/` off the ref, queries for every app with auto-deploy enabled whose tracked branch matches the pushed branch, and enqueues a deploy pinned to the exact pushed commit rather than to the branch name. That last detail matters more than it looks: pinning to the SHA means two rapid pushes produce two deterministic deploys instead of two races to resolve the same moving ref.

Then the deploy converges. It provisions a fresh persistent sandbox, clones, detects the framework, installs and builds, starts the process, health-checks the port, and only then flips the app's pointer to the new sandbox and tears down the old one. Blue-green, atomically, with the previous deployment marked superseded rather than deleted so a rollback has something to roll back to. That is a reconciliation of observed state toward a declared commit, and it is genuinely the good version of the pattern.

The honest limitation is that it is edge-triggered. If GitHub fails to deliver the webhook, or the API is down during the delivery window, nothing re-converges on its own. A pull-based GitOps agent that polls the repository every few minutes would catch that; a push webhook does not. This is a real trade, and it was made in the direction of latency: you find out about a missed delivery from GitHub's own delivery log, and the remedy is a manual deploy that pins the same ref. If your organisation cares more about eventual convergence than about deploy latency, a small poller comparing each app's active deployment commit against the branch head is about twenty lines and closes the gap.

The controller that does converge, and its restart budget

There is a genuine reconcile loop in the control plane, and its shape is instructive because of what it refuses to do. A single background goroutine ticks every 30 seconds. On each tick it sweeps idle apps for hibernation, then reconciles running apps, hibernated apps, waking apps, apps parked waiting for capacity, PR previews, and scale-out. Textbook controller: a periodic pass over a set of resources, driving each toward its declared state.

The interesting part is the guard rails on the running-app pass. It confirms the runtime sandbox still exists; a persistent sandbox that has vanished because the host was lost flips the app to an error state rather than being silently recreated. It health-checks the app's port from inside the sandbox and, after two consecutive failures, re-launches the stored start command in place — the sandbox is persistent, so the build artefacts are still there and a restart is cheap. And then it counts. Restarts are capped at five per app for the process lifetime, and exceeding that budget parks the app in an error state until a human or a new deploy intervenes.

Note what that design concedes. The loop is willing to converge on "running", and it is willing to give up. Giving up is not a bug in the reconciler — it is the acknowledgement that some divergence is a fact about the workload rather than drift to be corrected, and that the correct response is to stop and tell someone.

Drift detection when half the inventory is meant to vanish

Conventional drift detection compares a declared inventory against a live one and treats every difference as a finding. Run that against a sandbox platform and it produces noise proportional to your throughput, because the live set is churning by design and none of it was ever declared.

The platform's own loop for this runs in the opposite direction. A reaper ticks on an interval, walks the lifecycle table, skips anything marked persistent, computes idle time since last activity for the rest, and deletes whatever has exceeded its TTL. It converges toward absence. There is a nice detail in it: an idle reap deliberately does not cascade-delete the sandbox's snapshots, because snapshots are durable artefacts that are meant to outlive the compute that produced them. Even inside a single object there are two lifetimes with two different reconciliation targets.

So the practical rule for inventory work is to partition by declared intent before you diff anything. Three buckets:

• Declared and durable — apps, schedules, templates, domains, quotas. A missing one is a real finding. Diff these against Git and alert on the difference.
• Undeclared and transient — sandboxes with a TTL, forks, function runs, deployment build sandboxes. These should never appear in a drift report. What you monitor here is age and count, not identity: a sandbox older than its tier's maximum TTL, or a count that is climbing when it should be flat.
• Undeclared and durable — the dangerous middle. Sandboxes created with `persistent: true`, volumes, snapshots, managed databases. Nothing reaps these, and nothing declares them either, so they accumulate silently until they show up as a bill. Give them an owner tag at create time and reconcile the tag, not the object.

That third bucket is where the real money leaks on every ephemeral platform I have looked at, including this one. The reaper's `persistent` exemption is the correct behaviour and it is also a permanent invitation to forget something exists.

Doing it without a Terraform provider

There is no official PandaStack Terraform provider, and pretending otherwise would waste an afternoon of yours. What there is: a REST API, SDKs for Python and TypeScript that wrap it, and a CLI. That is enough for the pattern that actually gets used in practice, which is a YAML file in the repo and a small sync script in CI. The script is not a controller — it runs on push, it is idempotent, and it does not run in a loop.

# apps.yaml — the declarative half, checked into the repo
apps:
  - name: marketing
    git_url: https://github.com/acme/marketing
    git_branch: main
    framework: nextjs
    port: 3000
    env:
      NEXT_PUBLIC_API: https://api.acme.dev

  - name: docs
    git_url: https://github.com/acme/docs
    git_branch: main
    framework: static
    build_command: npm run build

#!/usr/bin/env python3
"""sync.py - apply apps.yaml. Creates and updates. Never deletes."""
import sys
import yaml
import pandastack

client = pandastack.Client()  # reads PANDASTACK_API_KEY
spec = yaml.safe_load(open("apps.yaml"))
live = {a["name"]: a for a in client.apps.list()}

for want in spec["apps"]:
    name = want["name"]
    fields = {k: v for k, v in want.items() if k != "name"}
    have = live.get(name)

    if have is None:
        app = client.apps.create(name=name, **fields)
        client.apps.deploy(app["id"])
        print("created", name)
        continue

    drift = {k: v for k, v in fields.items() if have.get(k) != v}
    if not drift:
        print("ok     ", name)
        continue

    client.apps.update(have["id"], **drift)
    client.apps.deploy(have["id"])  # config changes need a deploy to take effect
    print("updated", name, "->", ", ".join(sorted(drift)))

orphans = sorted(set(live) - {a["name"] for a in spec["apps"]})
if orphans:
    print("not in apps.yaml, NOT deleting:", ", ".join(orphans), file=sys.stderr)

Two deliberate choices in there. The first is that an update is followed by a deploy, because changing an app's build configuration does not retroactively rebuild it — the running sandbox was built from the old spec and stays that way until something rebuilds it. The declaration and the reality reconverge at deploy time, not at PATCH time, and pretending otherwise is how you end up debugging a build command that the logs say is in effect and the process says is not.

The second is the refusal to prune. Flux and Argo both offer automatic pruning of resources that have left the declaration, and on Kubernetes that is usually safe because the objects are cattle. Here an app has a persistent sandbox behind it, possibly an attached managed database, and a URL that something external depends on. A create-and-update-only reconciler is boring and survives a bad merge; a delete-happy one turns a reverted commit into an outage. Print the orphans, let a human decide.

The split that works

Put in Git, and reconcile freely:

• Template definitions — the Dockerfiles that become microVM root filesystems, and which template each workload uses.
• App specifications — repository, branch, framework, install/build/start commands, port, environment variable names, template, auto-deploy and hibernation policy.
• Schedules — cron expression, target function, paused flag.
• Quotas and limits — concurrency ceilings, per-workspace CPU and memory totals, maximum TTL, hourly create limits.
• Agent and host configuration — the fleet's own settings, which are long-lived by definition.

Leave to the API, and never declare:

• Sandbox instances. Create them from code, give every one a TTL, and let the reaper be the only thing with an opinion about when they end.
• Forks and fork trees. A branch-and-explore fan-out is a computation, not an inventory.
• Deployments. The app is declared; each deployment is an event with a commit attached to it.
• Function and schedule runs. The schedule is state; the run is history.
• PR previews. Their lifecycle is driven by pull-request state that lives in GitHub, not in your default branch — a preview environment declared in `main` is a preview environment that outlives the pull request.
• Secret values. Names in Git, values in a secret store. This is not novel, but it is the failure that costs the most when it happens.

The thing worth taking away is that this asymmetry is not a gap waiting to be filled by a better tool. Apps are declarative because an app is supposed to keep existing; sandboxes are imperative because a sandbox is supposed to stop. Building a controller that made sandboxes look declarative would mean building something whose primary behaviour is resurrecting compute that your application deliberately destroyed, and then building a second mechanism to suppress the first. GitOps is very good at the half of your infrastructure that is standing still. Let it have that half, and give the moving half an API and a TTL.