How to run ephemeral test environments from GitLab CI

GitLab's `services:` keyword is genuinely good at what it does. You declare a Postgres image, it comes up on the job's network, your tests connect to it, it goes away. For unit tests against a fresh schema that is the right tool and you should keep using it.

It stops being enough at a specific point: when the thing you need to test is the deployed application rather than the code. A reviewer wants a URL. A Playwright suite needs a real origin with real cookies. A migration needs to run against a database that has the shape of production, not an empty one. None of that fits inside a job container on the runner's network.

I'm Ajay; I build PandaStack. We have a GitHub App that does this automatically, and we do not have a GitLab equivalent — so if you are on GitLab, you wire it up yourself with the REST API and about forty lines of YAML. Honestly, the explicit version is easier to reason about, and it is what I would show you first regardless. Everything below is a real, working pipeline.

The shape

Three jobs and one GitLab feature doing the heavy lifting.

1. `provision` creates a database from a point-in-time clone of production and deploys the merge request's branch as an app, then publishes the app URL as a dynamic environment.
2. `test` runs the end-to-end suite against that URL.
3. `teardown` deletes both, triggered by GitLab's `on_stop` when the merge request merges or closes.

The GitLab feature is `environment:` with `on_stop`. It is the part people skip, and skipping it is why teams end up with 200 orphaned environments and a bill. Get the teardown right before you get the provisioning right.

Setup

Mint an API key, put it in your project's CI/CD variables as `PANDASTACK_API_KEY`, and mark it masked and protected. Keys are scoped to the organisation that was selected when you created them, so if you keep staging and production in separate orgs, make sure you minted this one in staging.

variables:
  PANDASTACK_API: "https://api.pandastack.ai"
  SLUG: "mr-$CI_MERGE_REQUEST_IID"

.ps: &ps
  image: alpine:3.20
  before_script:
    - apk add --no-cache curl jq
    - |
      ps_api() {
        curl -sS --fail-with-body \
          -H "Authorization: Bearer $PANDASTACK_API_KEY" \
          -H "Content-Type: application/json" "$@"
      }

`--fail-with-body` is doing real work there. Plain `curl` exits zero on a 500 and your pipeline goes green while nothing was created; `--fail` alone exits non-zero but eats the response body so you never see why. You want both the failure and the message.

The provision job

Two resources: a database cloned from production, and an app deployed from the branch with that database's connection string in its environment.

provision:
  <<: *ps
  stage: deploy
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: $DYNAMIC_URL
    on_stop: teardown
  script:
    # 1. clone production as of 10 minutes ago
    - |
      DB=$(ps_api -X POST "$PANDASTACK_API/v1/databases/$PROD_DB_ID/clone" \
        -d "{\"label\":\"$SLUG\",\"size\":\"1g\"}" | jq -r '.id')
      echo "database=$DB"
    # 2. clone returns 202 — poll until it is actually up
    - |
      for i in $(seq 1 60); do
        STATUS=$(ps_api "$PANDASTACK_API/v1/databases/$DB" | jq -r '.status')
        [ "$STATUS" = "running" ] && break
        [ "$STATUS" = "error" ] && { echo "clone failed"; exit 1; }
        sleep 5
      done
      [ "$STATUS" = "running" ] || { echo "clone timed out"; exit 1; }
      DB_URL=$(ps_api "$PANDASTACK_API/v1/databases/$DB" | jq -r '.connection_url')
    # 3. deploy the branch with that DSN wired in
    - |
      APP=$(ps_api -X POST "$PANDASTACK_API/v1/apps" -d "{
        \"name\": \"$SLUG\",
        \"git_url\": \"$CI_REPOSITORY_URL\",
        \"git_branch\": \"$CI_COMMIT_REF_NAME\",
        \"env\": { \"DATABASE_URL\": \"$DB_URL\", \"NODE_ENV\": \"test\" }
      }" | jq -r '.id')
      ps_api -X POST "$PANDASTACK_API/v1/apps/$APP/deploys" -d '{}'
    # 4. hand the URL and the ids to the later jobs
    - |
      URL=$(ps_api "$PANDASTACK_API/v1/apps/$APP" | jq -r '.url')
      echo "DYNAMIC_URL=$URL"  >> deploy.env
      echo "APP_ID=$APP"       >> deploy.env
      echo "DB_ID=$DB"         >> deploy.env
  artifacts:
    reports:
      dotenv: deploy.env

The `dotenv` report is what makes the whole thing hang together. Variables written to that file become real CI variables in every downstream job in the pipeline, which is how `test` and `teardown` learn the ids they need to act on. Without it you are re-deriving resource ids from naming conventions, and naming conventions are how you eventually delete the wrong thing.

The test job

e2e:
  stage: test
  needs: [provision]
  image: mcr.microsoft.com/playwright:v1.47.0-jammy
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - npm ci
    - npx playwright test --reporter=line
  variables:
    BASE_URL: $DYNAMIC_URL
  artifacts:
    when: always
    paths: [playwright-report/]
    expire_in: 1 week

`BASE_URL` is a real HTTPS origin, not a container hostname. That difference matters more than it looks: secure cookies work, redirects resolve, third-party scripts that refuse to load on plain HTTP behave, and CORS behaves the way it will in production. A surprising share of the bugs that only show up in staging are actually origin bugs, and testing against a real one catches them a week earlier.

The teardown job — write this one first

teardown:
  <<: *ps
  stage: .post
  needs: [provision]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: manual
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop
  script:
    - ps_api -X DELETE "$PANDASTACK_API/v1/apps/$APP_ID"  || true
    - ps_api -X DELETE "$PANDASTACK_API/v1/databases/$DB_ID" || true

GitLab runs this automatically when the merge request merges or closes, because it is the `on_stop` target of the environment. The `when: manual` rule is not a contradiction — it means the job does not run as part of the normal pipeline, only when the environment is stopped, and a reviewer can also stop it by hand from the environments page.

The `|| true` is deliberate. If the app was already deleted, you still want the database delete to run. A teardown job that aborts halfway is worse than no teardown job, because it leaves exactly one resource behind, every time, and nobody notices for a quarter.

The safety net you will need anyway

Teardown will not always run. Pipelines get cancelled, runners die, someone force-deletes a branch. Two things save you.

First, put a TTL on anything you create that supports one, so an abandoned resource expires on its own. Second, run a scheduled pipeline once a day that lists apps in the org, and deletes anything named `mr-*` whose merge request is no longer open.

ps_api "$PANDASTACK_API/v1/apps" \
  | jq -r '.items[] | select(.name | startswith("mr-")) | "\(.id) \(.name)"' \
  | while read -r id name; do
      iid="${name#mr-}"
      state=$(curl -sS -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \
        "$CI_API_V4_URL/projects/$CI_PROJECT_ID/merge_requests/$iid" | jq -r '.state')
      if [ "$state" != "opened" ]; then
        echo "reaping $name (mr is $state)"
        ps_api -X DELETE "$PANDASTACK_API/v1/apps/$id"
      fi
    done

This reaper has caught more strays than I expected. It is twenty lines and it is the difference between a review-app setup that stays cheap and one that quietly grows a long tail of dead environments.

What this actually costs

The reason this is affordable is that idle review environments are not running. An app with no traffic is torn down and restored from a snapshot when a request arrives, so between the moment the reviewer closes the tab and the moment they come back, you are paying for stored bytes rather than for a running machine.

That changes the arithmetic in a way that is easy to under-appreciate. A review environment that lives for four days but is actually looked at for twenty minutes costs roughly twenty minutes of compute. On a platform where the environment runs continuously for four days, it costs four days. Same feature, and a bill that differs by two orders of magnitude — which is usually the real reason teams cap review apps to "important branches only".