Debugging a browser failure you can't reproduce

A browser test fails in CI. You run it locally and it passes, forty times in a row. You add a retry. It goes green. Three weeks later the same test fails and now nobody trusts the suite.

The gap is not mysterious. The CI environment differs from your laptop in a specific and enumerable set of ways, and each difference produces a characteristic failure. Knowing the list turns this from a guessing game into a lookup.

What's actually different

• Speed. CI machines are usually slower and always more contended. Anything with an implicit timing assumption fails there first — which is a feature, since the assumption was already a bug.
• Viewport and device pixel ratio. Headless defaults to 1280×720 at 1× while your laptop is larger and probably 2×. Elements below the fold aren't clickable, and responsive breakpoints land differently.
• Fonts. CI containers ship with a minimal font set. Text renders in a fallback, wraps differently, changes element heights, and moves everything below it. This is the single largest cause of visual test failures.
• Timezone and locale. CI runs UTC with the C locale; you're in a timezone with a comma or a dot in your decimals. Any date or currency assertion is exposed.
• Animations. A slower machine renders animations at different phases, so a screenshot catches a transition mid-flight.
• Network. Different latency to your app, different DNS, occasionally no outbound internet at all — which surfaces as a mysterious hang on a third-party script tag.
• Memory. Several workers plus browsers on a constrained runner produces OOM kills that look like flakiness.
• Parallelism. Locally you run one test; in CI you run twelve that may be sharing a backend.

Reproduce the environment, not the test

Before adding instrumentation, remove the differences you can. Most of them are configuration.

// playwright.config.ts — pin everything that CI would otherwise decide
export default defineConfig({
  use: {
    viewport: { width: 1280, height: 720 },
    deviceScaleFactor: 1,
    timezoneId: "UTC",
    locale: "en-US",
    // Kill animations: the largest single source of screenshot flake
    launchOptions: { args: ["--force-prefers-reduced-motion"] },
  },
  expect: {
    toHaveScreenshot: { animations: "disabled", maxDiffPixels: 100 },
  },
});

Then run the suite locally with those settings, headless, at CI's worker count. A meaningful share of 'unreproducible' failures reproduce immediately once your local run stops being a privileged environment.

# Slow the machine down deliberately — timing bugs surface under CPU throttling
npx playwright test --workers=4
# Chrome DevTools Protocol can throttle CPU 4x; a test that only fails
# under throttling has a race, not a flake.

Capture enough on the first failure

The reason these take weeks is that each failure yields one line of stack trace, so you learn one thing per occurrence. Configure the run to capture everything on failure and you diagnose on the first one.

export default defineConfig({
  use: {
    trace: "retain-on-failure",   // the single most valuable setting here
    screenshot: "only-on-failure",
    video: "retain-on-failure",
  },
});

The trace is the one that matters. It contains a DOM snapshot at every step, all network requests and responses, console output, and a timeline you can scrub through. Opening a trace from a failed CI run puts you in roughly the position of having watched it happen.

- uses: actions/upload-artifact@v4
  if: failure()
  with:
    name: playwright-trace-${{ github.run_attempt }}
    path: test-results/
    retention-days: 7

# Then, locally
npx playwright show-trace trace.zip

Failure signatures

Once you have a trace, the error text usually maps to a cause.

'Target closed' or 'browser has disconnected'

The browser died rather than the test failing. Nearly always memory — the kernel's OOM killer took the browser process. Check the runner's memory against your worker count; the arithmetic is roughly 300 to 500 MB per Chromium plus your application. Reducing workers frequently fixes what looks like a code problem.

Timeout waiting for an element that the screenshot shows

It's there but not actionable — covered by an invisible overlay, still animating, zero-size, or inside a container with `pointer-events: none`. Playwright waits for actionability rather than mere presence, which is correct and confusing. The trace's DOM snapshot shows the overlay the screenshot doesn't.

Screenshot differs by a small amount everywhere

Font rendering. Either the font is missing in CI and everything reflowed, or antialiasing differs between platforms. Install the fonts in the CI image, and generate baselines in the same environment that compares them — a baseline captured on macOS will never match one rendered on Linux.

Fails on the first run of a session, passes on retry

Something wasn't ready. A cold cache, a service that hadn't finished starting, a database connection pool still filling. Add an explicit readiness check in global setup rather than letting the first test absorb the warm-up.

Only fails at a particular parallelism

Shared state. Two tests are using the same user, the same record, or the same global setting. This one won't reproduce with one worker no matter how many times you run it, which is why it survives so long.

Removing the parallel-interference class entirely

The shared-state failures are the worst of these because they're intermittent by construction and depend on timing between tests. You can chase them individually forever, or you can make them impossible.

// Each worker gets a private backend, restored from a prepared snapshot
import { Sandbox } from "@pandastack/sdk";

export default async function globalSetup({ workerIndex }) {
  const env = await Sandbox.fork(process.env.STACK_SNAPSHOT);
  process.env.BASE_URL = env.url;
}

If no two workers share a database, one test cannot affect another's data, and any failure that survives is genuinely in the code under test. That's a much better place to be than a suite where every failure requires deciding whether it's real.

On retries

Retries have a legitimate use: they keep an unrelated infrastructure hiccup from blocking a merge. They also hide real bugs, and the hidden ones are disproportionately race conditions — which is to say, the bugs your users will hit.

• Set retries to 1 or 2 in CI, never locally. A test that flakes on your machine should stop you.
• Track which tests actually use their retry. A test that retries frequently is a bug report, not a fact of life.
• Fail the build if the flake rate crosses a threshold. Otherwise 'the suite is a bit flaky' ratchets in one direction forever.
• Never retry the whole suite. Retrying one test costs seconds; retrying everything hides which test was unstable and burns a full run.

The suite you want is one where a red result means something. Getting there is mostly capturing enough evidence to fix failures rather than paper over them — and removing the environmental differences that manufacture failures with no bug behind them at all.