Why your visual regression tests are flaky

Visual regression testing has an obvious premise: screenshot the UI, compare against a baseline, fail on differences. It catches the class of bug no assertion catches — the CSS change that shifts a button eight pixels, the font fallback that breaks a heading on one page, the dark-mode variant nobody remembered to check.

It also has a well-earned reputation for flakiness. A suite that fails on four unrelated pages because a font rendered a quarter-pixel differently gets muted within a month, and then it's just a slow job producing images nobody looks at. The good news is that almost every false positive comes from a small, enumerable set of environmental causes.

Where the pixel differences come from

Font rendering — the biggest one

Text rendering depends on the font file, the rasterizer, hinting settings, subpixel antialiasing, and the operating system's font configuration. Change any and every glyph shifts subtly. Between a developer's Mac and a Linux CI runner, essentially all text differs — which is why baselines generated locally never match CI.

Worse, it drifts silently. A base image update that bumps fontconfig or adds a font package changes rendering everywhere, and your entire baseline set fails at once with no code change. Everyone accepts the mass re-baseline, and the one real regression hiding in those 300 diffs ships to production.

GPU and compositing

Hardware acceleration changes how gradients, shadows, transforms, and filters rasterise. Different GPUs, different drivers, or a fallback to software rendering under load all produce visibly different output for the same CSS. Headless Chrome's behaviour also differs depending on the flags and the runner's capabilities.

Timing and animation

This is the one that produces genuinely random failures. Screenshot a page mid-animation and you capture a frame that depends on when the machine got scheduled. Under CPU contention — a busy shared runner — a transition that normally finishes in 200ms takes 400ms, and your screenshot lands mid-fade. The test failed because another job was busy, which is not a fact about your code.

Dynamic content

Timestamps, relative dates, avatars from a randomised set, ad slots, A/B assignments, and any 'recently viewed' widget. Each is a guaranteed diff on every run, and each has to be handled deliberately.

Fix one: make the environment identical, not similar

The root cause of most of the above is that the rendering environment varies between runs. So pin it completely and version it like code.

• Pin the browser version exactly. Not 'latest Chrome' — a specific build. Browser updates change rendering, and you want that to be a deliberate change with a re-baseline, not a Tuesday surprise.
• Bake the fonts into the image and pin the font configuration. Ship the exact font files, and disable or explicitly fix hinting and subpixel antialiasing so the rasterizer behaves identically everywhere.
• Force software rendering. Slower, and completely deterministic. For visual regression the trade is obviously worth it — you are testing layout and style, not GPU performance.
• Fix the viewport and device pixel ratio explicitly, in the test rather than relying on a default.
• Version the whole environment. When the image changes, the baselines are expected to change, and that becomes a reviewed pull request rather than an anomaly.

This is where a microVM helps more than a container. The environment includes the kernel, and a snapshot captures the entire machine state — browser build, font files, font cache, libraries, configuration. Restoring that snapshot gives byte-identical starting conditions rather than a similar-enough userspace on whatever kernel the runner happens to have. And because every restore starts from the same snapshot, the font cache is warm and identical rather than being rebuilt slightly differently each time.

Fix two: stop sharing CPU with other jobs

Timing flakiness is a resource contention problem. A shared runner executing four jobs gives your browser unpredictable CPU, so animations and network-dependent renders complete at unpredictable times, so screenshots land at unpredictable moments.

One microVM per browser, with its own vCPUs, removes the variable. Every run gets the same compute, so the same page takes the same time to settle. And a fleet is practical because creating a machine through snapshot-restore is about 179ms at p50 — twenty parallel browsers are live in well under a second, so isolation costs you almost no wall clock.

from concurrent.futures import ThreadPoolExecutor
from pandastack import Sandbox

PAGES = ["/", "/pricing", "/docs", "/blog", "/login"]

def shoot(path: str) -> tuple[str, bytes]:
    # One microVM per page: own vCPUs, so animation timing does not depend
    # on what another job is doing. The snapshot has the pinned browser
    # build, the exact font files, and a warm font cache already baked in.
    with Sandbox.create(template="snap-visual-chrome-131", ttl_seconds=600) as vm:
        vm.filesystem.write("/work/shot.js", SCRIPT.encode())
        res = vm.exec(f"cd /work && node shot.js '{path}'", timeout_seconds=180)
        if res.exit_code != 0:
            raise RuntimeError(f"{path}: {res.stderr[-500:]}")
        return path, vm.filesystem.read("/work/out.png")

# ~179ms p50 per create, concurrent -- isolation costs almost no wall clock
with ThreadPoolExecutor(max_workers=len(PAGES)) as pool:
    shots = dict(pool.map(shoot, PAGES))

Fix three: make the page itself deterministic

Environment pinning handles the platform. The page still has to cooperate.

// shot.js -- the page-side determinism work. Most 'flaky' visual tests
// are missing three or four of these.
const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({
    args: ['--disable-gpu', '--force-color-profile=srgb',
           '--font-render-hinting=none', '--disable-lcd-text'],
  });

  const page = await browser.newContext({
    viewport: { width: 1280, height: 800 },
    deviceScaleFactor: 1,
    // Pin these: a different locale or timezone re-renders every date.
    locale: 'en-GB',
    timezoneId: 'UTC',
    colorScheme: 'light',
  }).then(c => c.newPage());

  // Freeze the clock BEFORE any script runs, so relative timestamps
  // ('3 minutes ago') render identically on every run.
  await page.addInitScript(() => {
    const FIXED = new Date('2026-01-01T12:00:00Z').getTime();
    Date.now = () => FIXED;
    Math.random = () => 0.42;          // kill randomised avatars/ordering
  });

  await page.goto(process.env.BASE_URL + process.argv[2],
                  { waitUntil: 'networkidle' });

  // Stop animation rather than racing it. Disabling transitions is far
  // more reliable than sleeping and hoping they finished.
  await page.addStyleTag({ content: `
    *, *::before, *::after {
      animation-duration: 0s !important;
      animation-delay: 0s !important;
      transition-duration: 0s !important;
      transition-delay: 0s !important;
      caret-color: transparent !important;
    }
    html { scroll-behavior: auto !important; }
  ` });

  // Wait for fonts specifically -- a screenshot taken during the fallback
  // font's brief appearance is the classic one-in-twenty failure.
  await page.evaluate(() => document.fonts.ready);

  await page.screenshot({ path: '/work/out.png', fullPage: true, animations: 'disabled' });
  await browser.close();
})();

The font-ready wait deserves special mention. It's the cause of the intermittent failure that appears roughly once in twenty runs and cannot be reproduced locally: the screenshot caught the page during the flash of fallback text. One line fixes it permanently.

On diff thresholds

A pixel tolerance is the usual response to flakiness, and it's a trade rather than a fix: set it high enough to absorb your noise and it will also absorb a one-pixel border change or a subtle colour shift — often exactly the regressions you built this to catch.

Better to reduce the noise until you can run near zero tolerance. If you must use a threshold, prefer tooling that reasons about perceptual difference and clustered changed pixels rather than a raw count: 500 pixels changed in one small region is a real layout shift, and 500 pixels scattered evenly across the image is antialiasing noise. Those should not be treated identically.

Is it worth it?

For a design system or component library, unambiguously — the components are the product and unintended visual change is the bug you most need to catch. For a marketing site with frequent design changes, probably not: you'll re-baseline constantly and learn nothing.

For a typical application, target the parts where visual regressions are expensive and rare: checkout, forms, dashboards, and anything with a complex responsive layout. A focused suite of thirty stable screenshots that people trust is worth vastly more than four hundred flaky ones that get bulk-approved on Friday afternoon. The environment work above is what makes the difference between those two outcomes — and almost none of it is about the testing tool you picked.