A browser test that fails only after a deployment, and only on some runs, is often not testing the feature you think it is. If the page loads old JavaScript from a CDN edge, a browser cache, or a service worker, the test can fail for reasons that have nothing to do with the app logic under test.

The first mistake is to treat every one of these failures as a flaky test. Sometimes the test is wrong. Sometimes the deployment pipeline is wrong. Sometimes both are wrong. If you want to debug browser tests with stale cached javascript, you need to separate three failure classes quickly:

  1. The app shipped a real regression.
  2. The browser got stale assets.
  3. The deployment exposed a mismatch between HTML, JS bundle hashes, and cache propagation timing.

If HTML and JavaScript are not versioned and invalidated together, your test may be the first system that notices.

The short version

My default rule is simple:

  • If the failure disappears after a hard refresh or a fresh browser context, suspect cache or service worker overlap.
  • If the failure disappears only after CDN propagation finishes, suspect partial rollout or asset hash drift.
  • If the failure persists across fresh contexts, no cache, and a direct origin hit, treat it as a product regression until proven otherwise.

The point is not to ignore failures. The point is to avoid wasting debugging time in the wrong layer.

Why this failure mode is so confusing

Modern web deployments often combine several caching layers:

  • Browser disk cache and memory cache
  • Service workers
  • CDN edge caches
  • Reverse proxies or shared application caches
  • Build artifacts with content-hashed filenames

Each layer can be correct by itself and still create a broken end-to-end page. A typical failure looks like this:

  1. The HTML shell updates first.
  2. The browser gets a new document.
  3. The document references a JavaScript bundle that the CDN edge has not fully propagated yet.
  4. The browser loads the new page structure with old bundle behavior, or vice versa.
  5. A selector, event handler, or route guard no longer matches the rendered UI.

That is not a Selenium problem or a Playwright problem. It is a deployment consistency problem that automation exposed.

A useful distinction: cache vs stale asset vs propagation

These terms get blurred together, so I separate them this way:

  • Cache means a storage layer is intentionally reusing a response, often based on Cache-Control, ETag, Last-Modified, or service worker rules.
  • Stale asset means the browser or edge is serving an old JavaScript file for a page that expects a newer one.
  • Propagation problem means some CDN nodes or regions have new assets while others still have old ones.

A cache can be healthy and still serve an outdated asset if the invalidation strategy is wrong. A propagation problem can happen even when your cache headers are fine.

Decision tree: clear cache, bust assets, or fix deployment?

Use this before you start rewriting tests.

1) Does the failure disappear in a fresh browser context?

In Playwright, a new context gives you a clean cookie jar, storage state, and cache boundary. In Selenium, a new session gives you a cleaner baseline, though browser and driver behavior still vary by setup.

  • Yes: suspect browser cache, service worker, or local storage coupling.
  • No: continue.

2) Does the failure disappear when you bypass the CDN or hit origin directly?

If your environment allows it, compare the failing URL against origin or a debug hostname.

  • Yes: suspect CDN invalidation, edge propagation, or cache header mismatch.
  • No: continue.

3) Does the page reference a bundle hash that no longer exists or does not match the HTML shell?

Look for bundle names like app.3f2c1.js and verify the referenced file exists and matches the deployment artifact.

  • Yes: fix build/version consistency first.
  • No: continue.

4) Does a hard reload still fail, but disabling service workers makes it pass?

  • Yes: fix the service worker update path or cache strategy.
  • No: treat it like a product issue until a specific caching layer proves otherwise.

Do not clear cache blindly in CI and call the issue solved. That often hides a bad deployment contract.

What to inspect first in the network and application state

When a browser test fails on a stale asset suspicion, inspect these artifacts before touching assertions:

Response headers

Check the document and JS bundle responses for:

  • cache-control
  • etag
  • last-modified
  • age
  • via
  • CDN-specific headers such as x-cache or x-served-by

You are looking for contradictions, for example a long-lived HTML response that points to short-lived or invalidated assets, or an asset response that is cached longer than the HTML it depends on.

Asset fingerprinting

If your build uses content hashes, the HTML page should reference only the current version of the bundle. Any mix of old HTML and new JS, or new HTML and old JS, is a deployment smell.

Service worker scope

If the app registers a service worker, confirm whether it controls the route used by the test. A service worker can continue serving older cached JS even after the CDN is fixed.

Storage state

Local storage, session storage, and cookies can gate feature flags or release toggles. If your app reads a build version from storage, stale state can make the UI behave as if it were still on the old release.

Playwright debugging setup that makes stale-asset failures visible

Playwright is a good fit for this kind of debugging because it exposes browser context boundaries, network events, and request inspection. The key is to stop using a default happy-path test and start collecting evidence.

The Playwright docs are the right starting point for the API surface.

import { test, expect } from '@playwright/test';

test('capture JS responses', async ({ page }) => {
  page.on('response', async (response) => {
    const url = response.url();
    if (url.endsWith('.js')) {
      console.log(response.status(), url, response.headers()['cache-control']);
    }
  });

  await page.goto('https://example.com');
  await expect(page.locator('body')).toBeVisible();
});

That is intentionally minimal. The goal is not to assert everything. The goal is to answer:

  • Which script URL was requested?
  • What status did it return?
  • What cache headers came back?
  • Did the browser request the expected version at all?

For a deeper check, capture the actual bundle reference from the HTML and compare it to the asset request list.

Selenium debugging approach when you need the same evidence

Selenium can still do this, but the shape is different. You will usually lean more on browser logs, network proxies, or external HAR capture, depending on your grid and browser support.

If the failure is cache-related, I would not spend time on brittle waits first. I would check whether the browser session is clean and whether the test runner is inheriting persistent state between runs.

A useful pattern is to run the same test in a freshly created profile or container and compare outcomes. If the issue disappears only in a fresh profile, the root cause is outside the page itself.

When clearing cache is the right move

Clearing cache is a diagnostic step, not a fix. It is appropriate when:

  • You need to confirm whether the failure is tied to browser-local state.
  • The test suite reuses sessions or profiles across runs.
  • A service worker or local storage artifact is suspected.

It is not the right fix when:

  • The HTML references a bundle that has not propagated.
  • The CDN is serving mixed versions across edges.
  • Cache headers allow the wrong asset to live too long.

If clearing cache makes the test pass, ask what changed. A passing rerun only tells you that the browser can recover from stale state. It does not tell you the deployment contract is safe.

When to bust assets instead

Bust assets when the problem is version mismatch, not browser state.

Good signals include:

  • HTML and JS bundle names are out of sync.
  • The app loads successfully after a full deployment settles, but fails during rollout.
  • The same test fails on a subset of regions or nodes.

The fix is usually in one of these places:

  • Content-hashed filenames for JS and CSS
  • Atomic release promotion
  • CDN invalidation sequencing
  • Immutable asset hosting with short-lived HTML
  • Service worker update logic

If you can make the HTML point only to a complete release set, you remove a large class of browser test failures.

When to fix the deployment pipeline instead of the test

This is the part teams often postpone.

If the test fails because the release process exposes a temporary mismatch, the test is doing useful work. Changing the test to ignore the problem will hide a real user-facing risk.

Fix the pipeline when you see one of these patterns:

  • HTML is released before assets are fully available.
  • CDN invalidation is asynchronous, but the rollout assumes immediate consistency.
  • Multiple origins or regions serve different build versions during the same deployment window.
  • Service worker updates lag behind the HTML contract.

A safer deployment pattern is:

  1. Upload immutable assets.
  2. Wait for asset availability.
  3. Publish HTML that references those assets.
  4. Verify a sample of URLs from the CDN and origin.
  5. Promote only after the verification passes.

That verification step is cheap compared with debugging a broken nightly suite.

A compact debugging checklist

Use this sequence when a browser test fails and cache is suspected:

  1. Re-run in a fresh browser context.
  2. Disable or bypass service worker handling if the app uses one.
  3. Inspect the HTML for the bundle reference.
  4. Confirm the referenced JS file exists and matches the deployment artifact.
  5. Inspect cache-control, etag, and age on the document and script responses.
  6. Compare origin and CDN behavior.
  7. Check whether the failure is region-specific or time-sensitive.
  8. Decide whether the fix belongs in the test, the cache policy, or the release pipeline.

What not to do

Do not add random sleeps and hope the CDN catches up. That turns a versioning bug into a timing bug.

Do not globally disable caching in CI unless the purpose of the test is explicitly to validate cache-free behavior. A browser that never experiences cache is not the same as a real user browser.

Do not make selectors more vague just because the app is showing the wrong version intermittently. If the DOM structure changes between build versions, that is often the signal you need.

My recommendation

If your suite fails only when a CDN or aggressive cache serves stale JavaScript, treat the failure as a deployment consistency problem first and a test problem second.

I would use this order of operations:

  • First, prove whether the failure survives a fresh browser context.
  • Second, compare CDN and origin asset versions.
  • Third, inspect cache headers and service worker behavior.
  • Fourth, fix release ordering, asset fingerprinting, or invalidation logic.
  • Last, adjust the test only if the assertion was genuinely coupled to unstable release state.

That approach keeps the suite honest. It also prevents teams from papering over a broken rollout process with retries and cache wipes.

FAQ

Why do browser tests fail only after a deployment?

Because the HTML shell and JavaScript bundle can be served from different versions if the CDN, browser cache, or service worker is not updated atomically.

Should I always clear browser cache in CI?

No. Clearing cache can help diagnose stateful failures, but it can also hide deployment bugs that real users will hit.

How do I tell cache corruption from a real regression?

Re-run in a fresh context, compare CDN and origin responses, and verify that the HTML references the same bundle version the deployment produced.

Can service workers cause stale JavaScript failures?

Yes. A service worker can keep serving cached assets after the page has updated, which makes the app appear inconsistent across reloads.

Is cache-control the only header that matters?

No. ETag, Last-Modified, Age, and CDN-specific headers also help explain whether the browser or edge is serving old content.

When should the deployment pipeline be fixed instead of the test?

When the test exposes an inconsistent release state, such as HTML publishing before dependent assets are fully available, or invalidation finishing after traffic has already switched.