What to Log When Playwright Tests Fail Only in CI but Pass Locally

If a Playwright test passes on your laptop but fails in CI, the problem is usually not that the test is “mysteriously flaky” in the abstract. It is one of a few concrete categories: timing drift, environment drift, missing dependencies, network differences, parallelism, or a real product bug that only shows up under CI conditions.

The fastest way to stop guessing is to log the right things. Not everything, just the signals that help you compare a clean local run against a failing CI run. When I debug this class of issue, I want enough evidence to answer four questions quickly:

Did the browser see the same app state locally and in CI?
Did the failure happen because something arrived late, not because it never arrived?
Did the environment differ in a meaningful way?
Is this a test problem, an application problem, or a setup problem?

This checklist is the set of logs and artifacts I reach for when Playwright tests fail in CI but pass locally. It is aimed at SDETs, QA leads, DevOps engineers, and frontend developers who need to debug CI-only failures without turning every run into a firehose of useless output.

The short version, log these first

If you only add a few things, start here:

Playwright trace artifacts for failed tests
Browser console logs
Network request and response logs for the failing flow
Timing data around the exact action that fails
Environment metadata, including OS, browser, Playwright version, and CI runtime details
Screenshots or videos at the failure point
Test retries, if enabled, with a clear record of which attempt passed or failed

The goal is not “more logs”. The goal is to make a CI failure reproducible enough that local and CI runs can be compared line by line.

1) Log the execution context before you inspect the failure

A CI-only failure is often caused by hidden drift. If your logs do not capture the run context, you will waste time chasing symptoms.

Record this information at the start of the test run or in a setup step:

Git commit SHA
Branch name
CI provider and job identifier
Node.js version
Playwright version
Browser name and version
Operating system and container image, if applicable
Number of workers used by the test run
Retry count and whether retries are enabled
Environment variables that affect the app or test behavior
Base URL and API endpoints used by the test suite

In many teams, this can live in the CI job summary or be printed once per run. The point is to make a failure traceable back to the exact runtime conditions.

A small helper can print this data in a structured way:

import { test } from '@playwright/test';

test.beforeAll(async () => { console.log(JSON.stringify({ commit: process.env.GITHUB_SHA, branch: process.env.GITHUB_REF_NAME, node: process.version, playwright: require(‘@playwright/test/package.json’).version, ci: process.env.CI, workers: process.env.PLAYWRIGHT_WORKERS, baseUrl: process.env.BASE_URL })); });

Structured logs are better than ad hoc text because they are easier to search, diff, and feed into your CI system’s artifacts.

2) Always capture trace artifacts on failure

For Playwright, trace files are the single most useful artifact for CI-only failures. A trace gives you a step-by-step view of the test: DOM snapshots, network activity, console messages, screenshots, and action timing.

If you are not already recording traces for failed runs, do that first. In playwright.config.ts:

import { defineConfig } from '@playwright/test';

export default defineConfig({ use: { trace: ‘retain-on-failure’, screenshot: ‘only-on-failure’, video: ‘retain-on-failure’ } });

Why this matters for CI-only failures:

A trace shows whether the element existed when the action ran
It shows whether a network request was still pending
It can reveal a slow hydration or a rendering race
It often exposes a locator that is too broad or too eager

When I inspect a trace, I look for the moment the failure begins, not just the assertion that fails. If the UI was still loading, the test may be too early. If the selector resolved to a stale or hidden element, the locator may be wrong. If a request got a 500 in CI but not locally, the app or environment is the likely cause.

3) Log browser console output, not just test runner output

Playwright’s own failure message is usually not enough. Browser console logs often show the real clue, especially for frontend apps that hydrate asynchronously or rely on client-side data.

Capture console, pageerror, and requestfailed events for the failing test or the specific suite:

import { test } from '@playwright/test';

test('checkout flow', async ({ page }) => {
  page.on('console', msg => console.log(`console:${msg.type()}:${msg.text()}`));
  page.on('pageerror', err => console.log(`pageerror:${err.message}`));
  page.on('requestfailed', req => {
    console.log(`requestfailed:${req.method()} ${req.url()} ${req.failure()?.errorText}`);
  });

await page.goto(‘/checkout’); });

Useful browser logs include:

JavaScript errors during hydration
CORS or CSP violations
Failed API requests
Warnings from the app framework, such as React hydration mismatch messages
Resource loading failures, such as fonts, scripts, or images that affect layout

Do not log every console message for every test in every run if the suite is large. That gets noisy fast. A common compromise is to enable deeper browser logging only for failed tests or for a targeted debug job in CI.

4) Log network activity around the failing step

A large share of CI-only failures are really network timing or backend dependency problems. In local runs, your laptop may have lower latency, fewer competing jobs, or cached auth. In CI, you may be on a cold container with shared resources, a different DNS path, or a rate-limited test environment.

For the failing test, log the relevant requests and responses:

URL and method
Status code
Duration
Request headers that matter, such as authorization or correlation IDs
Response body snippets for non-2xx responses, if safe to capture

Example:

page.on('response', async response => {
  if (response.url().includes('/api/checkout')) {
    console.log(JSON.stringify({
      url: response.url(),
      status: response.status(),
      method: response.request().method()
    }));
  }
});

If you need more detail, add request timing around the exact operation:

typescript

const start = Date.now();
await page.getByRole('button', { name: 'Place order' }).click();
await page.waitForResponse(resp => resp.url().includes('/api/orders') && resp.status() === 200);
console.log(`order flow took ${Date.now() - start}ms`);

This helps you distinguish:

Slow backend response, which may require a longer wait or indicate a performance issue
Missing request, which may point to a UI event problem
Failing request, which can be a genuine application or environment bug

5) Log action timing around the exact failure point

If a test passes locally and fails in CI, the first suspicion is often a race condition. Timing logs tell you whether the test is simply too optimistic.

Add timestamps before and after the step that fails, plus any prerequisite waits. This is especially useful for actions like:

waiting for a spinner to disappear
clicking a button after a route transition
reading a toast message that appears after async work
asserting a row count in a table that is loaded progressively

Example:

typescript

const t0 = Date.now();
await page.goto('/dashboard');
console.log(`goto:${Date.now() - t0}ms`);

const t1 = Date.now();

await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
console.log(`heading-visible:${Date.now() - t1}ms`);

Timing logs are valuable because they show variance. A step that takes 300ms locally and 6 seconds in CI is not the same problem as a step that never happens.

If you see large timing spread, that usually points to one of these:

app initialization is slower in CI
resources are constrained in the CI container
the test relies on a hard-coded wait that is too short
parallel execution is increasing contention

6) Log the DOM state that matters, not the whole page source

When a selector fails, it is tempting to dump the entire HTML. That usually creates huge logs that are hard to read and often still miss the real issue. Instead, log the smallest DOM fragment that explains the failure.

Useful examples:

the target element’s text content
whether an element is visible or disabled
the number of matching nodes
relevant attributes such as aria-label, data-testid, or disabled

Example:

typescript

const button = page.getByRole('button', { name: 'Pay now' });
console.log('button count', await button.count());
console.log('button visible', await button.first().isVisible());
console.log('button disabled', await button.first().isDisabled());

If a locator matches multiple nodes in CI but only one locally, you may have a layout or timing issue that changes the DOM structure at runtime. If the element exists but is disabled or hidden, the page may still be mid-transition when the test runs.

7) Log worker count, retries, and test isolation details

A test that passes alone may fail in CI because it runs in parallel with other tests. This is one of the most overlooked sources of CI-only failures in UI automation.

Record:

number of workers
shard number, if using sharding
retry count
whether tests run serially or in parallel
whether a shared account, database, or queue is reused

If the failure disappears with one worker and returns with many, you may have hidden shared state. That can mean:

tests share the same user account
tests depend on cleanup that is not deterministic
backend data collides across parallel jobs
an idempotency bug is exposed only under concurrent load

When this happens, the log should tell you whether the failing case is truly isolated or just the first visible symptom of test data contention.

8) Log the environment differences that matter most

“Works locally” is not a useful diagnosis unless you can compare the local machine against the CI runtime. For browser tests, these environment gaps are common:

different browser versions
headless versus headed behavior
different viewport sizes
different locale or timezone
different CPU and memory constraints
missing fonts, system libraries, or certificates
containerized network rules or proxies

At minimum, log viewport, locale, timezone, and browser channel if you rely on them.

import { defineConfig } from '@playwright/test';

export default defineConfig({ use: { viewport: { width: 1280, height: 720 }, locale: ‘en-US’, timezoneId: ‘UTC’ } });

If a test depends on date formatting, relative time labels, or responsive layout, environment drift can change the DOM just enough to break a locator or assertion.

A lot of “flaky” browser tests are actually environment-sensitive tests that were never pinned down enough to be deterministic.

9) Log authentication and session state carefully

CI-only failures often happen in auth-protected flows. Local runs may use cached browser state, while CI starts from a clean profile on every job.

Log:

whether storage state is reused
whether login is performed through the UI or via API setup
whether tokens expire during the run
whether the test starts authenticated or anonymous
whether the same account is shared across jobs

For example, if a local developer browser has a long-lived session but CI uses a fresh context, the test may fail because it forgot to navigate through the login flow. Conversely, a login flow that depends on third-party identity providers may be unstable in CI because of bot detection, MFA, or rate limiting.

When possible, prefer deterministic setup through API-level auth seeding, and log the resulting account or session marker. That makes failures easier to separate from login infrastructure problems.

10) Log app and API correlation IDs when available

If your app already emits correlation IDs, request IDs, or trace IDs, make sure your test logs capture them. These identifiers are extremely useful when a failure starts in the browser but ends in a backend service.

Good candidates to log:

X-Request-Id or equivalent response headers
server trace IDs
API response identifiers for created records
job or tenant identifiers in multi-tenant apps

A browser test can then be matched to server logs without guessing.

Example pattern:

page.on('response', async response => {
  const id = response.headers()['x-request-id'];
  if (id) {
    console.log(`request-id:${id} ${response.status()} ${response.url()}`);
  }
});

If your backend logs the same ID, you can quickly determine whether the issue is in the frontend, API, or downstream service.

11) Log screenshots only where they help, and name them clearly

Screenshots are useful, but only if they are easy to match to the failure and understand in context. A screenshot without test name, retry attempt, and timestamp is less useful than it should be.

Good screenshot metadata includes:

test name
project name, such as chromium or firefox
retry attempt
timestamp
failing step or assertion

If you store screenshots as artifacts, make sure the filename encodes enough context to compare runs.

A practical pattern is to rely on Playwright’s built-in failure capture, then add targeted screenshots for known problematic steps only when tracing a difficult issue. Too many screenshots can obscure the signal, especially in suites with many failures.

12) Log the last successful step before the failure

When a failure occurs, one of the most valuable pieces of data is the last step that definitely succeeded. It narrows the search space dramatically.

For example:

Page loaded successfully
User logged in successfully
Product added to cart successfully
Checkout page rendered successfully
Click on “Place order” failed, or next assertion never became true

That single boundary often tells you whether you are dealing with a navigation issue, a state transition issue, or a backend dependency issue.

You can make this explicit in logs:

typescript console.log(‘step: checkout page loaded’);

await expect(page.getByRole('heading', { name: 'Checkout' })).toBeVisible();

console.log(‘step: place order clicked’);

await page.getByRole('button', { name: 'Place order' }).click();

This sounds basic, but in practice it saves time because CI logs are often read under pressure. Clear step markers help you avoid re-running the same test six times just to see where it got stuck.

13) Separate evidence for timing issues from evidence for product regressions

Not every CI-only failure is a flaky test. Some are real regressions that just happen to show up under different timing. Logging should help you distinguish the two.

A likely test issue looks like this:

DOM element appears later in CI than locally
selector is too specific or too brittle
test assumes synchronous rendering
fixed timeout is just a little too short
retry makes the test pass

A likely product issue looks like this:

network request fails in CI with a real error code
console shows application exception
trace shows the UI is broken, not merely slow
same user journey fails consistently under CI conditions
the backend response is malformed or missing fields

When a failure is ambiguous, log both the browser-side symptom and the network response. That often separates “wait longer” from “fix the app”.

14) Use one debug mode, not five conflicting ones

Teams often pile on debug settings until the output becomes unreadable. I see this a lot in CI pipelines where people enable screenshots, video, trace, verbose Playwright logging, shell set -x, and application debug mode all at once.

That can be useful temporarily, but it is not a sustainable default.

A cleaner approach is to define a debug profile for failure analysis, for example:

traces on failure
screenshots on failure
browser console and request logs for the specific test
a single environment metadata dump at startup
targeted network logging for the suspected endpoint

Then keep the normal CI run lean. If you need more detail, trigger a dedicated debug job rather than making every pipeline noisy.

15) Know which logs belong in the test and which belong in the pipeline

Some data belongs inside the test, and some belongs in CI.

Put in the test:

step markers
page-level console output
request and response logs for relevant endpoints
selector state checks
timing around suspicious interactions

Put in the pipeline:

Node, OS, and container metadata
Playwright install and browser versions
shard, worker, and retry settings
environment variables and secrets presence checks, without printing the secrets themselves
artifact links and retention policy

This split matters because if a test fails, you want the evidence in one place. If the pipeline is misconfigured, you want the job logs to show that before the browser even starts.

A practical CI-only failure checklist

Use this as a runbook when a Playwright test is green locally and red in CI:

Capture trace, screenshot, and video on failure
Print run context, including versions and CI metadata
Log browser console errors and page errors
Log relevant network requests and non-2xx responses
Record timing around the failing step
Check whether the problem appears only with parallel workers
Compare local and CI viewport, locale, and timezone
Verify auth state, storage reuse, and test data isolation
Correlate browser failures to backend request IDs
Determine whether the failure is a race, an environment mismatch, or a real regression

A small Playwright config that helps a lot

If you want a baseline that is useful in most teams, I usually start here and then tighten or expand it based on the app:

import { defineConfig } from '@playwright/test';

export default defineConfig({ use: { trace: ‘retain-on-failure’, screenshot: ‘only-on-failure’, video: ‘retain-on-failure’, actionTimeout: 10_000, navigationTimeout: 30_000 }, retries: process.env.CI ? 1 : 0 });

The retry is not there to hide the problem. It is there to help classify it. If the second attempt passes and the trace shows a slow render or a transient network miss, that is different from a deterministic failure on every run.

Final thoughts

When Playwright tests fail in CI but pass locally, the mistake is usually not a lack of information, it is logging the wrong information. Browser automation lives at the intersection of DOM timing, network behavior, runtime environment, and test data setup. A useful debug trail should reflect all four.

If you capture execution context, traces, browser logs, network activity, and timing around the failure point, you can usually tell which bucket the issue belongs to within one or two iterations. That is the difference between a flaky test backlog that grows forever and a test suite that becomes easier to trust over time.

For teams building a wider testing strategy, this same discipline applies across software testing and test automation: collect evidence that answers the question, not just evidence that something broke.

If you are tightening up your CI pipeline, start with the smallest possible set of logs that distinguishes local-only success from CI-only failure. In practice, that is usually enough to expose the real problem before it turns into a long debugging session.