AI-generated test code in CI is not automatically useful just because it compiles and opens a browser. I care about a narrower question: will this test keep paying its way after the first merge, or will it become another fragile artifact that my team has to babysit? That is the real bar. A test that looks impressive in a demo can still create a long tail of review overhead, selector churn, debugging ambiguity, and flaky test risk once it starts running against a changing application in a real pipeline.

I treat generated tests the same way I treat any code that is about to become part of a delivery system, with one extra layer of skepticism. The test is not just a script, it is an operational commitment. It will run in CI, consume browser minutes, fail on bad days, and become part of the team’s definition of confidence. If AI has helped draft it, fine. But the review standard should be higher, not lower, because the cost of hidden assumptions is paid later.

The question is not whether AI can write test code

AI can usually produce plausible Selenium or Playwright code quickly. That is not the interesting part. The interesting part is whether the generated test reflects the shape of the product, the testing strategy, and the team’s maintenance model.

A useful way to think about this is to separate three layers:

  • Syntax correctness, does the code run
  • Test correctness, does it check something meaningful
  • Operational correctness, will the test stay readable, stable, and owned in CI

Most AI-generated test code clears the first layer. Many snippets partially clear the second. Fewer clear the third. The third layer is where teams usually lose time, because that is where flaky selectors, overbroad waits, and unclear assertions turn into recurring triage work.

A test that is easy to generate but hard to own is not a productivity gain, it is deferred maintenance.

That is why my review process starts with architecture, not line-by-line style. I want to know whether the generated test fits the suite, whether it is trying to do too much, and whether it will make future changes more expensive.

My review checklist for AI-generated test code in CI

Here is the framework I use before I trust a generated test enough to let it run in the main pipeline.

1) Does the test match the purpose of the suite?

A good generated test should have one clear job. If the test covers login, profile editing, and billing in one flow, it may be demonstrating something, but it is also increasing failure blast radius. In CI, that becomes a problem when one unrelated change breaks a large end-to-end path and obscures the root cause.

I ask:

  • Is this a smoke test, a regression test, or an end-to-end user journey test?
  • Is the test trying to validate behavior that belongs in API or component tests instead?
  • Would a smaller test give a faster, clearer signal?

For example, a generated Playwright test that signs in, adds an item to a cart, checks out, and verifies an email receipt may be too broad for a normal merge pipeline. A better split is often:

  • one fast smoke test for auth and route health
  • one focused UI test for cart behavior
  • one API or integration test for order submission
  • one separate async verification for email delivery

The general testing model behind this is not controversial, it follows the basic test automation principle of using the right layer for the right signal, instead of forcing the browser to prove everything. If you need a refresher on the underlying categories, the standard definitions for software testing, test automation, and continuous integration are worth keeping in mind.

2) Are the selectors resilient, or just convenient?

Selector strategy is one of the fastest ways to tell whether generated code will survive in CI.

I look for selectors in this order of preference:

  1. data-testid or another explicit test hook
  2. stable accessibility-based locators when the app supports them well
  3. clearly semantic locators tied to durable UI structure
  4. brittle CSS chains, positional selectors, or text that changes with copy edits

Playwright supports robust locator patterns and recommends locators that reflect user intent, not DOM accident. The Playwright docs are a good baseline for the style of assertions and locators I expect generated tests to follow.

A generated test like this is usually acceptable:

typescript

await page.getByTestId('submit-order').click();
await expect(page.getByTestId('order-confirmation')).toBeVisible();

This is much better than a selector chain that depends on layout nesting or a text string likely to change in a copy review. I do not require every test to use data-testid, but I want the choice to be intentional and stable. If AI is inventing selectors from visible text alone, I treat that as a warning sign unless the product explicitly treats that text as part of the contract.

A common failure mode is generated code that grabs the first matching element on the page because it was easier to synthesize. That may pass locally and fail in CI when the page contains duplicate labels, modal variants, or hidden content. If the app has multiple similar elements, the locator needs to narrow scope with intent.

3) Do the assertions prove behavior, or just observe output?

Generated tests often contain weak assertions. They click a button, wait for navigation, then assert that the page exists or that a string appears somewhere. That is not always bad, but it is often too shallow to justify a place in the pipeline.

I want to see assertions that answer one of these questions:

  • Did the user-visible state change as expected
  • Did the system prevent an invalid action
  • Did the correct record, banner, or route appear after the operation
  • Did the test fail for the right reason when a precondition is broken

Compare these two examples:

typescript

await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByText('Saved')).toBeVisible();

This is serviceable, but it may be too generic if multiple save paths exist.

typescript

await page.getByRole('button', { name: 'Save profile' }).click();
await expect(page.getByTestId('profile-last-updated')).toContainText('Just now');

This is more specific and more diagnosable. It ties the assertion to the feature under test, not just to a positive toast.

What I do not want is a test that only proves the page did not crash. If that is the intended signal, it should be treated as a smoke check and named accordingly. A vague assertion is a hidden scope decision.

4) Does the code handle waits in a way that fits the framework?

Generated test code often gets waiting wrong. It either sleeps too much, which slows CI and hides timing problems, or it waits on the wrong condition, which creates flakes.

In Playwright, I want to see built-in waiting behavior used intentionally, not waitForTimeout as a first answer. The framework already provides locators and assertions that wait for expected state transitions. Manual sleeps are usually a sign that the generator is guessing instead of observing.

In Selenium, I want explicit waits, not hard-coded pauses. That means waiting on a condition that matches the UI state you actually need, such as element visibility or URL change, rather than sleeping and hoping the app gets there in time.

A bad pattern looks like this:

typescript

await page.click('button#save');
await page.waitForTimeout(5000);
await expect(page.locator('.toast')).toBeVisible();

A more defensible pattern is to wait on the state directly:

typescript

await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByTestId('toast')).toHaveText(/saved/i);

If AI-generated test code in CI relies heavily on sleeps, I expect a future triage burden. That burden shows up as slow pipelines and intermittent failures that are hard to reproduce.

5) Is the test structured for ownership, not just execution?

Ownership boundaries matter. A test is easier to maintain when someone can answer who owns the fixture, who owns the selectors, and who is expected to update the code when the UI changes.

I look for structure that makes ownership obvious:

  • one test file per feature or flow, not a pile of unrelated steps
  • shared helpers for repeated login or setup, but not a giant abstraction layer that hides intent
  • page objects or component abstractions only when they reduce duplication and reflect stable UI boundaries
  • clear naming that matches business behavior, not implementation trivia

This is where generated code often becomes expensive. AI can produce a dense abstraction, but if it introduces a helper for every action, the test suite becomes a maze. On the other hand, a totally flat script can be equally painful if it repeats setup in ten places.

The practical goal is not elegance for its own sake, it is reducing the cost of change. If a button label changes, I should know which helper to update. If a flow changes, I should know whether the test still represents the product requirement or whether it now encodes obsolete implementation details.

6) How much review time will the code cost after it merges?

This is the question most teams forget to ask. AI-generated test code can save five minutes today and cost five hours over the next month.

I estimate future review cost by looking for:

  • duplicated setup logic
  • overuse of magical helper functions
  • unclear naming for selectors and assertions
  • dependence on flaky page state or race conditions
  • broad end-to-end flows that will fail for unrelated reasons
  • unclear failure messages when assertions break

A generated test that is slightly verbose but obvious is usually cheaper than a compact, clever script nobody wants to touch. The best review outcome is not “this looks impressive,” it is “the next person can debug this at 9:00 a.m. without asking the generator for help.”

What I want to see in generated Playwright tests

When the output is Playwright, I want it to use the framework idioms that reduce waiting pain and improve traceability.

Some signs of good generated output:

  • locators based on roles, labels, or stable test ids
  • assertions that use expect(...) against state, not raw timeouts
  • small, readable steps
  • a clear separation between navigation, action, and verification
  • no unnecessary framework gymnastics

A reasonable pattern might look like this:

import { test, expect } from '@playwright/test';

test('user can update profile name', async ({ page }) => {
  await page.goto('/settings/profile');
  await page.getByLabel('Display name').fill('Alex Rivera');
  await page.getByRole('button', { name: 'Save changes' }).click();
  await expect(page.getByTestId('save-status')).toHaveText('Saved');
});

That test is not perfect, but it is readable and reviewable. It reveals the business action, the control used, and the expected state. It also makes failures easier to interpret than a script full of chained selectors and retries.

What I would still inspect carefully:

  • whether /settings/profile is stable enough for CI
  • whether the save-status hook is really durable
  • whether the assertion should check a persisted value after reload instead of a toast
  • whether the test should reset data to avoid cross-run interference

If the generated test adds cleanup and test data setup in a way that makes ownership clearer, that is a plus. If it invents a lot of fixture machinery without a clear need, I usually trim it.

What I look for in Selenium-generated code

Selenium can be perfectly fine in CI, but AI-generated Selenium code often reveals weaker waiting and locator habits than a human would choose after years of debugging.

I am especially skeptical of code that:

  • uses time.sleep() instead of explicit waits
  • leans on XPath when a simpler locator would do
  • asserts too soon after clicking
  • mixes setup, action, and assertion in a way that makes failures hard to localize

A more defensible Selenium example looks like this:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10) driver.get(‘https://example.com/login’) driver.find_element(By.CSS_SELECTOR, ‘[data-testid=”email”]’).send_keys(‘user@example.com’) driver.find_element(By.CSS_SELECTOR, ‘[data-testid=”password”]’).send_keys(‘secret’) driver.find_element(By.CSS_SELECTOR, ‘[data-testid=”login”]’).click() wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ‘[data-testid=”dashboard”]’)))

This is still just a starting point, but the waiting is explicit and the selectors are stable enough to review. If AI-generated code instead uses deeply nested XPath expressions, I assume it will be fragile unless there is a very specific reason for that choice.

Flaky test risk is a code review problem, not just a runtime problem

A lot of flaky test risk is visible before the first CI run. I look for the following patterns because they predict future instability:

Hidden dependency on shared state

If the test assumes a particular account, seed record, or environment toggle, it may pass only when the pipeline state is clean. That is not robustness, it is luck. Generated tests should make setup explicit or be isolated enough that the dependency is obvious.

Excessive reliance on UI timing

If the application has asynchronous work, a generated test must wait for the right signal. Otherwise, it becomes a race between CI and the browser.

Overassertion on volatile UI text

Copy changes should not break core behavior tests unless the text is part of the contract. If AI has asserted on every visible string, expect churn.

Multi-purpose flows

A long generated scenario can fail for ten different reasons, which makes reruns expensive. Split the test if the failure modes are unrelated.

In CI, a flaky test is not just noisy, it erodes trust in every other red build.

That is why I care about the shape of generated code before I care about its coverage claims. A broad suite of brittle tests can make teams slower, not safer.

A practical test code review checklist for AI output

When someone brings me AI-generated test code, I usually ask them to walk through this checklist before merge:

  • What business behavior does this test protect?
  • Would this belong in UI, API, integration, or component level automation?
  • Are the selectors stable and intention-revealing?
  • Are waits tied to real state changes?
  • Do the assertions prove the feature, or only the page response?
  • Is the test isolated, or does it depend on shared data?
  • What will break if the UI copy changes?
  • Who owns the fixture, helper, or page object if this needs adjustment later?
  • How many files would need to change if the feature flow changes?
  • Will this test still be understandable six months from now?

This is the heart of a useful test code review checklist. It is not about policing style. It is about reducing maintenance surprise.

When I would still accept AI-generated test code quickly

Not every generated test needs a long debate. I am comfortable merging AI-assisted code faster when all of these are true:

  • the test is small and focused
  • selectors are stable and explicit
  • the assertions check meaningful state
  • the code follows the team’s established framework style
  • the failure mode is easy to diagnose
  • the team already owns the relevant fixture and page surface

In other words, AI is most useful when it produces a first draft that already respects the team’s automation boundaries. It is not a substitute for judgment about what belongs in CI.

When I would slow down or reject it

I slow down when AI-generated test code tries to do too much, hides intent, or creates a maintenance path nobody wants. Specific red flags include:

  • a giant all-in-one journey test that tries to verify too many business rules
  • selectors that are brittle by design
  • unbounded retries or sleeps added to cover timing uncertainty
  • helper layers that obscure the flow instead of clarifying it
  • assertions that would pass even if the feature were partially broken
  • code that does not fit the team’s ownership model

If a test requires a lot of explanation in review, that is usually a sign that the generated draft is more expensive than it looks.

The real cost is not generation, it is governance

Teams sometimes frame the conversation as whether AI can write tests faster than humans. That is the wrong comparison. The meaningful comparison is whether the whole system, generation plus review plus maintenance plus CI triage, is cheaper and safer than hand-built automation alone.

That is why I care about governance. A good governance model for AI-generated test code in CI should answer:

  • what types of tests AI is allowed to draft
  • which selectors and assertions are acceptable
  • what review standard applies before merge
  • who owns follow-up fixes when the UI changes
  • how flaky failures will be triaged
  • when a generated test should be rewritten by hand instead of patched repeatedly

Without those rules, AI output tends to drift into the path of least resistance, which is often the most expensive path later.

My bottom line

I trust AI-generated test code in CI only when it behaves like disciplined engineering work, not like autocomplete for the entire quality strategy. It needs clear scope, stable selectors, strong assertions, sane waiting, and obvious ownership. If it does not have those properties, it may still be useful as a draft, but it is not ready to become a permanent resident of the pipeline.

The goal is not to avoid AI. The goal is to stop it from importing invisible maintenance debt into the test suite. If a generated test reduces review time today but increases flaky test risk and ownership ambiguity tomorrow, it is not a gain. If it helps produce small, readable, durable tests that fit the team’s CI discipline, then it is worth considering.

That is the standard I use: not whether the code looks clever, but whether it will still be worth running after the team has forgotten how it was generated.