Flaky tests are not just an annoyance, they are a reliability problem with operational consequences. Every time a test fails for reasons unrelated to product behavior, it taxes the same systems that should be accelerating delivery: CI capacity, engineer attention, release confidence, and triage time. The hard part is that not every flaky test deserves the same fix. Sometimes a retry is a reasonable pressure valve. Sometimes the test is telling you the system under test is coupled to shared state and needs isolation. Sometimes the test is so entangled, slow, and brittle that a rewrite is the cheapest path forward.

When I decide how to fix flaky tests, I do not start with the tool. I start with the failure mode, the release risk, and the cost of ongoing ownership. A temporary mitigation can be rational. So can a permanent structural fix. The mistake is treating them as interchangeable.

The decision starts with triage, not ideology

My first question is simple: what kind of flake is this?

I usually group flaky test triage into four buckets:

Transient environment failure
- Browser process crash
- Network hiccup
- CI runner resource starvation
- External dependency unavailable
Timing and synchronization failure
- Assertions racing the UI
- API eventually consistent but test assumes immediate consistency
- Elements present, but not yet interactable
State contamination failure
- A test depends on previous tests
- Shared account, shared queue, shared database records
- Parallel execution exposes hidden coupling
Test design failure
- Locator is too brittle
- The test asserts implementation details instead of user-visible behavior
- One test is doing too much, so any small change breaks it

That classification matters because it tells me whether the fix belongs in the test, the environment, the product, or the architecture around the test suite. For background on how automated testing and CI fit into delivery systems, the general concepts are well documented in software testing, test automation, and continuous integration.

If a test only fails when the environment is sick, a retry may be acceptable. If it fails because the test and the system are coupled poorly, retries only hide the debt.

My default bias: do not normalize failure

I have a strong bias against using flaky test retries as a permanent solution. Retries reduce noise, but they also reduce signal. If a test fails 1 in 20 times and the pipeline auto-retries until it passes, the team may never feel the pain strongly enough to fix the real issue. The release pipeline looks green, but reliability degrades quietly.

That does not mean retries are always wrong. It means retries should be explicit about what they are buying us:

keep the pipeline moving while the underlying defect is being investigated
mask known transient infrastructure issues with a bounded blast radius
absorb rare external dependency failures that the team does not control

Retries are a tool, not a diagnosis. If I cannot explain the failure mode, I do not want retries to be the final answer.

When flaky test retries are acceptable

I will tolerate flaky test retries when all of the following are true:

The failure is rare and clearly transient
The test covers a path already covered by other stable tests
The team has a ticket and owner for the underlying fix
The retry is bounded, visible, and monitored
A failure still produces enough evidence to diagnose the root cause later

In practice, that means a single retry, maybe two, at the CI layer, with logs and artifacts preserved. Not infinite retry loops. Not silent automatic healing. Not a hidden mechanism that makes the pipeline look healthy while the same test fails every other night.

Here is a simple Playwright retry configuration that is acceptable as a temporary control, not a cure:

import { defineConfig } from '@playwright/test';

export default defineConfig({ retries: 1, use: { trace: ‘on-first-retry’, screenshot: ‘only-on-failure’, }, });

This is useful because it keeps the blast radius small and preserves debugging evidence. The failure is still visible in CI, and the trace gives you a path to root cause analysis. If the same test keeps needing retries, that is a signal to stop treating it as a transient issue.

The simplest fix is often test isolation

If a flaky test is failing because it shares state, I usually start by isolating it before I consider rewriting it. Test isolation is not glamorous, but it is often the highest-return fix because it improves both correctness and parallelism.

Common isolation problems include:

Shared test accounts that accumulate state
Reused database rows or IDs across tests
Non-unique email addresses, usernames, or workspace names
Cleanup that depends on order of execution
Tests that assume a database starts empty when it does not

A test suite with weak isolation often gets worse as it scales. The more parallelism you add, the more hidden coupling turns into visible flakiness.

What good isolation looks like

Good isolation usually means each test owns its setup and teardown boundaries. That can look like:

create a fresh user per test
generate unique data with a timestamp or UUID
seed data through APIs instead of UI flows
reset mutable state between tests
avoid shared mutable accounts in a parallel suite

Example in Selenium with Python, where the test creates unique data and avoids reusing a shared account:

import uuid
from selenium import webdriver

email = f”test-{uuid.uuid4()}@example.com”

driver = webdriver.Chrome() driver.get(“https://app.example.com/signup”)

fill form using unique email

The code is not the important part. The principle is. If the suite depends on global state, the test is not isolated, and retries only postpone the failure.

Isolation is better than retries when the same pattern repeats

I usually prefer isolation over retries when I see one of these patterns:

failures grow as the suite runs longer
parallel execution increases failure rate
tests pass alone but fail in the full suite
the same resource name, account, or record shows up in multiple failures

Those are not random flakes. They are dependency leaks.

If the suite needs a database reset or a deterministic fixture builder, that work pays off across many tests. It also reduces time spent on flaky test triage, which is often more expensive than the original fix.

When a rewrite is the right call

Sometimes the honest answer is that the test should be rewritten. I do not mean rewritten because the assertion changed once. I mean the current structure is so brittle that incremental repair is wasteful.

I reach for a rewrite when the test has several of these characteristics:

long, multi-purpose flow with many assertions
hard-coded sleeps or fragile waits everywhere
locators tied to CSS structure instead of stable app semantics
repeated failures after multiple narrow fixes
expensive runtime that slows CI enough to discourage execution
the test is so coupled to UI implementation that product changes constantly break it

A rewrite is not an admission of defeat. It is often the cheapest way to reduce long-term maintenance cost.

Signs the old test is the real problem

If a test requires constant patching, I ask whether it is trying to prove too much at once. For example, a single end-to-end test that creates an account, verifies onboarding, updates profile data, triggers a notification, and checks analytics is not one test. It is a chain of assumptions. Any one step can fail for many unrelated reasons.

That is a maintenance trap.

A better structure is to split the critical path into narrower checks:

one test for account creation
one test for profile update
one test for notification delivery
one test for analytics side effect, if that is worth automating at all

Playwright makes this kind of refactor practical because it encourages isolated test files and reusable fixtures. A focused assertion is much easier to reason about than a monolithic script:

import { test, expect } from '@playwright/test';

test('user can update profile name', async ({ page }) => {
  await page.goto('/settings/profile');
  await page.getByLabel('Display name').fill('New Name');
  await page.getByRole('button', { name: 'Save' }).click();
  await expect(page.getByText('Profile updated')).toBeVisible();
});

That test still can fail, but the failure surface is much smaller. The diagnostic value is higher. The maintenance burden is lower.

Rewrite when the test architecture is upside down

I also rewrite when the entire approach is wrong. Examples:

UI test used to validate business logic that belongs in API or unit tests
a brittle XPath chain is compensating for missing test identifiers
a test is dependent on arbitrary waits because the app has no stable readiness signal
one test file has become a miniature framework nobody wants to touch

A rewrite is often justified when the supporting product code has changed enough that the original assumptions no longer hold. In that case, continuing to patch is like keeping old scaffolding around a new building.

A practical decision framework I use

When I need to decide between retry, isolation, or rewrite, I ask these questions in order.

1. Is the failure clearly transient?

If yes, and the underlying system is external or uncontrollable, a limited retry can be justified. That includes browser crashes, brief network errors, and infrastructure blips.

If no, do not hide it behind retries.

2. Does the test fail because of shared or hidden state?

If yes, fix isolation first.

This is especially true if:

the same test fails more in parallel runs
rerunning alone makes it pass
other tests influence its outcome

3. Is the test structure itself the problem?

If the test is fragile, slow, and difficult to understand, a rewrite is often cheaper than repeated patching.

4. What is the release risk if we leave it as-is?

This is the part teams often skip. A flaky test in a rarely run nightly suite is not the same as a flaky test gating production deployment. If a test gates release decisions, its false failure rate matters more than its convenience.

5. How much ownership time will this consume going forward?

A fix is not done when the test passes once. The real question is how often it will need attention after the patch.

That is where ownership cost matters more than code churn. A retry has low implementation cost but potentially high ongoing trust cost. An isolation fix has moderate upfront cost but can eliminate repeated triage. A rewrite has the highest initial cost but can lower total maintenance if the original test was a bad design.

Release risk changes the answer

The same flaky test can deserve different treatment depending on where it sits in the delivery pipeline.

Low-risk, low-signal tests

If the test guards a non-critical path and has nearby coverage, retries or delayed repair may be acceptable while you schedule better work.

Release-gating tests

If a test blocks deployment, the tolerance for uncertainty should be much lower. At that point, a flaky test is not just noisy, it is actively distorting release decisions. Teams can end up ignoring failures, which is worse than a few false alarms.

Security, billing, and data integrity paths

For high-stakes flows, I want tests to be boring. If they are flaky, I treat that as a production risk. In those cases, I prefer a structural fix over retries because the cost of false confidence is too high.

The more a test influences production decisions, the less acceptable it is to leave uncertainty in place.

The hidden cost of retries

Retries seem cheap because they only take a few lines of config, but they have real costs:

they lengthen CI time when the first attempt fails
they can mask genuine regressions for too long
they make failure rates harder to interpret
they can encourage teams to stop investigating flaky test triage
they can hide systemic problems in the app or environment

This is why I do not use retries as a substitute for diagnosis. I use them as a temporary control when the failure mode is understood and bounded.

If a suite is full of retries, I usually read that as a quality signal failure. The team has likely accepted noise as normal.

How I decide between a small fix and a rewrite

When I am on the fence, I compare the following:

Choose a small fix if

the test is otherwise well structured
the failure mode is localized
the fix has a narrow blast radius
the test reflects a valuable user flow
the code change is likely to survive future product changes

Choose a rewrite if

the test has already been patched multiple times
the current structure makes debugging expensive
the test asserts too many things at once
the locator strategy is fundamentally fragile
the same kind of failure keeps coming back in different forms

A good rule of thumb is this: if the fix leaves the same maintenance burden in place, it is probably not a real fix.

The CI angle: make the failure visible and useful

Whatever path I choose, I want CI to tell me something useful. That means preserving evidence and keeping the signal clean.

For example, in GitHub Actions I want failed test runs to upload artifacts, not just print a wall of logs that nobody reads:

- name: Run Playwright tests
  run: npx playwright test

name: Upload traces if: failure() uses: actions/upload-artifact@v4 with: name: playwright-traces path: playwright-report/

Artifacts matter because they shorten time to root cause. If you are going to tolerate a retry, you should still collect the data needed to remove it later.

The CI system should also show whether a failure was retried. If a team cannot distinguish first-pass failures from retried passes, then the pipeline is lying by omission.

A debugging checklist I use before I change anything

Before I decide on retry, isolation, or rewrite, I look at:

failure frequency, not just one instance
whether it fails in one browser or all browsers
whether it fails locally, in CI, or both
whether parallelism changes the behavior
whether the test depends on ordering
whether screenshots, traces, or video show a real defect or a test artifact
whether the app emitted an error, timeout, or empty state

If I do not have evidence, I do not have a decision. I have a guess.

That sounds obvious, but flaky tests are often managed by instinct and pressure instead of data. The result is churn: retries added because people are busy, isolation ignored because fixtures are annoying, rewrites postponed because the current test is “good enough”.

What I tell engineering managers

If you manage a team, the important question is not “why is this test flaky?” The more useful question is “what is the cheapest reliable way to stop paying this tax?”

A manager should care about:

time spent on flaky test triage
CI stability and developer trust
release gating reliability
ownership concentration, especially when only one person understands the suite
whether the test suite is getting harder to change over time

A flaky test is often a symptom of a deeper design issue. If the same class of failure keeps recurring, the organization may need better test data management, better app test hooks, or a stronger separation between fast feedback tests and slower end-to-end coverage.

My default decision order

If I had to compress this into one policy, it would be:

Retry only for known transient failures, and keep retries bounded.
Fix isolation when shared state or execution order is the cause.
Rewrite when the test structure is the main source of cost and instability.

That order works because it matches the shape of the problem. Retries address temporary noise. Isolation addresses hidden coupling. Rewrites address bad design.

The practical takeaway

If you are trying to figure out how to fix flaky tests, do not ask which option is most convenient in the moment. Ask which option best reduces future uncertainty at acceptable cost.

If the failure is transient and rare, a retry can buy time.
If the failure comes from shared state, test isolation is the real fix.
If the test is brittle by design, rewrite it before it keeps draining the team.

The best flaky test strategy is not the one that makes CI green fastest. It is the one that makes the pipeline trustworthy again, without creating a maintenance burden you will regret next month.