Flaky Tests: How to Fix the Suite You Stopped Trusting

I once worked on a suite where a red build stopped meaning anything. When it failed, someone would say “just run it again,” the second run would go green, and we would move on. Nobody was being lazy. The suite had failed for no reason so many times that rerunning until green was the only sensible thing left to do. But somewhere in there it stopped doing its job. A test is supposed to tell you the truth about your product. Ours had started telling us noise.

A flaky test is one that passes and fails on the same code, with nothing changed underneath it. The thing I learned from that suite is that a flaky one is worse than a small one. A small suite is honest about how little it checks, so you cover the rest yourself. A flaky suite goes red when nothing is wrong, so you train yourself to ignore red, and once “just run it again” is the habit you cannot trust green either.

So this is what I do with a flaky test now:

• Confirm it is really flaky. Run it on repeat with retries turned off.
• Read the trace. Find what actually went wrong, instead of guessing.
• Make one of three calls. Fix it, quarantine it with an owner and a deadline, or delete it.

Deleting a test sounds like losing coverage, but a test nobody trusts was not giving you coverage anyway.

What counts as a flaky test

People use “flaky” loosely, so I want to be exact about it. A flaky test gives you both a pass and a fail on the same code, with nothing changed underneath. It has stopped measuring your product and started measuring noise.

This is not a rare problem. DORA’s test automation capability guidance sets the bar plainly:

• When tests pass, you should trust the software is releasable.
• When tests fail, the failure should point at a real defect.

Flakiness breaks both halves. Almost every suite I have seen has some. What matters is what you do next, which is to treat a flake as a bug in the test and give it the same seriousness you would give a bug in the product.

Flakiness is the false alarm, the red that is not real. The opposite problem, a green build hiding a real failure, I cover separately.

How to fix flaky tests: confirm first, then trace

Do not diagnose from a single red run. Before anything else, reproduce the instability on purpose, because a test that failed once might be flaky, or it might have caught a real intermittent bug, and you need to know which one you are dealing with.

The simplest way is to run the test over and over against the unchanged code and watch what it does. In Playwright, turn retries off so nothing hides the behaviour, and repeat it:

npx playwright test tests/checkout.spec.ts --repeat-each=20 --retries=0

Two outcomes, and they point in opposite directions:

• Passes every single time (say twenty out of twenty). You might have caught a real intermittent defect. Chase it as a product bug.
• Passes some, fails some on the same commit. It is flaky, and now the test itself is what you investigate.

Next, read the actual failure instead of guessing at it. Playwright’s trace viewer records the whole timeline of the failing run: every action, the locator it used, a snapshot of the page at each step, the network calls, and the console. Set it to capture on retry and open it on the failure.

// playwright.config.ts
export default defineConfig({
  retries: 2,
  use: { trace: 'on-first-retry' },
})

Most of the time the trace points straight at the cause. You watch the test click something before it was ready, or fetch data that arrived late, or trip over state another test left behind. Once you can see what happened, you fix the right thing instead of guessing at it.

Flaky tests come from a short list of repeat offenders. The one I have had to debug most often is a missing or naive wait, a test that assumes the page is ready before it is. Here are the ones worth recognising on sight, and the fix for each.

Root cause	How it shows up	The fix
Hard waits and naive timing	A fixed sleep that is too short on a slow run	Web-first assertions that wait for the real condition
Brittle locators	Test breaks when markup or a CSS class changes	User-facing locators: role, label, then visible text
Shared state between tests	Passes alone, fails when run with others	Isolate each test: its own data, storage, and session
Test data collisions	Two tests fight over the same record	Give each test fresh, unique data (see below)
Uncontrolled third parties	A real external site or API wobbles	Mock what you do not own; only test what you control
Time, time zones, and order	Fails overnight, or in a different region	Pin the clock and time zone; never assume run order
Animations	Click lands mid-transition	Wait for the end state, not a timer

Data collisions are worth a note, because they are the cause I see misread as flakiness most often: two tests fighting over the same record, leftover state from a run that never cleaned up, a shared account someone else changed underneath you. When the suite goes red around data, that is where I look first, and the full playbook for fixing it (unique records per run, cleanup by run ID, isolation) lives in test data management for QA. Two of the other causes are worth showing in code, because between them they are most of the real-world flakiness I have chased.

The hard wait is the classic. A sleep is a guess about how long something takes, and the guess is wrong the moment the environment runs slower than usual.

// Flaky: a guess that the toast appears within 2 seconds
await page.click('#save')
await page.waitForTimeout(2000)
expect(await page.locator('.toast').isVisible()).toBe(true)

// Stable: wait for the actual condition, with no fixed timer
await page.getByRole('button', { name: 'Save' }).click()
await expect(page.getByText('Changes saved')).toBeVisible()

The brittle locator is the other one. A locator chained to the markup breaks every time a designer touches the layout, and that reads as flakiness even though the product is fine. Reaching for the element the way a user would is far steadier, and it is worth keeping a locator cheat sheet open while you write.

// Brittle: breaks the moment the DOM shifts
await page.locator('div.cart > div:nth-child(3) > button.primary').click()

// Stable: finds it by what it is, not where it sits
await page.getByRole('button', { name: 'Checkout' }).click()

Fix, quarantine, or delete: the decision flow

Once you know the cause, you have three honest choices, not two. Fixing is the default. But a flake you cannot fix today still cannot sit in the set of tests that block a merge, because a test that randomly blocks merges is exactly what teaches a team to rerun until green in the first place.

Fix it when you can find the cause and the test is worth keeping. This is where most flakes should end up, and the table above is your starting kit.

Quarantine it when it is genuinely flaky, you cannot fix it this minute, and the behaviour it covers still matters. Quarantine means you pull it out of the merge-gating set so it stops blocking the team, but you keep running it and you keep it visible. The one thing that keeps quarantine from turning into a graveyard is that every quarantined test gets an owner and a deadline. A quarantined test with no owner is not quarantined, it is abandoned.

QUARANTINED TEST LOG
Test:        checkout.spec.ts > applies coupon after tax
Quarantined: 2026-06-10
Owner:       you
Root cause:  suspected race between coupon API and total render
Fix by:      2026-06-24  (one sprint)

Delete it is the third choice, and it gets its own section, because it is the one teams get wrong most often and the one that pays off the most.

When to delete a flaky test

Delete a flaky test when it is low value, when it duplicates coverage you already have, or when it has sat past its deadline with nobody willing to own it.

Teams resist this harder than anything else here, because deleting a test feels like losing coverage. But coverage is not the number of tests in the repository. It is how much of the product the team actually learns about when the build finishes, and a test only counts toward that if people act on its result. A test everyone reruns past is not adding anything. It is costing you, because every unexplained red it throws teaches the team to rerun past every red, including the one that was real.

That is why deleting an untrusted test can give you more real coverage, not less. Take out the handful of tests everyone reruns past, and the reds that are left start getting looked at again. Not every test deserves to be automated, and not every automated test deserves to stay.

A smaller suite that people believe is worth far more than a large one they have learned to ignore.

Deleting is still a testing decision, not a cleanup chore, so I do it deliberately:

• Check what the test proves. What does this actually prove? If the honest answer is “the same thing three other tests already prove,” the coverage was never at risk.
• Check the record. A test that has sat past its deadline with no owner has already been deleted in practice. Removing the file just makes the suite honest about it.
• Write one line about why. A short note in the pull request (“duplicated by X” or “chronically flaky, behaviour covered by Y”) means nobody re-adds it in six months without knowing its history.

Then close the loop so fixed flakes cannot creep back in quietly. Playwright has a --fail-on-flaky-tests flag that fails the build if any test is flagged flaky, which turns retries from something that hides the problem into something you can see. Retries are fine as a stabiliser while you investigate. They are not a cure, and a suite that needs three retries to stay green is telling you something worth listening to.

Treat the suite like a product

Flakiness is usually a sign of test code that nobody maintains the way they maintain production code. The suite is a product, and it has users: the developers who depend on its signal. If you would not ship the application with shared mutable global state and hard-coded sleeps, do not ship your tests that way either.

A few habits keep a suite trustworthy over years, not just weeks.

• Isolate every test. Each one should run on its own, with its own data, storage, and session, in any order. Tests that depend on order break in ways that take days to debug.
• Prefer user-facing locators. Role, label, then visible text. They survive redesigns that shatter CSS-chained selectors.
• Only test what you control. Mock the third-party site or API you do not own, so an outage on their side does not turn your build red.
• Cut duplication. Centralise selectors and flows so a UI change is a one-line update, not a fifty-file edit.
• Delete dead weight. A test that has not failed meaningfully in a year, or that overlaps another, is cost without coverage. What you want is a suite small enough, stable enough, and meaningful enough that when it fails, people pay attention. That same discipline is what separates automation that pays off from automation that just accumulates.

Habits only hold when someone owns them and there is a clear finish line. The minimum set worth agreeing on:

• Owner: QA. Confirm and triage every reported flake. Done when: the test has been repeated with retries off, the cause is in the quarantine log, and it has an owner and a deadline.
• Owner: Dev. Fix or delete each quarantined test by its date. Done when: nothing sits past its deadline with nobody willing to own it.
• Owner: EM. Gate on flakiness so fixed flakes cannot creep back. Done when: the build fails on a test flagged flaky rather than passing quietly on a retry.

This matters more now that AI writes so many tests, not less. An agent will happily generate a hundred plausible assertions, and a good number of them will be brittle in exactly the ways above. AI is genuinely useful on the diagnosis side: hand it a trace or a pile of CI logs and it is fast at spotting the timing pattern or the shared-state smell. What it should not do on its own is decide which tests are worth keeping. That judgment, what actually protects the product and what is just noise, is yours, and it is the heart of reviewing AI-generated tests rather than trusting them.

Where to start

Pick your most-rerun test, the one everyone groans about. Run it twenty times with retries off and confirm it is flaky. Open the trace, find the cause, and either fix it, quarantine it with an owner and a deadline, or delete it and write the one-line reason why. Then do the next one. You do not fix a flaky suite in a sprint, you fix it one honest test at a time. What you get back is the thing automation was supposed to give you in the first place: a red build that means something again, and a green one you can actually act on.