How to Review AI-Generated Automated Tests

Start with the purpose, not the syntax

Before you read a single assertion, ask what the test is trying to prove. A good test has a job. It protects a behaviour the team cares about, and you should be able to say in one sentence what failure it would catch and why that matters to a user or the business.

Say the agent hands you a test that checks a payment modal opens when you click “Update card”. That is a real thing the app does, so the test passes and looks legitimate. But opening the modal is not what the feature is for. What the user actually needs is to update their payment details, see a confirmation, and keep their subscription active. The modal is step one of five, and a test that stops there stays green through a broken save, a swallowed error, and a subscription that silently lapses.

The habit to build is tracing every test back to the thing the feature is supposed to do for the user. When the purpose is thin, the test is thin no matter how clean the code looks, and no amount of polishing the syntax fixes a test aimed at the wrong target.

Check that the assertions reach a real outcome

Weak assertions are the most common problem I see in generated drafts, and they are easy to miss precisely because the test still passes.

Watch for assertions that confirm the mechanics of an action without confirming its result:

• checking that a button still exists after you click it
• checking that the URL changed without checking what loaded
• checking that a success toast appeared without checking that anything was saved
• checking a single visible label in a workflow that touches five systems
• checking the mock data you handed in, which proves only that your mock works

Strong assertions connect to outcomes a user or the next system would actually notice:

• the record was created with the expected fields
• the user’s role changed after the permission update, and they can now reach the protected page
• the entitlement exists after the purchase completes
• the error message appears and the user can recover from it
• the same state survives a refresh or a restart

The question I hold every assertion to is simple: does this test verify the outcome that actually matters, or only the first thing that was easy to check? A test should make the product safer, not make a coverage number look better. And to know whether an assertion is real or decorative, you do not have to guess: break the behaviour and see if the test notices.

Watch for over-mocking that hollows the test out

Mocking is not the enemy. A unit test should mock the network so it stays fast and deterministic. The problem is that mocking is the agent’s path of least resistance, so left alone it mocks everything, and once you mock everything there is nothing left to test.

Picture a checkout test that mocks the application programming interface (API) response, the payment provider, the database write, and the state store. What is actually exercised?

You are verifying that the component renders whatever you told it to render. The real integration, the part that breaks in production, was mocked away before the test ran.

I have caught exactly this on real work. On a promotions service I tested, a generated test for the offer-evaluation endpoint stubbed out the part that decides whether each item is even eligible for the promotion. It set every line to eligible and then asserted the discount applied. It was green and it proved nothing, because the real bug we worried about lived in that eligibility step: an item that should not have counted toward the tier was counting anyway, which threw off the whole discount. By faking that dependency away, the test had mocked out the one piece of logic worth testing. The fix was to let the real eligibility logic run and stub only the genuinely unstable outside dependency, which is the whole point of a component test: keep the core real, mock the edges.

A smaller front-end version of the same mistake is a data-table test that stubs the API and feeds the table a canned, non-empty response, so it passes every time. But on real environments that table can legitimately come back empty, and the empty path is often where the screen breaks: blank axes and zero-height bars instead of a proper “no data” state. The mock quietly asserted away the one condition real users were hitting. The fix is to stop assuming a populated response: intercept the real call, wait on it, and branch on whether data actually exists, instead of retrying an assertion that only ever passes against the stub.

When you see a stack of mocks, work through these questions:

• What level is this test meant to be: unit, integration, API, or end to end?
• Which dependency genuinely needs to be faked to keep the test fast and stable?
• Which dependency has to stay real for the test to prove anything?
• Could this test stay green if the actual integration were completely broken?

That last question is the sharp one. If a test would pass while the feature is broken, the mocks have hollowed it out, and you either make a dependency real or move the check to a level where it can hit something real.

Inspect the locators before they rot

AI-generated user interface tests tend to grab whatever selector is most visible in the markup: often a generated class name, a deep CSS path, or text that marketing will change next sprint. Those tests pass today and break the first time someone touches the design, and a suite that breaks on every cosmetic change is one people start ignoring.

Prefer locators that follow how a person, or a screen reader, actually finds things; the closer a locator tracks how the software is genuinely used, the less it breaks on a redesign. Here is the split I hold a generated test to when I review its selectors.

Better: holds up over time	Risky: rots on the next change
Accessible role, such as getByRole('button', { name: 'Save' })	Auto-generated class names, such as .css-1q2w3e
Visible label tied to its control, such as getByLabel('Card number')	Positional selectors, such as :nth-child(3)
Intentional test identifier, such as data-test="checkout-submit"	Brittle XPath, such as //div[2]/div/span/button
Accessible text the user reads	Deep CSS hierarchy, such as div > div > button
User-visible behaviour, such as a heading or a confirmation message	Marketing copy that changes next sprint

The left column survives a redesign because it tracks what the product means. The right column tracks how the markup happened to be arranged the day the agent wrote the test, and a locator like div > div:nth-child(3) > button breaks the first time someone reorders that layout.

For a fuller side-by-side of which selectors hold up and which rot, my locator cheat sheet is the reference I point people to.

When an agent reaches for a brittle selector, it is usually because it cannot see how the component is built, so it grabs whatever sits in the rendered markup. Give it the frontend and backend code and it finds the stable hook instead of guessing. Connecting those sources is the first thing I set up in give your AI real context for QA, and it does more for locator quality than any amount of reviewing after the fact.

Often the deeper fix is not in the test at all. It is a missing testability hook in the application. Adding a stable data-test attribute to the product code pays off across every test that touches that element, and it is exactly the kind of thing a generated test will never think to ask for.

Review against your standards, not the agent’s defaults

An agent writes whatever the codebase nudges it toward, plus whatever it picked up from a million public repositories. Left alone it picks its own conventions, and you end up reviewing the same arguments over and over:

• raw framework calls scattered through the spec
• selectors and text inlined instead of kept in one place
• a different waiting strategy in every file

You can shut most of that down before the agent writes a line by giving it a standards file at the repo root, a CLAUDE.md or AGENTS.md that spells out how your team writes tests. I keep a real, ready-to-paste one in my AI test automation standards, the same Cypress file I hand an agent.

Then your review checks the diff against rules you already agreed on, instead of relitigating style every time.

Two conventions are worth holding the line on, because they decide how the suite ages:

• Interactions go through a driver, not raw commands. A spec should read like a user action, selectGroupAndWaitForData() or openFirstRowDrawer(), not a primitive like page.click('.btn-primary'). When the next change touches that flow, you fix one driver instead of forty specs. If a generated spec calls cy.get or page.click directly and no driver exists, add the driver, do not wave the raw command through.
• Selectors and visible text live in page objects, not inline. One place to update when the UI shifts. A test littered with inline selectors breaks in twelve files the day a label changes.

When you hand an agent that standards file, also write down what it must never produce. The review checklist further down doubles as that list, and a single violation in a diff is enough to send the test back.

The fastest tell that an agent ignored your standards is a raw command or a fixed sleep in a spec. When you spot one, do not assume the rest of the file is fine. It usually means the agent was working from its own habits, not yours, and the whole batch deserves a closer read.

Insist on the negative paths

An agent left to its own devices writes the happy path and stops. Your users live everywhere else. For each generated workflow, ask what the test does about what goes wrong:

• the network call fails or times out
• the data the screen expects is missing or empty
• the user does not have permission
• the same action is submitted twice
• the user refreshes in the middle of the flow
• the screen is a narrow mobile viewport
• the response is slow enough to expose a race

You do not have to automate every one of these on day one. You do have to decide, on purpose, which risks are worth covering rather than letting the agent’s silence decide for you. The question I put to the workflow is the one I put to the team before any release: what would make us uncomfortable shipping this? Whatever the answer is, that is the path the test should cover, and it is almost never the one the agent reached for first. A suite that only proves the feature works when everything cooperates is not protecting the feature at all.

Make the test fail on purpose

This sounds obvious, and people skip it constantly. A passing test tells you the test ran. It does not tell you the test is watching anything. The only way to know is to break the thing it covers on purpose and watch it go red. So before you trust a generated test, sabotage the feature under it. Pick the cheapest break that should be caught:

• comment out the line that saves the record
• return the wrong status code from the endpoint it relies on
• hand it data that should be rejected
• flip a boolean in the logic it claims to protect

Run it, and watch what happens. You are checking two things, and they are equally important:

1. That the test fails at all. A surprising number of generated tests stay green through a broken feature, because they never asserted the outcome. If your sabotage does not turn the test red, the test was decoration.
2. That it fails for the right reason. A good failure names the break. A bad one dies three steps later on a confusing timeout and sends whoever is on call at 2am hunting in the wrong place.

With a batch rather than one test, Stryker automates this same idea: it makes small changes to your production code, flipping a > to a >=, deleting a line, swapping a boolean, then runs your suite. Any test that stays green while the code is broken just told you it was never watching that behaviour. Pointing a mutation run at a fresh batch of generated tests is the fastest way to find the ones that assert nothing of consequence.

This is the habit that lets you ship with evidence instead of crossed fingers. When someone tells me a feature is covered, the first thing I want to know is whether they have ever watched the test fail. A green suite you have never seen go red is not proof the feature works. It is proof the test ran, and those are not the same thing.

Treat the test data as part of the design

Generated tests love to invent data that looks fine and quietly causes trouble:

• email addresses that collide when two runs overlap
• dates that pass today and expire next month
• hardcoded identifiers that only exist in one environment
• realistic-looking values pointed at a shared or production-like database
• cleanup steps that delete records other tests depend on

Test data is test design, not an afterthought. For anything an agent generated, make sure:

• each test creates the data it needs
• the data is unique enough to survive parallel runs
• nothing it touches can damage state another test or person relies on

This is the same independence and data-ownership principle good human-written suites rely on, and generated tests need it even more because the agent has no idea what else is running.

Keep the shape of the suite healthy

An agent can produce a hundred tests in the time it takes you to read ten, and left unchecked it will make all hundred slow browser tests, because a browser flow is the easiest thing to describe in plain English. That is how you end up with an upside-down suite that takes an hour to run and flakes constantly.

The test automation pyramid still holds: a wide base of fast unit tests, a solid middle of service and integration tests, and a small, carefully chosen set of end-to-end tests on top.

Generated tests pull against this, so part of your review is asking where each behaviour belongs and pushing it down to the lowest level that still proves the risk. Validation logic does not need a browser. A contract between two services does not need a full checkout flow. Reserve the slow, expensive top layer for the handful of journeys that justify it.

The review checklist

Everything above collapses into one list. It follows the five gates in the diagram, and it is the version I paste into the pull request when a batch of generated tests lands:

• Purpose. The test states, in one line, what behaviour it protects and what failure it would catch.
• Purpose. It sits at the lowest level of the pyramid that still proves the risk.
• Assertion. It checks a real outcome, not just that an action fired.
• Assertion. Test data is unique, self-created, and safe for the target environment.
• Assertion. The test stands alone and passes in any order, with no dependency on another test’s state, and never asserts an exact count against live data.
• Mocking. Mocks isolate noise without faking away the thing under test.
• Mocking. At least one real dependency stays real, or the test lives at a level where it can.
• Locators and standards. Locators use roles, labels, or intentional test identifiers, never layout paths or generated class names.
• Locators and standards. Interactions go through a driver, and selectors and text live in page objects, not inline in the spec.
• Locators and standards. There is no fixed-time sleep anywhere; waits are on a real signal. The test carries a non-empty, stable ID.
• Fails on demand. Breaking the feature on purpose makes the test fail with a clear message.
• Fails on demand. The important negative paths were considered, even if not all are automated yet.
• Fails on demand. A mutation run, or a deliberate break, confirms the batch actually catches regressions.

Some of those gates need your judgement. Others are mechanical, and mechanical checks should be mechanical; you do not want a tired reviewer at 5pm to be the only thing between a hollow test and the main branch. Wire the pass-or-fail ones into continuous integration (CI) so a generated test cannot merge until it clears them:

• a lint rule that blocks fixed sleeps and raw framework calls in specs
• a check that rejects inline selectors, nth-child chains, and missing test IDs
• the suite running with retries set to zero, so a flaky test cannot hide behind a rerun
• a mutation run on changed tests, plus the coverage delta on changed files, flagged for a person rather than blocking, because a low score or a flat delta is a smell, not always a defect

Once CI rejects a fixed sleep on its own, you never leave that comment again, and you spend your review on the only question that needs you: would this test catch the bug we are actually afraid of.

The review, applied to one test

Here is the same review in practice. The agent’s first draft passes, and it is nearly worthless:

import { test, expect } from '@playwright/test'

// generated draft: green, but proves almost nothing
test('update payment', async ({ page }) => {
  await page.route('**/api/payment', (route) =>
    route.fulfill({ json: { ok: true } })
  )
  await page.goto('/account/billing')
  await page.click('.btn-primary')
  await expect(page.locator('.modal')).toBeVisible()
})

It mocks the only call that matters, asserts a styled element is visible, and leans on a class name a designer will rename. Now the reviewed version, aimed at what the user actually needs:

import { test, expect } from '@playwright/test'

test('verify that a user can update their card and keep the subscription active', async ({ page }) => {
  await page.goto('/account/billing')
  await page.getByRole('button', { name: 'Update card' }).click()

  await page.getByLabel('Card number').fill('4242424242424242')
  await page.getByLabel('Expiry').fill('12/30')
  await page.getByRole('button', { name: 'Save card' }).click()

  // outcome the user actually cares about
  await expect(page.getByText('Card updated')).toBeVisible()
  await expect(page.getByTestId('subscription-status')).toHaveText('Active')

  // and it survives a reload, so we know it was really saved
  await page.reload()
  await expect(page.getByText('•••• 4242')).toBeVisible()
})

Same feature, same tool, completely different value. The second test names the behaviour it protects, drives the real flow against a real billing endpoint, checks an outcome the user would notice, and confirms the change persisted. I kept the steps inline so you can read the two side by side, but on a real suite I would move that flow behind a driver and put the selectors in a page object, so the next change to the billing screen touches one file instead of every test that visits it.

That gap between the two drafts is the entire point of the review, and the same gap I kept running into when I had an AI agent write my Playwright and Cypress tests end to end in Playwright and Cypress.

Where AI fits, and where you do

None of this means the agent is a bad idea. An agent that drafts the structure, the boilerplate, and the obvious happy path is genuinely useful. The shift is where your effort goes: less time typing tests, more on the work that actually protects the product:

• deciding which generated tests deserve to live
• sharpening their assertions until they reach a real outcome
• adding the failure cases the agent skipped
• pushing checks down to the right level of the pyramid

If you want to go deeper on running that kind of review as a standing practice across your pipeline, I have written more about it in the QA control layer for AI-assisted development.

The agent can write a hundred tests before lunch. Deciding which ten are worth keeping, and making those ten genuinely catch regressions, is still your craft. When an agent drafts the tests, reviewing them is the real work now, not overhead on top of it.

import { test, expect } from '@playwright/test'

// generated draft: green, but proves almost nothing
test('update payment', async ({ page }) => {
  await page.route('**/api/payment', (route) =>
    route.fulfill({ json: { ok: true } })
  )
  await page.goto('/account/billing')
  await page.click('.btn-primary')
  await expect(page.locator('.modal')).toBeVisible()
})

import { test, expect } from '@playwright/test'

test('verify that a user can update their card and keep the subscription active', async ({ page }) => {
  await page.goto('/account/billing')
  await page.getByRole('button', { name: 'Update card' }).click()

  await page.getByLabel('Card number').fill('4242424242424242')
  await page.getByLabel('Expiry').fill('12/30')
  await page.getByRole('button', { name: 'Save card' }).click()

  // outcome the user actually cares about
  await expect(page.getByText('Card updated')).toBeVisible()
  await expect(page.getByTestId('subscription-status')).toHaveText('Active')

  // and it survives a reload, so we know it was really saved
  await page.reload()
  await expect(page.getByText('•••• 4242')).toBeVisible()
})