Yesterday the model generated inputs for you; today it generates whole tests, and the review you give them decides whether your suite gets stronger or just bigger.
An agent will happily write a hundred tests in a minute. Volume is not coverage, and a suite full of weak assertions is worse than no suite at all, because it manufactures confidence you have not earned. A test that turns green while the user experience is broken is not evidence. It is a liability with a passing badge.
Start at the fence
The Never-generate list from Day 4 is not just generation guidance. It is my review checklist. When a generated spec comes back, I scan for the eight fence items first: a fixed sleep, a raw framework command in a spec, an inline selector, a brittle locator, hardcoded data, a test that depends on another test, an exact count against live data, an empty test ID. Any hit goes back. That scan takes two minutes and catches most of what matters. Then comes the harder question: does this test prove anything?
The test that lies
Here is the most common failure I see, and an agent will hand you one this week:
test('checkout works', async ({ page }) => {
await page.goto('https://www.saucedemo.com/')
await page.getByPlaceholder('Username').fill('standard_user')
await page.getByPlaceholder('Password').fill('secret_sauce')
await page.getByRole('button', { name: 'Login' }).click()
await expect(page).toHaveURL(/inventory/)
})
That test is named “checkout works” and never reaches checkout. It logs in and confirms the URL changed. It stays green through a broken cart, a failed payment, or an order that never gets created.
Here is the same intent, written so it can actually fail for a real problem:
test('verify that a logged-in user can complete a checkout',
async ({ page }) => {
await page.goto('https://www.saucedemo.com/')
await page.getByPlaceholder('Username').fill('standard_user')
await page.getByPlaceholder('Password').fill('secret_sauce')
await page.getByRole('button', { name: 'Login' }).click()
await page.getByRole('button', { name: 'Add to cart' })
.first().click()
await page.locator('[data-test="shopping-cart-link"]').click()
await page.getByRole('button', { name: 'Checkout' }).click()
await page.getByPlaceholder('First Name').fill('Julia')
await page.getByPlaceholder('Last Name').fill('Pottinger')
await page.getByPlaceholder('Zip/Postal Code').fill('00000')
await page.getByRole('button', { name: 'Continue' }).click()
await page.getByRole('button', { name: 'Finish' }).click()
await expect(page.getByText('Thank you for your order!'))
.toBeVisible()
})
This one walks the actual flow and asserts the outcome a customer cares about. If the cart, the form, or the order confirmation breaks, the test goes red and tells you why. That is the bar, and it is the keep-or-kill test for every generated spec: a test stays when it would fail for a genuine product bug and pass the rest of the time. Everything else is noise wearing a green checkmark.
Send it back with specifics
When a test fails review, do not fix it by hand and do not say “make it better”. Specific corrections get a good rewrite in one round; vague ones get a different flavour of the same problem. This is the shape I use:
Revise tests/search.spec.js with these specific changes:
1. Replace the XPath locators on lines 12 and 18 with
role-based locators, and move them into the page object.
2. Remove the waitForTimeout on line 14. The search fires
GET /api/products. Wait on that response instead.
3. Rename "test_2" to describe the behaviour it proves,
as a readable sentence.
4. The assertion on line 22 only checks the page loaded.
Assert the first result contains the search term.
Do not change any other file. Show me the diff before
committing anything.
Copy the shape: line numbers, the exact change, the reason, and a hard stopping point. And every correction you find yourself giving twice belongs in the Day 4 rules file, so you stop giving it at all.
The almost-right bug an agent review should catch
Not every bug turns a test red. Some sit in a single value that looks reasonable until you hold it against the source of truth, and almost right is the dangerous kind of wrong, because it survives a quick glance and ships. Tumble Kitchen now runs on a real REST API at https://juliapottinger.com/api/tk/ with the repo right beside it, so an agent with that access can review the screen against the data behind it. Two planted bugs are exactly this shape:
- The dish page disagrees with the price. The Fried Chicken Plate shows 1,650 on its detail page, while the menu and the API both return 1,600. The menu is right, the dish page is wrong, and nothing goes red.
- The delivery total drops the fee. Send a delivery order to /api/tk/cart/validate and the total comes back 500 short, because it leaves the delivery fee out of the sum it returns. The line items look fine. The number a customer would actually pay does not.
Point the agent at the running page and the API and make it compare, rather than assume the screen is right. This is the exact prompt I use:
Open the Fried Chicken Plate dish page at
/practice/tumble-kitchen/dish/fried-chicken/ and read the
price shown to the customer. Then call the API at
https://juliapottinger.com/api/tk/products/fried-chicken and
read its price field. Compare the two, state both values,
and tell me plainly whether they match. If they disagree,
treat the API as the source of truth and report it as a
data-integrity bug with both numbers.
Copy it, swap in any dish. What comes back is a verdict you can act on: the two numbers, and whether they agree. When they do not, the agent has surfaced a bug no passing test would have shown you, and the call on what to do about it stays yours.
Never merge a generated test you did not run and read. A generated test nobody ran is a guess in a green wrapper.
Try this today
Take one AI-generated test, yours or from any open repo. Run the fence scan, then ask the harder question: what real product bug would make this fail? If you cannot name one, write the send-back with line numbers, even if you have nowhere to send it. Writing the correction precisely is the skill.
Tomorrow: the layer above a single test. The five questions I ask before any AI-assisted work ships at all.