New Testing Pitfalls to Avoid in the Age of AI

Make “done” mean evidence, not a green suite

Start here, because it is the guardrail that unlocks all the others. When an agent generates a hundred tests and they all pass, it is tempting to read green as covered. A passing test only earns that read if it would fail when the feature breaks.

Left alone, generation produces a lot of assertions that confirm an element is visible, or that a response came back with a 200 status, without ever checking that the right thing happened. Those tests stay green through almost any bug you can imagine, so the bug sails through to the user while your pipeline reports success.

The fix is not to read every line the agent wrote. It is to change what “done” means before the agent starts.

Done is not “it passed.” Done is “I broke the feature on purpose, watched the test go red for the right reason, then watched it go green when I fixed it, and the evidence is on the PR.”

Put that in the definition of done and the agent has to satisfy it to call anything finished. Now green actually means something, and you can trust it instead of re-auditing it.

Here is the shape the assertion problem takes. Compare these two tests for the same checkout flow:

import { test, expect } from '@playwright/test'

// Decorative: passes even if the total is wrong
test('verify that the cart page loads after adding an item', async ({ page }) => {
  await page.goto('/cart')
  await expect(page.getByTestId('cart')).toBeVisible()
})

// Meaningful: fails the moment the total is wrong
test('verify that the cart total reflects two items at the listed price', async ({ page }) => {
  await page.goto('/product/widget')
  await page.getByRole('button', { name: 'Add to cart' }).click()
  await page.getByRole('button', { name: 'Add to cart' }).click()
  await page.goto('/cart')
  await expect(page.getByTestId('cart-total')).toHaveText('$59.98')
})

The first one will happily pass while the price calculation is broken, the wrong currency is shown, or the second item silently failed to add. The second one is tied to the actual behaviour a customer cares about, so it goes red when that behaviour goes wrong.

Without a rule to follow, generation reaches for the first kind, because checking that something rendered is the easiest assertion to write and the safest one to keep green.

So make the rule. The first few times you correct an agent, you will tell it the same thing: assert on behaviour, not presence; a test has to fail when the feature breaks.

Do not keep saying it. Write it into a CLAUDE.md or AGENTS.md at the repo root, with a short example of a passing assertion and a decorative one, and the agent reads it before it writes the next test. The question you used to ask on every review, “would this go red if the behaviour were wrong,” becomes a rule the agent applies by default.

A smaller suite of tests that actually fail for the right reason is worth far more than a large one nobody trusts.

Let fluent output earn trust through CI, not through how it reads

The agent’s output reads beautifully, and that is the one thing you cannot let stand in for proof. A Stanford-led study, Do Users Write More Insecure Code with AI Assistants?, found that developers given an AI assistant wrote less secure code than those without one, and, more troubling, were more confident that their code was secure.

Well-formatted, confident-looking output lowers your guard, so you examine it less closely than you would the same work from a colleague. Polish is not proof, and a green build is very good at hiding the gap between the two.

This is not a one-study fluke either. Veracode’s 2025 GenAI Code Security Report tested more than 100 large language models across 80 coding tasks and found that 45% of the generated code introduced a known security weakness, with cross-site scripting and log injection failing the most often.

Close to half of what these tools hand you carries a real flaw, and it arrives looking just as polished as the safe half.

Reading every diff more suspiciously by hand does not scale, and it puts the catch in your attention instead of in the system. So make fluency prove nothing.

Wire the checks the polish hides into the pipeline so they run on every PR the agent opens:

• A security scanner, dependency checks, and the linters your team already trusts, running on every PR the agent opens.
• GitHub Copilot reviewing the PR can flag a lot of this before you look.
• The recurring problems you do catch by eye, the cross-site scripting pattern, the unsanitised input, go into the rules file so the agent stops producing them in the first place.

Your eyes move from “is this line safe” to “does the evidence on this PR prove it.” That is the judgment only you bring, and it is where your attention is worth the most.

Encode the review once instead of running it every time

An agent produces a test in seconds, and the work that protects you is making sure the agent’s test holds up the way a person’s would.

The failures show up in the same predictable places: hard-coded data that breaks the moment the database is reseeded, no teardown so they pollute the next test’s state, and a hidden dependence on run order that only works on the machine where they were generated. None of that surfaces until the suite runs in a different order, on a fresh environment, on a Friday afternoon.

The instinct is to hand-review every generated test forever, the way you would a colleague’s pull request. Do it the first few times, absolutely, because that is how you learn what this agent gets wrong on your suite.

Then stop repeating the catch and encode it. The checklist below belongs in the rules file, so the agent meets it before it opens the PR, plus a CI step that re-runs the suite in a randomised order on a clean environment so an ordering dependency fails loudly instead of hiding. Before any generated test is done, it should:

• isolate its own data, so it does not depend on whatever happens to be in the database
• tear down what it creates, so it leaves no mess for the next test
• assert on behaviour rather than presence, so it actually checks the thing that matters
• fail for the right reason, not because of run order or a hard-coded value

Encode those four and your review stops being a re-derivation of the same checklist every run. I have a full walkthrough of how to review AI-generated automated tests you can lift the rules from directly.

Point the agent’s speed at pruning, not just adding

The same speed that lets an agent write tests by the dozen can be turned on the suite itself. The trap is using it only to add. A team that used to add tests deliberately starts generating them in bulk, and within a few months the suite takes far too long to run and nobody can say which tests still matter.

This is not just a feeling I have. GitClear’s AI Copilot Code Quality 2025 research, which analysed more than 200 million changed lines, found that duplicated and copy-pasted code blocks rose sharply once AI assistants became common, while refactoring fell, and the share of code revised or reverted within two weeks of being committed climbed year over year.

Test code is not exempt. Cheap creation plus no refactoring is exactly how you end up with thousands of near-identical tests.

I have worked on suites that got there the slow way, long before agents, and the end state is always the same: slow pipelines and a layer of flaky tests everyone has quietly learned to ignore. That is its own danger, because if the team ignores the failure, the test has already lost.

So treat volume as a cost in the rules, and hand the cleanup to the same agent. It is good at exactly this work. Point it at the suite and ask it to find duplicated and overlapping tests, surface the ones that have never failed, and propose a consolidation as a PR with the evidence.

You make the call on what gets deleted; the agent does the combing. A suite is something you maintain, not something you accumulate, and now you have a tireless partner for the maintenance.

Test the AI inside your own product, not just the AI that helps you test

It is easy to forget that your product now likely contains AI features that need testing themselves, while you spend all your attention on the agent that tests your product. Those features behave differently from everything else you ship:

• They are non-deterministic, so the same input can return different output on two runs.
• They can be pushed into misbehaving by a cleverly worded request.
• They fail in ways your normal assertions will never catch, because there is rarely a single correct string to assert against.

If your product has a chat feature, a summariser, or a recommendation engine, that surface needs its own deliberate strategy, not a toEqual against one expected sentence. In practice that means a few things working together:

• an evaluation set of known-good inputs and acceptable outputs
• criteria-based scoring instead of exact string matching
• runtime guardrails that catch bad output before it reaches the user
• a deliberate batch of adversarial and prompt-injection cases

I go deep on the mechanics in how to test AI features with non-deterministic output, so I will not repeat all of it here.

The point for this list is simpler: skipping it is not a low-stakes gap. Depending on the product, an untested AI feature can hand someone wrong information, treat people unfairly, or fail a user at the exact moment they needed it to work.

The one mechanism behind most of these

Most of these traps share one mechanism, and seeing it laid out makes them easy to handle. A normal bug surfaces as a red signal somewhere you are already looking. The AI-era pitfalls do the opposite: they produce a green signal you trust, while the real risk passes underneath it.

So the fix is the same shape every time. You make green hard to earn, by encoding the checks into the rules file and CI, so the agent has to clear them before it can hand you a green build.

The teal box on the right is where you stay, and it is a smaller, sharper job than reading every line. You set the checks up once, in the rules file and CI, and from then on the agent clears them before it can show you green.

None of these pitfalls is solved by a better tool. They are solved by a person who decides what green has to prove, encodes it, and then judges the evidence instead of re-deriving it.

The full case for that rules-file-and-evidence loop, and how it sits at the centre of an AI-assisted team, is its own piece: QA as the control layer for AI-assisted development.

Keep the judgment, and the skill behind it, human

Everything above depends on one human ability: reading the agent’s work critically enough to know what green has to prove and when a test is hollow. It is the core of what a tester brings to an AI-assisted team, and it only comes from having done the work by hand, so protect it deliberately.

Have people write tests from scratch often enough that they can feel when generated output is off. A team that lets juniors lean on the agent entirely keeps its short-term output looking fine and quietly loses the ability to catch the tool when it is wrong, and the tool will be wrong.

That same judgment is what keeps the decisions where they belong. The more the agent can do, the more tempting it is to let it decide as well as do:

• it says coverage is good, so we believe it
• it flags these areas as high risk, so we ignore the corner of the system the team is quietly nervous about
• it grades the AI feature’s output as fine, so we ship

Each of those is a decision, not a task. The agent works from estimates; it does not know your users, your release pressure, or the part of the codebase that has burned you twice.

It is also weak exactly where the call is hardest: it cannot see that a UI is visually wrong, the button slightly too big, the spacing off, the screen that renders but looks broken. I hit this building my own game. The agent could not tell me a screen was misaligned or an element was the wrong size, and on real devices across a pile of screen sizes, that is most of what matters.

So keep the visual and UX call, the exploratory testing, the read on real risk, and the decision to ship. Handing those to the agent is the one pitfall that gives every other one a clear runway.

A quick gut-check for your team

If you want a fast way to tell whether your guardrails are doing the work, ask your team three questions and listen for hesitation:

• What does our definition of done make the agent prove before a test counts as finished?
• Which of our corrections live in the rules file, and which are still in someone’s head?
• What is our actual, written-down plan for testing the AI inside our own product?

Silence on any of those is a finding, and it is your next piece of work. None of it takes long to stand up: a definition of done that means evidence, a rules file the agent reads first, the recurring checks moved into CI, and a real evaluation strategy for any AI feature you own.

If you do not have a written quality strategy to hang these on yet, the strategy builder is a good place to start one.

None of this is an argument against agents. Used like this, one takes a real load off your desk, and the guardrails are what let you trust it with that load instead of babysitting it.

The work shifts from re-reading every line to deciding what green has to prove, encoding it once, and judging the evidence the agent brings back. You go faster precisely because the caution is built into the system. Asking for the evidence stops being something you repeat in review and becomes the thing the agent has to clear before it calls anything done.