Classic Testing Mistakes AI Brought Back at Machine Speed

Teams keep telling me their testing has never been faster. The agent writes tests in seconds, coverage is climbing, the dashboard is a wall of green. I believe them. Then I ask what any of it actually proves, and the room usually goes quiet. They have more tests than ever and less confidence than they want to admit.

I have watched that gap between motion and confidence my whole career, long before any of us had an agent writing assertions. None of these failures are new. Testers have been naming them for decades, and one of the clearest write-ups is still Brian Marick’s from 1997. It reads like it was written about a team using AI last week. AI did not invent a single new mistake. It took the old ones and made them cheap, fast, and convincing, which is worse, because now you can make them at scale before lunch.

This is not an anti-AI piece. I use AI every day and it genuinely helps. It is a recognition exercise. Once you can see the old mistake under the new speed, you already know the fix, because you have known it all along. Here are the ones I keep running into.

The suite got huge overnight, and nobody can say why

Ask an agent for tests and it will happily hand you a hundred. That feels like progress, and sometimes it is. But cheap to create has never been the same as cheap to own. Every one of those tests still has to be read, trusted, and maintained when the product changes, and a pile of shallow, overlapping checks is the same liability it always was. The only thing that changed is how fast you can build the liability.

Not every test deserves to be automated. That was true when a person wrote them one at a time, and it is true when a model writes them in bulk. Automate what gives the team repeatable confidence in behaviour that matters, not what fills a folder and looks busy in a report. When the tests come from an agent, my standard does not move. I read each one and ask whether it would fail for a reason I actually care about. The ones that would, I keep.

A suite full of tests nobody trusts makes a team slower and less certain, not more mature.

Reading each one and cutting the rest is most of the work when you are reviewing AI-generated tests.

The coverage number went up and nothing else did

Coverage is the easiest number to move and the easiest to fall in love with. Tell me how a team is measured and I will tell you how it behaves: make coverage the goal and people write tests that lift the percentage the fastest way possible, which is almost never the way that finds bugs.

AI made that trivial. Point an agent at “get coverage up” and it will, line by line, with tests that run the code without ever checking the code is right. You end up at ninety percent with the same defects still in there, because most bugs depend on how a line runs, not merely whether it ran at all. A high coverage number looks like rigour and proves almost nothing on its own. Coverage is a hint about where your testing is thin, worth a glance, never a target to chase. It is one of the QA metrics that mislead teams, and AI just made it easier than ever to game.

Everything is passing, and I still would not ship it

The most seductive thing an agent produces is not a test at all. It is a verdict. “All tests passing,” with a tidy paragraph of analysis under it. It reads as truth because it is fluent and instant, but fluent is easy to mistake for verified, and a green result from a process you cannot see into is a claim rather than evidence.

So I ask the same questions I always have. What ran, on what build, against what data, and what is still shaky enough to make us uncomfortable. A thousand passing tests can sail right past the one workflow that loses customers. A beautiful dashboard can sit on top of bad data. A green pipeline can make a whole team feel safe while the product is quietly wrong. The fix is context around the result, whoever produced it. It is the same question I keep coming back to in is that green checkmark real.

Every feature has a neat little test, and the seams have none

Hand an agent a feature and you get lovely tests for it: clean, isolated, one assertion per behaviour, every happy path covered. Genuinely useful, and also exactly the trap testers have warned about forever. Nobody uses software one feature at a time. People string features into tasks, and the expensive bugs live in the seams between them, in the sequence no single feature test ever walks.

Add an item, send the app to the background, come back, change the quantity, lose the network, retry the payment. No tidy per-feature test catches that, because the failure only exists when the steps are combined. AI is strong at the isolated check and weak at imagining the messy, combined journey a real person takes, which is precisely where your scenario thinking earns its keep.

When the suite is green but the product still feels fragile, this is usually why: every part was tested and the path through them never was.

The tests prove it worked, not that it left everything else alone

This is the one that gets past almost everyone, and AI misses it completely. We check that the thing we changed changed. We hardly ever check that nothing else did.

Picture an admin updating the billing address on one customer’s account. Nearly every test confirms the new address saved. Far fewer confirm that no other customer’s record moved, that the plan tier and payment method and invoice history are untouched, and that the audit log recorded one change and not fifty. The ugly bug here is the update that quietly writes to every record instead of one, and it walks straight through a suite that only asserts the happy path. Generated tests inherit that blind spot by default, because the prompt described what should happen, not what should stay exactly as it was. So I check both: that the intended change landed, and that the record next to it never moved.

The pattern under all of it

Line these up and the shape is obvious. Every one is a case of producing more testing output while quietly losing the thinking that made testing worth doing. That is not really about AI.

AI is an amplifier: point it at a team with sharp judgment and it makes that team faster; point it at a team chasing numbers and it makes the numbers lie faster.

I make that case in full in QA as the control layer for AI-assisted development.

Here is the same idea in one picture, because the speed changes and the mistakes do not.

Keep this where you will see it. When testing feels busy but not safe, find your symptom and the fix is already there.

What it looks like now	What is really happening	What to do instead
The agent generated 200 tests overnight	You are building a liability fast	Keep the ones that fail for a reason you care about; cut the rest
Coverage jumped to 90% this sprint	A number moved, the risk did not	Read where coverage is thin, then design for the risk, not the percentage
”All tests passing,” no detail	A verdict with nothing behind it	Attach the build, the data, what ran, and the unresolved risks
Every feature has its own clean test	The path through the features is untested	Add scenario tests that combine features the way users do
Tests assert the happy path only	The side effects are invisible	Assert what should have stayed unchanged, not just what changed

Where this leaves you

None of this needs a new skill. It needs an old one, pointed at new output. The questions a good tester has always asked still cut straight through an agent’s confidence: what does this actually prove, where can this break, what happens with real data, and what would make us uncomfortable shipping it. Speed without judgment is just faster guessing, and an agent guessing at scale is still guessing.

So the next time your testing feels fast and frictionless, treat that feeling as a signal to look closer. Pick one green suite this week and put it through those questions. You will almost certainly find an old mistake running at new speed, and you will already know what to do about it. The tools keep reinventing the same problems, and knowing your craft means you keep recognising them on sight.

This piece is about spotting the old mistake under the new speed. The companion is the how-to: once you can see these, the way you take the agent off the leash without getting bitten is to set the guardrails once and let it run. I walk through those guardrails in new testing pitfalls to avoid in the age of AI.