QA Is the Control Layer for AI-Assisted Development

The real enemy is false confidence

The problem was never that teams use AI. They should. It is one of the most useful tools to arrive in our field in years.

The problem is when speed starts pretending to be proof.

Watch how it happens. A team asks an agent to fix a bug. It reads the files, changes the code, adds a few tests, and opens a pull request. The summary sounds reasonable, the tests are green, the diff looks tidy. Someone says “looks good,” and the team moves on.

That “looks good” is the enemy. Not AI. False confidence: a confident summary, generated tests that assert nothing, a fast pull request, and a team mistaking motion for quality. A green check is comforting, but comfort is not proof. If you have shipped for any length of time, you already know exactly how this goes wrong. You have seen it.

In the 2025 Stack Overflow Developer Survey, the single biggest frustration developers named with AI tools was solutions that are “almost right, but not quite.”

Almost right is the most dangerous kind of wrong, because it passes a quick glance. It survives the demo. It clears the happy path. It gives everyone just enough confidence to stop reading. Catching it before it reaches a real user is exactly what you are for.

QA is not the end gate. It is the control layer.

A control layer is the set of checks and decisions that sit between “the work is generated” and “the work is in front of a real user.” It is not one person standing in the doorway saying no. It is the system of judgment that keeps speed connected to trust.

QA has always been that judgment: the person helping the team understand risk before the damage gets expensive. AI has not changed the role. It has made it more urgent, because the volume arriving at the decision point went up by an order of magnitude. More generated code, more places for a weak assumption to hide. More generated tests, more chances to confuse activity with coverage.

The control layer is where you define the things the model cannot decide for itself:

• what risks have to be considered before any work starts
• what evidence is required before a change is accepted
• which tests are strong enough to keep, and which are theatre
• what signals get watched in production after the release goes out
• which failure modes are unacceptable, even when the happy path works

None of that is new QA thinking. What is new is where it sits. It can no longer live at the tail end of the process. It has to wrap around the work as a loop.

The dashed line is the part teams skip. What breaks in production is the best possible input to the next risk check. Skip it and you are back to shipping on crossed fingers, just faster.

The room you are actually in

Being the control layer is a set of conversations you will have this week, and they are not always comfortable. Three lines you will hear, and the answer that holds:

• “The agent already tested it.” Show me the test that would have failed before this change, and what it proves about the user outcome.
• “We are moving fast, can you just sign off?” Here is what I can sign off on with the evidence we have, here is what I cannot, and here is the monitor for the gap.
• “The build is green, why block the PR?” Green is the floor, not the proof. What does this actually prove?

You are not blocking the team. You keep speed from turning into damage, and teams that learn that ship faster, because they stop re-fixing the same almost-right in production.

Check the real behaviour, not just the screen

The easiest AI mistake to miss is a clean screen with the wrong system behind it. That is just as true when AI builds an entire feature or app as when it edits a single file. The more it generates in one pass, the more surface there is where the shape of done hides a system that is not.

When AI built this blog, the homepage looked finished on my laptop. On a phone, sections overflowed the edge. Dark mode swallowed headings. The newsletter box looked completely real and did nothing useful until I wired it to an email platform and sent myself a test. The agent was not lying on purpose. It had produced the shape of done.

I see the same pattern everywhere I have worked, in fintech, in mobile, and in the game I am building at JPott Studios, Tropic Tumble. The button works, the toast appears, the test passes, and the record underneath is wrong. An agent that “fixes” a duplicate-player bug by hiding the duplicate from the owner’s view will hand you a clean diff and a passing test. A tester who understands the system says: this only hides the symptom. Prove the database cannot create a second player in the first place.

That one sentence is worth more than the entire generated patch. That sentence is the control layer.

Check what the system actually did, not just what the screen showed.

An agent tests the visible step, because that is the easiest thing to observe. You check the real behaviour: not just the success message, but the receipt validated, the entitlement granted exactly once, the state that survives a reinstall. I walk that whole chain, link by link, in Done Is the Whole Chain.

Testing a whole feature AI built, not just a diff

Reviewing one AI change is one thing. Handing an agent “build the checkout” and getting back a working feature is another. Same control layer, bigger surface, different traps: no one made the tradeoffs on purpose, so the gaps are accidental; the bugs hide in the seams where parts meet; and every layer arrives polished at once, so nothing tips you off that something underneath is missing.

Test it like a system you did not design, because you did not:

1. Start from what the user was told, not the code. Write down what a real user should be able to do end to end before you read a single generated file.
2. Walk the whole journey on real infrastructure, with real data. The blog’s newsletter box looked real until I submitted it and watched nothing arrive.
3. Push on every seam: bad input, a dropped network, a second submit, a reload halfway through.
4. Check state, not just screens. AI is very good at making a screen say success while the record behind it is wrong.
5. Test the negatives it never wrote: the error, the refusal, the permission check. Almost always missing, almost always where the risk lives.

A whole AI-built feature is finished exactly up to the edge of what the agent thought to try. Your job is everything past that edge.

What AI helps with, and what you still own

Drawing this line clearly is most of the job. AI is genuinely good at breadth and speed. It does not decide whether the evidence is enough to ship.

AI helps with	You still own
Drafting the test-plan structure and the obvious cases	Deciding which risks actually matter for this release
Generating a first pass of automated tests	Judging whether a test proves anything real
Summarising a pull request and what changed	Confirming the change solved the right problem
Producing test data and fixtures at volume	Knowing what real-world and edge data looks like
Combing contracts and integrations for a broken response	Seeing that a UI is visually or practically wrong
Writing the boilerplate around a check	Designing the assertion the check hangs on
Spotting patterns across a large diff	Understanding the human cost when it breaks

You bring the judgment; the AI saves you the typing. That is not a smaller job than it used to be. It is a harder one, because everything in the right-hand column is exactly what you cannot fake with a green checkmark.

Hand AI the mechanical work. Keep the human for judgment, visual sense, risk, and the release decision. I go deeper on where AI is strong and weak in the MCPs every QA should switch on.

Read AI-generated tests with real suspicion

AI will happily generate a hundred tests in a minute. Volume is not coverage, and a suite full of weak assertions is worse than no suite at all, because it manufactures confidence you have not earned.

A test that turns green while the user experience is broken is not evidence.

When you review tests an agent wrote, hunt for these tells.

What the generated test does	Keep it?
Assertion only checks the page loaded	✗
Mocks out the exact behaviour under test	✗
Brittle locators copied from the live DOM	✗
Asserts implementation details, not behaviour	✗
No negative path or error case	✗
No check that state persists	✗
Nothing for permissions, timeouts, or network failure	✗
Fails for a real product problem, passes otherwise	✓

The bottom row is the only one you want. A generated test earns its place when it would fail for a genuine product bug and pass the rest of the time. Everything above it is noise.

The most common version I see is the assertion that proves nothing. An agent gives you something like this:

import { test, expect } from '@playwright/test'

test('checkout works', async ({ page }) => {
  await page.goto('https://www.saucedemo.com/')
  await page.getByPlaceholder('Username').fill('standard_user')
  await page.getByPlaceholder('Password').fill('secret_sauce')
  await page.getByRole('button', { name: 'Login' }).click()
  await expect(page).toHaveURL(/inventory/)
})

That test is named “checkout works” and never reaches checkout. It logs in and confirms the URL changed. It stays green through a broken cart, a failed payment, or an order that never gets created.

A real checkout test has to walk the whole flow and prove what the customer and the business actually care about, which is a lot more than one success toast:

• the right item, at the right price and quantity, made it into the cart
• the totals and any tax add up
• the order is truly created on the backend, not just shown on a success screen
• a confirmation the customer can rely on comes back
• and the paths that go wrong: a declined card, an empty cart, a double submit that must not double-charge

That is the bar, and it is exactly the depth AI skips. Reviewing generated tests against it, every time, is some of the highest-value work a tester can do now. I have a fuller checklist, with the passing and failing versions side by side, in how to review AI-generated automated tests.

The documents AI writes lie the same way

Tests are not the only thing an agent hands you with a confident face. It also drafts the specs, the acceptance criteria, the API docs, the release notes, and often the test plan itself.

A clean, well-formatted document is the same trap as a green check. It reads like it was written by someone who checked the software. Usually no one did.

And a wrong document costs more than a wrong test, because everything downstream inherits the mistake. If the spec describes a behaviour the product does not actually have, every test written from it proves the wrong thing, and the release note ships that same fiction to your users.

So read what AI wrote about the software against the software, not against the prompt that produced it:

• The spec: does it describe what the system actually does, or what someone hoped it would do?
• The acceptance criteria: do they name the real risk, or only the happy path?
• The API doc: does it match the real response, field for field, including the error cases?
• The release note: does it tell a user what actually changed, or just tidy up the diff?

The test is the same one you run on code. Would this document fail if the software were wrong? If it would happily stay “correct” while the product broke, it is decoration, not documentation.

Put your judgment where the AI can read it

The first few times you work with an agent on your codebase, you will correct it a lot.

• Brittle CSS selector? You tell it to use a role or a test id.
• A fixed sleep? You tell it to wait on a real signal.
• A test with no stable id and no real assertion? You send it back.

That correction work is the control layer in action, and most teams throw it away by keeping it in their heads and re-typing it every run. So the agent makes the same mistake next session, and you spend your judgment on the same five things instead of the new risk.

Do not. After a few rounds, have the agent write your corrections into a rules file at the repo root, a CLAUDE.md or AGENTS.md, so it follows them by default:

Based on every correction I gave you this session, write a CLAUDE.md at the repo root with the rules you should follow for this codebase. For each rule, give a short example of what passes and what fails. Link to our locator standards and our test-plan skill so you pull the whole system, not just this file.

What comes back is the start of a standard that lives in the repo instead of in your memory:

# Test rules (read before writing or changing any test)

- Locators use role, aria-label, or a data-test id. Never a CSS class or nth-child chain.
- Wait on a real signal: an API response, or an element becoming visible. Never a fixed sleep.
- Every test has a stable ID and would fail if the behaviour actually broke.
- Every change ships with evidence: the failing run, the passing run, and what it proves.

See ./locator-standards.md and the test-plan skill for the rest.

Now the agent reads that before it writes, so the corrections you made this week become the defaults next week. This is one of the most practical ways QA can lead AI adoption: not with a speech about quality, but by making the standard visible, repeatable, and enforceable. I go deeper in agentic testing and what QA teams should try first.

The five questions I will not skip

When an AI-assisted change lands in front of me, I do not start from “is the build green.” Green is the floor, not the proof.

I start from five questions, and I will not approve the work until I have a real answer to each:

1. What actually changed? The diff, not the agent’s summary. If you cannot tell me in one honest paragraph, neither of us understands it well enough to ship it.
2. What is the risk? Where can this break, who gets hurt, and what is the worst version? A receipt-validation change and a copy tweak do not deserve the same scrutiny.
3. What evidence proves it works? A test that would have failed before this change and passes now, a screenshot from a real environment, a checked API response. “It passed” is only evidence if the test could have failed.
4. What evidence proves the adjacent behaviour did not break? This is the one AI work skips almost every time. What proves the neighbouring flow, the shared component, the downstream report still does what it did yesterday?
5. What will we monitor after release? For anything you cannot prove before it ships, what signal tells you it went wrong, who owns it, and what is the rollback? An unverifiable change without a monitor is a guess with a deploy button.

None of these are AI questions. They are what a good tester has always asked at the edge of a release. What AI changed is the volume arriving there, which is why writing them down beats holding them loosely in your head.

Your Monday-morning control-layer checklist

You do not need permission, a new title, or a company AI strategy to start being the control layer. You need a short list you actually run on the next AI-assisted change that crosses your desk.

Print it, pin it, use it on Monday:

• Read the diff, not the summary. Can I describe what changed in one honest paragraph?
• Write the one-line risk note. Where can this break, and who gets hurt when it does?
• Find the proof it works. Is there a test that would have failed before this change and passes now?
• Find the proof nothing else broke. What covers the adjacent flow, the shared component, the downstream report?
• Check the evidence is real, not green. Does anything here only pass because the assertion proves nothing?
• Confirm it was checked where it runs. Real environment, real device, real data, not just a local happy path.
• Name the monitor for what you cannot prove. What production signal will catch this, who owns it, and what is the rollback?
• Decide out loud. “Ship, with this evidence” or “not yet, because of this gap.” Either is fine. Silence is not.

If a change clears every box, ship it and feel good about it. If it does not, you have just done the most valuable thing a tester can do in an AI-assisted team: you caught the almost-right before a user did. That is shipping with evidence instead of crossed fingers. To turn this into a documented approach for your whole team, the test strategy builder walks you through it question by question.

This is the job, and it is yours

This is not the moment to shrink back. It is the moment to become harder to replace, by becoming the person who turns speed into trustworthy delivery. Your value moves from “I ran the test cases” to:

• I named the risk before the work was built.
• I caught the test that passed but proved nothing.
• I saw the data was wrong even though the screen looked fine.
• I made the release decision visible instead of letting it hide behind a green build.

That is not a smaller job. It is a stronger one.

I started in manual testing and moved into automation because I saw how much repeatable evidence matters. But automation never changed the core skill. The job was never “click the thing.” It was understanding what the thing was supposed to do, where it could break, and what proof the team needed before trusting it. AI does not remove that skill. It exposes whether you have it. The 2025 DORA report found the same: AI is an amplifier, making a team with strong testing habits faster and safer, and a team without them faster and shakier.

The mission has not changed. QA has always existed to protect trust between a product and the people who use it. AI makes output cheaper. You make trust possible. When software can be generated faster than anyone can read it, someone has to stand at the edge of the release and ask, “Should this ship, and what proof do we have?” That someone is QA. That someone can be you, on the next pull request, with the checklist above and the five questions in your back pocket. You are not being left behind by AI. You are exactly who this moment needs.

# Test rules (read before writing or changing any test)

- Locators use role, aria-label, or a data-test id. Never a CSS class or nth-child chain.
- Wait on a real signal: an API response, or an element becoming visible. Never a fixed sleep.
- Every test has a stable ID and would fail if the behaviour actually broke.
- Every change ships with evidence: the failing run, the passing run, and what it proves.