Agentic Testing: What QA Teams Should Try First

What “agentic” means now

The word gets thrown around, so here is what I mean by it.

A traditional automation script does exactly what you wrote, in the order you wrote it. An agent is given an outcome and the freedom to decide the steps.

It inspects files, changes code, runs commands, reads the errors that come back, and tries again, looping until it reaches the goal.

What changed recently is the end of that loop. The agent no longer stops at “here is a suggestion.” It can commit, open a pull request, run your suite in continuous integration (CI), and drive a real browser to check its own work. A few tools show where the loop now ends:

• GitHub’s Copilot coding agent works in an Actions environment, makes changes on a branch, and opens a PR.
• Playwright’s Test Agents plan, generate, and heal tests.
• Playwright MCP is a server you wire up so an agent drives a real browser through the accessibility tree and reads real locators and real state instead of guessing.
• Claude Code, Codex, and Cursor are the coding agents I drive day to day: they read the repo, change code, run the suite, and open a PR.
• Computer use (Anthropic’s, and OpenAI’s Operator) lets an agent drive a real computer the way you would: open the browser, click through the app, read the screen. It is rougher than the rest, but real.
• MCP servers are the connectors that give an agent real access to your systems, not just your code: the Atlassian server for Jira and Confluence, a Slack server for the thread, GitHub for the repo. They respect your permissions, so the agent sees only what your own login can see.

You do not have to adopt any of them to get value here, but they are worth knowing exist, because they show where the loop now ends: in something you can review, not a pile of suggestions you have to assemble yourself.

That is more power and more rope. The same loop that fixes a flaky locator on its own will refactor ten files you never mentioned, with total confidence, if you let it. The work is to decide where the rope ends, and to make the agent show you what it did.

Stop re-reviewing. Start setting guardrails.

This is where the re-review habit pays off differently. Reading every line by hand teaches the agent nothing and costs you the same hour every run. Spend that hour once instead, on three guardrails:

1. A rules file the agent reads before it writes, so it makes your corrections by default.
2. A definition of done that means evidence, not “it passed.”
3. Real access to the system, so it works against what is actually built, not what it assumes.

Set those up and your review changes shape. You stop re-deriving every step and start judging a pull request with the proof already attached to it.

Here is how I build each of those three guardrails, and the one decision I never hand over.

Guardrail one: turn your corrections into a rules file

The first few times you work with an agent, you correct the same things: a brittle selector, a fixed sleep, a test with no stable id.

Instead of keeping that in your head and re-typing it every week, have the agent write those corrections into a CLAUDE.md or AGENTS.md at the repo root. It reads them before it writes, so your corrections become the defaults.

That file is the spine of the whole idea: the guardrails belong in the system, not in people’s heads.

Guardrail two: make “done” mean evidence

An agent will tell you the tests pass. That is not the same as done.

A green build is comforting, but it can make everyone feel safe while the product is quietly wrong.

“Done” is when the agent has run the real test in the real pipeline and can show you it proves something. Write that into the definition of done, so the agent has to meet it before it calls anything finished:

• It ran the test in CI, not just locally, and linked the run.
• It watched the test go red for the right reason. The fastest proof a test works is to break the feature on purpose, run it, and confirm it fails. A test that stays green while the behavior is broken is watching nothing.
• It attached the evidence to the pull request: the failing run, the passing run, the screenshots, and the locators it used against the running app.

So proof stops being something you ask for in review. It becomes the bar the agent has to clear before it calls the work done.

Your job moves up a level. The question is no longer “did it write a test,” it is “does this evidence prove the risk I cared about.”

Guardrail three: give it the real system

An agent working from your description of the app will guess. An agent with access to the real system does not have to.

• Give it the backend and frontend code, so it tests what is built, not what you assume.
• Point it at the running app through a browser-driving layer like the Playwright MCP, so it reads accessibility snapshots and pulls real locators and real timing instead of inventing them.
• For mobile, the same loop runs against a real device or an emulator over ADB, on the screen sizes your users actually hold.

For the practical setup list, use Give Your AI Real Context for QA.

This is also where the division of labour has to stay honest. The agent can drive the device, run the flow, capture the screenshots, and diff them against a baseline you approved.

What it cannot do is see that a layout is wrong when there is no baseline yet: the button slightly too big, the spacing that feels off, the thing that is technically rendered but visually broken. That call is yours, and on mobile and game work it is sharply yours. Building Tropic Tumble taught me this the hard way. The agent renders a board, every check passes, and a tile still sits a few pixels off centre in a way only a human eye catches. So I hand the agent the gathering and keep the seeing for myself.

Here are the three guardrails as concrete, checkable setup items. Paste them into your CONTRIBUTING.md or CLAUDE.md and work down the list:

## Agent guardrails for testing work

### 1. Rules file the agent reads before it writes (CLAUDE.md / AGENTS.md, repo root)
- [ ] No fixed sleeps; wait on state or a network response
- [ ] Prefer role- and label-based locators; no brittle CSS or XPath
- [ ] Every test checks the real behaviour, not just that the screen rendered
- [ ] New tests match our existing naming and folder conventions

### 2. "Done" means evidence, not "it passed"
- [ ] Test ran in CI, not just locally, with the run linked
- [ ] Test was watched going red for the right reason (break the feature on purpose, confirm it fails)
- [ ] Failing run, passing run, and screenshots attached to the PR

### 3. Real access to the system
- [ ] Agent has the front and back end code, not just a description
- [ ] Agent drives the running app for real locators and timing (e.g. Playwright MCP)
- [ ] Mobile runs against a real device or emulator over ADB

### The line that stays human
- [ ] A person owns the release call, the risk call, and the visual judgment

Good first tasks, and where not to start

Even with the guardrails, you do not hand an agent the whole release on day one. You build trust where the work is easy to check. These are the tasks where it saves real time and you can verify the result quickly:

• cluster a wall of CI failures into a handful of distinct problems,
• summarize a long log window into a readable breakdown,
• generate edge cases and exploratory charters for a feature,
• draft tests for a stable, well-understood flow, then run them and bring the evidence,
• explain why a specific test is failing, with the line quoted.

That last one is the cleanest first task, because the whole point of agentic testing shows up in it: you ask for a diagnosis with a boundary, and the agent brings evidence instead of a guess.

A green build would never have told you any of that. The agent ran the real thing, brought the actual output, named the layer, and stopped at the boundary. Every good first task has that shape.

And the places not to start an agent, even now:

• payments and billing, authentication, permissions, data migrations, security flows, and anything that changes production.

The thread is the same question I ask before any release: what would make us uncomfortable shipping this? Where a confident wrong answer touches money, identity, who can see what, or data you cannot un-migrate, the agent earns its way in slowly and behind a stricter evidence bar, not on day one.

The ladder runs lowest risk first. You move up a rung only once the one below it earns trust, and the climb itself is the trust-building.

That top rung is the one that does not move, and it points at the real shift in the role. QA’s job here is not to cheer for the tool. It is to set the guardrails, define what “done” has to prove, and keep the call to ship. The agent will happily climb to rung seven on its own if you let it. Deciding it cannot is the work.

Where agents are strong, and where you stay

If you remember one thing, make it this split. Give an agent guardrails and it will do far more than most teams expect. What it still does not do is decide what risk matters, whether the evidence is enough, or whether the thing looks right to a human being.

✓ let the agent run it   ~ let it run, review the result closely   ✗ keep it human

The strong column has a shape worth remembering: contracts and integrations, combing through data, and catching the backend returning the wrong thing. The weak column is everything that needs human eyes, seeing a UI is visually wrong on a real device, exploratory testing, and naming the real risk.

The 2025 DORA report found that around 90% of people now use AI at work and most feel more productive, while trust in the output stays split. That gap is the whole point of the guardrails above.

An agent accelerates whatever practice you already have; it does not hand you a good one. Keep the release call human and accountable, the way the NIST AI Risk Management Framework frames oversight that scales with risk. You move much faster with an agent in the loop. You do not hand it the keys.

The whole loop, end to end

Each guardrail above is one link. Connect them with real access and you get something bigger: an agent that runs the whole QA loop for a team, from a single message, while you hold the judgment.

This is the loop I am building. Some links are wired up today, others I am still connecting, so treat it as the shape to aim for, not a finished product.

It starts with one line in Slack: “Build a test plan for this and tell me what’s missing.” From there:

Every step above runs on the same three guardrails: it reads the real context through MCP, plans and writes by your rules, and proves its work with evidence. Nothing new to learn, just the loop connected end to end.

Then it stops at you. The agent gathers, runs, and proposes. You make the release call, weigh the risk, and make the visual judgment no agent can make yet. That ceiling does not move.

None of this replaces the QA engineer. It clears the busywork from around the judgment, so the judgment is the part left on your plate. That division, where AI does the work and a human holds the line, is the QA control layer drawn end to end.

Start this week

Pick one suite and run the whole shape once. It is part of the same test and quality strategy you already own, not a separate initiative:

1. Set up the three guardrails from the checklist above, or start from the AI test automation standards template.
2. Hand the agent one real task, correct it as you go, and let it write those corrections back into the rules file.
3. Let it open the PR. You review the evidence, not every line, and you keep the release decision.

Then watch it the way you would watch a sharp new teammate who is fast, well-equipped, and not yet trusted with the keys. That is not caution for its own sake. It is how the busywork finally comes off your desk while the judgment stays where it belongs.