AI Visual Testing: What It Can Check and Where Humans Decide

“The AI looked at the screen” is not enough evidence. I need to know what it actually inspected.

Did it read the accessibility tree? Compare a screenshot with a baseline? Send the image to a vision-capable model? Drive the app on a real device? Those are different checks. They find different problems, and they leave different gaps.

The phrase AI visual testing gets used as if all of that is one thing. It is not. If you do not separate the techniques, a confident report can make a weak test look much stronger than it is.

Do not ask only whether AI saw the screen. Ask what evidence it used and what that evidence can prove.

”AI can see the UI” can mean five different things

Start with the evidence. That tells you what kind of result you actually have.

Only one of those five techniques is explicitly a model interpreting an image. Screenshot comparison can be a deterministic pixel check. Structure inspection can be a normal automated assertion. Device automation can run without a vision model at all.

That distinction matters. Calling every tool “AI” hides the test design.

Structure and rendering are different contracts

A structural check answers questions such as:

• Is the sign-in heading exposed correctly?
• Does the email field have an accessible name?
• Is the submit button disabled in the expected state?
• Does the page contain the controls the flow requires?

A rendering check answers a different question: did the pixels change?

Playwright keeps those contracts separate. Its ARIA snapshot assertions compare the accessibility structure of a component or page. Its visual comparisons compare a screenshot with an approved baseline.

Here is a small pattern you can adapt to your own login screen:

import { test, expect } from '@playwright/test';

test('login keeps its structure and rendering', async ({ page }) => {
  await page.goto('https://the-internet.herokuapp.com/login');

  const login = page.locator('#content');

  await expect(login).toMatchAriaSnapshot(`
    - heading "Login Page" [level=2]
    - textbox "Username"
    - textbox "Password"
  `);

  await expect(page.getByRole('button', { name: /Login/ })).toBeVisible();
  await expect(login).toHaveScreenshot('login-content.png');
});

The first assertion can catch a lost label or a button that disappeared from the accessible structure. The second can catch an unexpected rendering change. Neither tells you, by itself, whether the new layout is an improvement.

Screenshot baselines also need controlled conditions. Playwright warns that rendering can vary by operating system, browser version, hardware, settings, and headless mode. Generate and compare baselines in the same environment. If your supported browsers or platforms render differently, keep the appropriate baselines instead of treating one machine as universal truth.

What a vision-capable model adds

Current multimodal models can process images. That means an agent can capture a fresh screenshot, send it to a vision-capable model, and ask questions about what is visible. The OpenAI image and vision guide documents image analysis directly. This is more than reading a DOM tree.

That review can be useful. Give the model the current screenshot, the approved reference, the target viewport, and the acceptance rules, and it may flag:

• text cut off inside a control;
• elements that appear to overlap;
• inconsistent spacing between repeated cards;
• missing imagery or an unexpected empty region;
• an obvious mismatch with the supplied reference;
• content that appears too small or low contrast and needs measurement.

Use those findings as triage, not as automatic truth. Anthropic’s vision documentation notes that low-quality or very small images can produce mistakes, spatial localisation is approximate, and outputs should be verified. A model can miss a subtle problem, describe the wrong region, or confidently report an issue that is not there.

It also cannot experience the interaction from a screenshot. One still image does not tell you whether the popup flashed open and closed on the same touch, whether the animation timing feels wrong, or whether a control is difficult to reach with a thumb.

This is why the input matters. A stale screenshot, a compressed screenshot, or one viewport presented without context creates false confidence before the model even starts.

Platform changes the evidence available

The test design has to match the product.

Web

On the web, you can combine the DOM, accessibility tree, browser events, network activity, console output, traces, and screenshots. That is a strong evidence set. It still needs coverage at the viewports, themes, zoom levels, browsers, and states your users rely on.

A desktop screenshot does not clear mobile. A light-mode baseline does not clear dark mode. A perfect DOM does not clear a clipped menu at 200 percent zoom.

Mobile

On mobile, you add device size, density, operating-system version, orientation, safe areas, virtual keyboard behaviour, gestures, and touch. Simulator and emulator runs are useful, but they are not the whole release decision. Apple provides separate workflows for simulated and physical devices, and Google provides Firebase Test Lab for testing across virtual and physical devices.

The useful question is not “Did it pass on mobile?” It is “Which build ran on which device, in which orientation and state, and what evidence did we capture?”

Games

Games add renderer state, animation, timing, input routing, scene transitions, layered interfaces, and feel. A screenshot can expose an overlap. It cannot replay the path that created it or tell you whether a touch landed on the visible art.

The concrete Tropic Tumble case study belongs in The Visual Bugs AI Missed in My Game, and How I Caught Them. That article has the actual screens, defects, risk areas, test cases, and release gate. This article stays with the testing model you can apply across products.

Build an evidence ladder

Do not make one technique carry the whole visual decision. Stack the evidence from the most repeatable checks to the most contextual review.

1. Assert the structural contract

Verify the roles, labels, states, data, routes, and required controls. Keep these checks focused enough that a genuine contract change produces a useful failure instead of a wall of snapshot noise.

2. Compare controlled screenshots

Capture stable states at named viewports. Control animations, seeded data, fonts, browser version, and environment. Review and approve baseline changes deliberately.

3. Add multimodal review where it helps

Use a vision-capable model to triage current screenshots against a reference and a written visual contract. Ask it to separate visible evidence from inference and to state what it cannot verify.

4. Run the real journey

Exercise the state transitions, not only the landing screen. On mobile and games, include representative real devices and actual touch. On the web, include keyboard, zoom, responsive navigation, themes, and the states that usually get missed: loading, empty, error, disabled, and long content.

5. Make the release call

A human reviews the evidence, the remaining risk, and the experience. Automation can show that the accepted baseline did not move. A model can suggest that the layout looks inconsistent. Neither owns the business decision that this is good enough to put in front of a user.

Give the model a visual-review contract

“Check whether this looks good” is a weak prompt. It gives the model no target, no boundaries, and no reporting standard.

Use a brief like this instead:

# Visual review brief

Screen: [name and user goal]
Build: [commit or build identifier]
Platform: [browser/device, operating system, viewport, orientation]
State: [loading, populated, empty, error, disabled, success]
Reference: [approved screenshot or design]

Review only what is visible in the attached current screenshot.
Compare it with the supplied reference and these rules:

- No text, image, or control may be clipped or overlap another element.
- The primary action must remain visible and distinct.
- Repeated cards must use consistent alignment and spacing.
- Text that may have low contrast must be flagged for measurement.
- Do not infer behaviour, touch accuracy, or off-screen content.

Report:

| Region | Visible evidence | Expected reference or rule | Confidence | Human check needed |
| --- | --- | --- | --- | --- |

If the image is too small, stale, obstructed, or missing a required state,
say that the review is incomplete. Do not invent a pass.

That contract does three useful things. It ties the review to a build and environment. It makes the model cite visible evidence instead of offering taste as fact. It also creates an explicit route back to a human when the image cannot answer the question.

You still validate every reported defect before filing it. If the model says two elements overlap, open the same build at the same size and confirm that they do. If it says the text may have low contrast, measure the actual colours. A suggestion becomes a QA finding only after the evidence holds up.

The human pass is smaller when the automation is better

Human review does not mean randomly clicking through every screen and hoping your eye catches something. Good automation narrows the work.

• Structural tests protect the product contract.
• Screenshot comparisons show where the rendering changed.
• Model review can triage a large screenshot set and point to suspicious regions.
• Device automation reproduces flows and collects logs, traces, and captures.
• A tester investigates the meaningful changes and makes the release call.

For the hands-on questions to run during that final pass, use the separate Visual QA Checklist. Keeping that checklist in one place means this article does not duplicate it every time the workflow changes.

AI visual testing is useful when you know what kind of evidence you are holding. A structure pass is not a rendering pass. A pixel change is not a defect. A model’s description is not a measurement. A simulator run is not every device. Stack the techniques, verify the findings, and keep the human where judgment is actually required.