On Day 0 I said the most disappointing AI output is the model working blind, so before we widen what it can see, today is an honest map of what a model sees well and what it does not see at all.
I felt this hardest building Tropic Tumble, my match-three game, with AI agents doing the coding. The agents were my development team, and they were good: they wrote the game logic and passed every unit test. What they could not tell me was that the pieces were overlapping on a small screen, that the tap targets were too tight for a thumb, or that an animation was subtly wrong. Every one of those I caught the same way: I opened the game on a real phone and looked. On mobile and on games this gap is at its widest, because so much of the product is visual, but it exists on every web app I have ever tested. The judgment about what was right stayed mine the whole way through, and nowhere was that clearer than the visual calls.
Where AI carries real weight
- Contracts and integrations between services.
- Combing through large amounts of data, logs, and test output.
- Spotting that the backend is not returning what it should.
- Generating web test scenarios, because it has read more of them than any of us ever will.
- Summarising logs and clustering failures.
Notice the shape of that list. It is all work where the truth lives in data a machine can read.
Where the human is not optional
- Seeing that a UI is off-centre or a button is too big.
- Visual and game testing across real devices and screen sizes.
- Unified end-to-end user-flow testing: the whole shape of the thing, not each step.
- Exploratory testing: deciding what to try next based on what you just saw.
- The business decisions and the call to ship.
The truth in this list lives on a screen, in a hand, or in a judgment about people. No log line carries it. A model reading a passing test log has no idea whether the screen it just tested looked right to a real person, and neither do you unless you looked.
A passing test proves an action was taken. A human on a real device proves the screen was right.
The habit: split your checklist in two
The exact split shifts by product, but the shape does not: there is a machine half and a human half, and the human half is bigger than people think. So split your own checklist deliberately.
- Hand the machine half to the tools. The contract checks, the data comparisons, the log combing, the hundred-variant runs. The rest of this course is about doing that handoff well.
- Reserve your own time for the human half. The flow, the feel, the device. My written version of that half is the visual QA checklist: layout, spacing, states, real devices, everything from the second list above. Open it and run it on something you are testing this week, on a real phone. That is the half no model covers, and running it deliberately is what keeps it from quietly falling off your plate while the tools speed everything else up.
- On any AI-assisted change, do a human pass on a real device before you ship. Not a screenshot. A real phone. This one is non-negotiable for anything customer-facing.
Try this today
Take one recent change you tested, AI-assisted or not. Write your checks for it in two columns: what a machine could have verified, and what needed your eyes. If the second column comes out empty, look harder. In more than a decade of testing I have never met a customer-facing change where it stayed empty.
Tomorrow: giving the model real access, which turns the first list from theory into work it actually does for you.