How to Test AI Features With Non-Deterministic Output

I was testing an AI feature that turned a conversation transcript into a short summary, and the model never wrote the summary the same way twice. There was no exact string I could assert against. What had to hold every single time was a privacy rule: if the transcript named a person, that name could never appear in the summary. So my input was a transcript with a known name in it, and my check was simply that the name, and its obvious variants, never showed up in the output, no matter how the model phrased the sentence. That one property caught a real bug where names were leaking into summaries. No exact-match test would have found it.

That summary feature lived in a product that runs realistic conversation simulations to train contact-centre agents, the kind of thing used in high-stakes settings like crisis support. The system answers differently every time, and a weak answer can carry real human weight. A plain exact-match assertion is the wrong tool from the very first run.

That habit is the one to unlearn first. Most of us reach for it without thinking: send a question, capture the answer, assert the response equals that string. It passes once. On the next run the model words the same correct answer differently, and the test fails on a response any reviewer would approve. Nothing is broken except the test.

So I stopped testing AI features that way. Three moves carry the work now:

• Assert properties, not exact answers. Check the things that must be true of any acceptable output, instead of one specific string.
• Ship on thresholds and distributions. Run each case many times and read the spread, rather than a single green or red.
• Do not lean on temperature zero. It does not buy back determinism, and I will show you why further down.

None of this needs new tooling. It starts with giving up that one habit, so let me start there.

Stop testing for an exact answer

So instead of checking for an exact output, I check the properties a good output must have. When a feature generates a response in a conversation, I am not asserting one specific string, because it will phrase the same correct answer a dozen ways. I am asserting the things that should be true of any acceptable response:

• It stays on topic.
• It does not contradict the information it was given.
• It does not invent facts that were never provided.
• It answers in the right language.
• It stays within a length the interface can handle.
• It holds the tone the product requires.

Each of those is something you can actually check. Length and language are straightforward. Staying on topic and not inventing facts can be approximated by checking that the response is grounded in the input it was given. Breaking a fuzzy question like “is this a good response” into a set of concrete properties is what makes a non-deterministic system testable, even though the exact words are never the same twice.

The privacy property I opened with is one of these. It was never about the words. It was about a fact that had to be true of every summary the model wrote, and a fact like that is something you can pin down and check on any output, however it comes out worded.

Whatever the feature, the properties fall into the same five groups. This is the checklist I start from for any AI output:

Property checklist for any AI output

Shape      [ ] Output is in the format the interface expects (parses, required fields present, right language)
Bounds     [ ] Length fits what the UI can render; any numbers fall inside allowed ranges
Safety     [ ] Nothing the product must refuse; no private data (names, emails, IDs) leaks into the output
Grounding  [ ] Every fact traces back to the input or the source material; nothing invented
Latency    [ ] Response arrives within the time budget the product promises

Here is the same shift laid out visually.

You can also assert relationships between runs, not only properties of a single output. Swap a word in the input for a synonym and the meaning of the answer should not flip. Negate the request and the answer should reverse. Those relationships hold even when the surface text is never the same twice.

Build a set of cases you genuinely understand

The backbone of testing an AI product is a curated set of representative inputs, each paired with what a good response looks like, defined by its qualities rather than its exact wording. Choose these cases deliberately and include the hard ones:

• The empty input.
• The hostile input.
• The input in a language the product does not support.
• The input that tries to push the system somewhere it should not go.

This set becomes your regression safety net. AI products change constantly, not only when the code changes but when the underlying model or its configuration changes, and a small adjustment that looks harmless can quietly break behaviour you relied on. Running your curated set after each change shows you what moved. Keep it small enough that you actually understand every case in it, and weighted toward your real risks rather than padded for the sake of a number.

Thirty cases you understand deeply are worth more than a thousand nobody has ever reviewed.

You do not need hundreds to start. Anthropic’s own guidance on evals for AI agents recommends beginning with twenty to fifty tasks drawn from real failures and bug reports, and growing the set over time. The same advice gives you a useful quality bar for each case: a good test case is one where two people who know the domain would independently reach the same verdict on whether it passed. If your team cannot agree on whether a response was acceptable, the case is too vague to test yet, and you have found a definition problem worth fixing before you write any code.

Evaluate quality with a clear rubric

For the parts of quality that simple rules cannot capture, such as whether a response is genuinely helpful or appropriate in tone, the practice that has settled in is to use evaluations, usually shortened to evals. You score outputs against a clear, written rubric rather than against an exact answer. Sometimes a person scores them, and increasingly a separate large language model applies the rubric at scale, an approach people call “LLM as a judge.”

This is useful, but you have to use it with your eyes open. A model scoring another model’s output is an estimate, not ground truth, and it carries its own biases. Two are well documented and worth designing around:

Bias	What happens	What to do about it
Verbosity bias	Model judges reward longer answers, often more strongly than human reviewers do, even when the longer answer is not better.	Cap or normalise for length in the rubric so verbosity cannot buy a higher score.
Self-preference bias	A judge tends to score its own generations more highly than a human would.	Grade with a model from a different family than the one being tested.

So treat eval scores as a signal that flags responses for human attention, not as a gate that ships product on its own. Validate the evaluation itself against cases where you already know the right answer before you trust it on cases where you do not. Two practical habits keep judges honest: write the rubric so each quality is scored separately rather than collapsed into one number, and calibrate the judge against human ratings regularly, since drift creeps in as the model and the rubric both evolve. The rubric is doing the real work here, and writing a good one is a testing skill in its own right.

Test the product adversarially

Adversarial testing is where I see teams put in the least effort, and on an AI product that is a costly place to go light. A real test effort does not only check that the product behaves well when used as intended. It actively tries to make it misbehave: the input crafted to pull the system off its instructions, the attempt to get it to produce something it should refuse, the prompt designed to extract information it should keep private.

This practice, often called red teaming, has moved from a niche security activity to an expected part of testing any AI product. Trying a few odd prompts before release does not count. What counts is a written, maintained set of adversarial cases that runs alongside the rest of your suite.

I treat this exactly like security testing, because that is what it is. And like security testing it is never finished, since new ways to break these systems keep appearing, so your adversarial cases grow over time as you and the wider field discover them. The stakes climb again when the product does not just generate text but takes actions on its own, which is where what QA teams should try first with agentic testing picks up. For a product in a sensitive domain, this is the part that protects the people using it.

Reduce the randomness where you can, and know its limits

One technique makes everything else easier. Where the product allows it, run one layer of your tests against the most stable configuration available, so the behaviour is as repeatable as you can get it and you can isolate logic and integration bugs without fighting the variation. Then keep a separate, smaller layer that runs against the real production configuration to catch the issues that only appear with genuine variability. Mixing the two in one suite is how you end up with tests nobody trusts, because they cannot tell a real failure from normal noise.

One caution that catches teams out: turning the temperature setting down to zero does not give you a reproducible system. It only makes the model’s token choice greedy. The output can still differ run to run, because most inference servers batch requests together and the result for one request depends on the batch it landed in, which shifts with server load. One study of nondeterminism in LLM inference demonstrated this plainly: the same prompt, a thousand times, at temperature zero.

So pursue stability as a way to make debugging tractable, but design your tests to tolerate variation rather than assuming you have eliminated it.

Accept a different definition of passing

The hardest adjustment has nothing to do with tooling. It is how you define success. A traditional suite is green or red. Testing an AI product lives in thresholds and distributions. You might decide that quality scores across your curated set must stay above a certain level on average, and that no single critical case may fall below a hard floor. You will have behaviour that is acceptable most of the time and occasionally not, and you have to decide in advance how much is acceptable for this particular product. For something low-stakes, a rare imperfect response may be fine. For a product operating in crisis support, the bar for the cases that matter most has to be far higher, and some failures cannot be tolerated at all.

Deciding which level of reliability is good enough to ship is a quality judgement, and it belongs to the testing function. Write it down where the rest of your approach lives; if you need a structure for that, my test strategy builder gives you one to start from. The tooling can measure the distribution. Deciding whether that distribution is acceptable for the people who depend on the product is the part of the job that does not change, pointed now at a system that will not give the same answer twice. It is one of the new testing pitfalls in the age of AI that a green pipeline can hide a model quietly drifting below the bar you set.

Hand AI the data, keep the judgment for yourself

The same split runs through testing an AI product as runs through every other kind: give AI the data, the contracts, and the analysis it combs faster than you can scroll, and keep for the human the judgment, the visual sense, the real user flow, the risk, and the call on what is good enough to ship. I make the full case for that division, and the guardrails that hold it in place, in the QA control layer for AI-assisted development.

If you take one idea from all of this, let it be the shift in the question:

Stop asking whether the output is exactly correct, and start asking what must be true of an acceptable output and how confident you need to be that it holds.

Framed that way, most of the testing skills you already have come straight back into play, pointed at a harder and more interesting problem.

This article touches on AI products used in sensitive settings such as crisis support. Where a product can affect someone's safety or wellbeing, treat the adversarial and quality testing as the priority rather than an afterthought.