Continuous testing does not mean running every test on every change. I learned that on a two-brand retail platform that shipped on a weekly release train. The functional suite was large, and the first instinct on the team was the obvious one: run all of it on every merge request, because surely more checking means more safety.
It did the opposite. The pipeline crept toward forty-five minutes, went amber for reasons nobody could explain, and got waved through with a tired “just rerun it”. The tests were still running. They had stopped meaning anything.
What fixed it was not cutting tests to go faster. It was staging them, so each check ran at the moment its answer was worth waiting for, and the slow, flaky ones came off the merge gate entirely. Quality went up, not down.
In practice that came down to four habits:
- Fast checks run early, while the change is small and fresh in the author’s head.
- Deeper checks run when they earn their cost.
- Risky changes get more scrutiny than a copy tweak.
- The release itself is confirmed in production with safe smoke checks and monitoring, not assumed to be fine because the pipeline went green an hour ago.
More tests was never the goal. What you are really chasing is more confidence per minute the pipeline spends.
DORA’s work on test automation makes the practical point clearly: running tests continuously only helps when the tests are reliable enough that a green run genuinely means the software is releasable.
A pipeline you trust changes how a team works. A pipeline you have learned to ignore is just cost with a green light on top of it.
Why late feedback costs so much
Without continuous testing, problems surface at the worst possible times:
- after a feature branch has grown large enough that nobody remembers all of it
- after QA receives the full build at the end of the sprint and finds a structural issue on day one of two
- after a release candidate everyone already promised to ship turns out to be broken
- after a customer reports it in production
Every one of those is the same bug. The only thing that changed is how much it costs to fix.
The cost climbs steeply the later you catch it, because by then it is tangled up with other work, the author has moved on, and the deadline is closer. Continuous testing is really just a way of moving feedback as close to the change as you can, so the cheap version of the bug is the one you find.
Think in stages, not in “all the tests”
A continuous integration and continuous delivery (CI/CD) pipeline answers different questions at different points, and the trick is to match each stage to the question it is actually there to answer.
| Stage | The question it answers | What runs |
|---|---|---|
| Local, pre-commit | Did I break something obvious? | lint, type checks, fast unit tests |
| Pull request | Is this change safe to merge? | unit, API, component, a few focused UI tests |
| Main branch | Is the integrated product still healthy? | broader regression, contract tests |
| Pre-release | Are the critical flows ready? | smoke, exploratory, cross-browser and device checks |
| Production | Did the release actually work? | safe smoke tests, monitoring, error and funnel alerts |
If your pipeline treats all of these as one big “run the tests” step, two things happen:
- It gets slow. The heaviest checks now gate the fastest feedback.
- It gets untrustworthy. A flaky cross-browser test failing on a one-line backend change teaches people that red does not really mean red.
Stage your checks so each one gives a clean answer to one question.
Start with a test selection model
Before you touch a single YAML file, decide which tests run where. Write it down, even roughly, because that decision is what keeps the pipeline honest as the suite grows.
If you do not have one yet, my test strategy builder walks you through making those choices on purpose. A reasonable starting point looks like this:
pull_request:
run:
- lint
- unit_tests
- api_contract_tests
- focused_ui_smoke
main_branch:
run:
- unit_tests
- api_tests
- ui_regression_subset
- accessibility_checks
nightly:
run:
- full_ui_regression
- cross_browser_suite
- visual_regression
- performance_baseline
production_deploy:
run:
- production_safe_smoke
- monitor_error_rate
- monitor_key_funnelsThis is a starting point, not a finished answer. After a few releases you will move things around, because you will see which checks actually catch bugs on pull requests and which ones only ever fail at three in the morning for infrastructure reasons.
That is the right way to use it. The point is to be deliberate about placement instead of dumping everything into one stage and hoping.
Keep pull request feedback fast
Pull request checks have one job: tell the author quickly whether the change is safe to merge, while they still have the change in their head. The number that decides whether they succeed is wall-clock time.
This is exactly what turned that weekly release train around. On the promotions service behind that platform, only a deliberately chosen, stable subset ran on the per-merge-request gate. The heavier and less reliable tests ran later, on the main branch and nightly, where their cost was paid less often.
The reason was not only speed, it was trust. One flaky test on the pull request gate teaches the whole team to wave red builds through, and once people learn to ignore a red gate, the gate is worthless.
We kept one escape hatch for genuine infrastructure blips: retrying a stuck pipeline by commenting retry on the merge request, instead of re-running everything by hand.
Fast, stable, and trusted beats comprehensive and ignored.
If pull request (PR) checks take forty-five minutes, the author starts something else, context switches away, and by the time the result comes back the feedback has lost its bite. The test ran, but the teaching moment is gone.
Aim to keep the pull request stage in the few-minutes range, and protect that budget. A good PR stage usually holds:
- linting and type checking
- unit tests
- API and contract tests
- component tests
- a small set of genuinely reliable UI smoke tests
Resist the urge to run the entire UI regression suite on every pull request. Unless your product is small and that suite is rock solid, it will be the thing that makes PR feedback slow and noisy, which is the opposite of what this stage is for.
Push the heavy regression into the main branch and nightly stages where its cost is paid less often.
Put each test at the level where it is cheapest and clearest
The layered automation model is what makes a pipeline fast, and the API and microservices testing guide is where I lay out the full pyramid and why the middle layer pays back the most.
In CI the rule is short: if a behaviour can be checked reliably at the unit or API level, do not drag it into a browser just because the UI is what the user sees.
| Behaviour | Where to test it |
|---|---|
| Discount calculation logic | unit or service test |
| Invalid coupon returns the right API error | API test |
| User sees the coupon error on the checkout page | UI test |
| The full checkout path still works end to end | one end-to-end smoke test |
The difference shows up the moment something breaks. When a unit test for the discount math goes red, you know where to look.
When an end-to-end checkout test fails at step twelve, you get to figure out whether it was the math, the API, the page, the test data, or the network before you have even found the bug. Same coverage, very different debugging cost.
Pushing checks down to the cheaper layers is one of the highest-leverage moves you can make in a pipeline.
Build a smoke suite people actually trust
A smoke suite answers exactly one question:
Is the product basically alive in the ways that matter most?
For an e-commerce product, that is usually:
- a user can log in
- a user can search or browse products
- a user can add an item to the cart
- a user can complete checkout with a test payment method
- the order confirmation appears
For a software as a service (SaaS) product:
- a user can log in
- the dashboard loads
- a user can create the core object
- a user can edit and save it
- a user can invite or share, if that is critical to the product
Keep this suite small, stable, and fast. Five to fifteen tests that almost never lie are worth far more than fifty that go red on a flake once a week, because the value of a smoke suite is entirely in its trustworthiness.
The first time it cries wolf and the team finds nothing wrong, its credibility starts to drain, and once people are routinely overriding it, you have lost the one signal that was supposed to be unambiguous. Earning that trust back takes real work, so guard it from the start.
Treat flaky tests as product bugs, not background noise
A flaky test is one that passes and fails without the code changing, and it is corrosive in a way that is easy to underestimate. Every flaky failure forces the team to stop and ask “is this real?”, and the honest answer is usually “probably not, just rerun it”.
Do that enough times and you have trained everyone to ignore red, which means the day a flaky test masks a real regression, it sails straight through.
This is not a small-team problem you will grow out of. Flakiness is one of the most-studied problems in automated testing, and the finding that holds everywhere is the same: larger, browser-driven tests are dramatically flakier than small unit tests, and at scale a steady trickle of false failures starts to swamp the genuine ones.
The lesson is not that flakiness is inevitable. It is that you have to manage it on purpose, with a policy, the same way you manage defects.
A flaky test policy can be this simple:
If a test fails twice for non-product reasons:
1. Create a ticket and assign an owner.
2. Tag the test as flaky.
3. Remove it from blocking the release if it is destabilising the pipeline.
4. Fix it or delete it within a defined window.
5. Record the root cause in a flaky-test log.The two rules that matter most are the owner and the window. A flaky test with no owner is a flaky test forever, and a quarantine with no deadline is just a quieter way of giving up.
Most flakes trace back to a short list of usual suspects:
- waiting for a fixed amount of time instead of for the actual state you need
- tests sharing data or running in an order they secretly depend on
- selectors tied to styling instead of behaviour, which break on any redesign
- animations, timing, and race conditions
- an unstable environment or a real external service in the loop
Most of these come down to locator strategy and waiting done right. Tie your selectors to behaviour rather than styling, and wait for the actual state you need instead of a fixed delay, and the bulk of your flakiness disappears.
Make test data part of the pipeline design
A surprising share of CI/CD testing pain is not a testing problem at all. It is a test data problem. Tests fail in parallel because they fight over the same account. Cleanup half-runs and leaves the next run in a weird state. The nightly suite quietly depends on data someone seeded last Tuesday.
Before you blame the tests, ask:
- Does each test create the data it needs?
- Can tests run in parallel without colliding?
- Is cleanup reliable, or does it leave debris?
- Are test accounts shared in ways that cause races?
- Can a test safely run against production?
- Does the pipeline secretly depend on yesterday’s data?
The fix is the same idea in every case: each test should own its setup, either by creating its own data through an API or by using clearly managed, isolated seed data.
Here is the pattern in a Playwright test, where the test builds the user it needs through the API before it ever opens the browser:
import { test, expect } from '@playwright/test'
import { createTestUser, login } from './helpers'
// Vendly (a made-up sample app): a meal-delivery checkout, so the test owns the user it needs
test('verify that a Vendly user can update their delivery address', async ({ request, page }) => {
const user = await createTestUser(request)
await login(page, user)
await page.goto('/account/delivery')
await page.getByLabel('Address line 1').fill('15 Hope Road')
await page.getByRole('button', { name: 'Save address' }).click()
await expect(page.getByText('Address saved')).toBeVisible()
})The tool is not the point. The point is that the test does not assume a user already exists, does not depend on another test having run first, and leaves nothing behind that the next run trips over.
A suite built that way parallelises cleanly, which is most of how you keep CI fast as it grows.
Test in production, but carefully
Production testing is genuinely valuable, because a green pipeline tells you the build was good, not that the deploy worked.
The catch is that production is full of real users, real money, and real emails, so the rules are stricter. Production checks I trust:
- run as clearly tagged test accounts
- avoid destructive actions entirely
- never send real customer emails or notifications
- never make real purchases unless they use an approved test payment method
- can be disabled instantly if they misbehave
- are monitored separately from customer traffic
A production smoke test exists to confirm the deployed system is alive and the critical paths work, paired with monitoring on error rates and key funnels.
It is not the place to re-run your whole QA suite against live infrastructure. Keep it small, safe, and reversible.
Add quality gates with judgment
A quality gate earns its keep when it stops a genuinely risky change from reaching users. It turns into a tax when it blocks delivery for reasons that do not map to real risk, and a gate that blocks for silly reasons gets routed around, which defeats the purpose.
The line between the two is whether the gate is tied to actual risk.
| Gates worth enforcing | Gates that just slow people down |
|---|---|
| no critical security or data-loss defects open | every test must pass, flaky ones included |
| the smoke suite passes | zero bugs of any severity allowed |
| migrations tested against production-like data | the test count must grow every sprint |
| crash rate below an agreed threshold | a 100% pass rate, with no one checking the tests matter |
| a rollback plan exists for the release | a coverage number hit, regardless of what it covers |
The gates on the right all share a flaw: they measure activity instead of risk, which is the same trap behind a lot of QA metrics that mislead teams.
Match the gate to the risk, and it protects you. Match it to a vanity number, and it just teaches people to game it.
A note on AI in the pipeline
Be clear-eyed about where AI fits, because in 2026 a lot of the code flowing through these pipelines is AI-assisted, and a lot of the tests are too. AI raises output, not judgment, which is why QA becomes the control layer for AI-assisted development.
DORA’s 2024 State of DevOps report found that AI adoption lifts individual productivity and satisfaction, but that without strong fundamentals it can actually hurt delivery stability and throughput.
AI lets people produce more change, faster. A continuous testing pipeline is exactly the safety net that turns that extra velocity into shipped value instead of shipped regressions.
The fundamentals do not go away when AI shows up. They matter more.
If your team is leaning on generated tests, make reviewing them a real step, the way I lay out in how to review AI-generated automated tests.
Where to start
Continuous testing, done well, should make releases calmer, not louder. It moves the scary moment from the night before the release to a series of small, early signals that the team actually believes.
If your pipeline today is slow, flaky, or stuffed with checks nobody reads, adding more tests will make it worse, not better.
So start small and deliberate, and give each move an owner and a finish line so it is a commitment rather than a wish:
- Owner: QA. Write down which tests run at which stage. Done when: the test selection model exists in the repo and the team has agreed to it.
- Owner: Dev. Get the pull request stage fast. Done when: PR feedback lands in the few-minutes range and the heavy regression has moved to main and nightly.
- Owner: QA. Build a smoke suite you would stake a release on. Done when: five to fifteen tests cover the critical paths and have not cried wolf in a month of runs.
- Owner: EM. Make the flaky-test policy real. Done when: every flake has an owner and a fix-by date, and nothing flaky is allowed to gate a merge.
Do those things and the pipeline starts working for you, which is the whole point of doing it continuously in the first place.
Sources
Written by
Julia Pottinger
Hi, I'm Julia. I've been in QA for over a decade. I spend my days testing software and my own time building apps and games, and I write here to share what I learn, the practical, honest lessons you can actually use.
Comments 0
Share your thoughts, ask questions, or add to the conversation.