Yesterday’s red was the suite at its most honourable; today is about the reds that are not, because a suite is not a monument, it is a garden, and ungardened suites die inside a year. Flake first. A flaky test changes its answer with no code change: green at ten, red at noon, green on the rerun. You built the habits against the two big causes already, honest waits on Day 9 and isolation on Day 10, but flake also creeps in through shared environments, live data, and plain entropy, so you need a policy for it, not just prevention.
Retries: useful, and dangerous
Playwright will rerun a failed test for you:
// playwright.config.js
import { defineConfig } from '@playwright/test';
export default defineConfig({
retries: 1,
});
One retry is reasonable in CI, where a network hiccup can fail an honest test. The danger is in how you read the result. A test that fails and then passes shows up in the report labelled flaky, and the run ends green. Teams see the green and move on, and that is how retries hide truth. A pass on retry is a fail that got lucky. Read the flaky label as a to-do with a deadline, not as a pass. When a suite needs retries to go green routinely, stop adding tests and fix the suite, because every new test is landing on a rotten floor.
Quarantine, with a name on it
You met the pattern on Day 12: a test that is red for a known, filed reason gets test.fixme with the ticket in a comment. The rules that keep quarantine honest:
- Every quarantined test carries a ticket and an owner. No ticket, no quarantine.
- Quarantine is a visa, not citizenship. Review the fixme list weekly; anything sitting for a month without movement gets escalated or deleted.
- Quarantine is for known product bugs and confirmed flake under investigation. It is never a place to hide a test you do not understand.
Deleting tests is maintenance, not defeat
Some tests stop justifying their run time, and the honest move is deletion:
- A test that has never failed and asserts something no other test would miss. It is spending minutes nightly to prove nothing.
- A test duplicating a check that now lives in a cheaper layer, like the feed test from Day 11 covering what three UI tests poke at slowly.
- A test for a feature that no longer exists, kept out of sentiment.
Deleting a bad test raises the value of every remaining green, because the suite gets faster and its answers get sharper.
The three health numbers
Track these over time, and you will see trouble months before the team feels it:
- Pass rate. Falling means product trouble or suite rot; either deserves attention.
- Retry rate. How often a run needed a retry to go green. This is your flake gauge, and it climbs quietly.
- Duration. Creeping duration pushes the suite out of the loop developers actually wait for.
When retries climb, stop adding tests and fix the suite. New tests on a rotten floor just add weight.
Do this today
Set retries to 1 in your config and rerun the suite, checking the report for any flaky labels; with Day 9 and Day 10 habits, you should find none. Then write down your suite’s three health numbers from yesterday’s runs as the baseline you will compare against. Tomorrow your suite stops waiting for you to remember it: CI.