Detect flaky tests from data, not from memory. Keep a result for every test on every continuous integration (CI) run, rerun each failure once on the same commit, and flag any test whose outcome changes without a code change. A test that flips between pass and fail on the same commit is flaky by definition, and history shows you which tests do it most.
A flaky test has a precise definition
A flaky test is a test that passes and fails on the same code. The definition matters because it gives you a test you can run automatically: same commit, same environment, different outcome. Everything else, such as a test that fails on every run after a change, is a real failure or a broken test, and it needs a different response.
Flakiness is common enough that you should assume you have it. Almost 16% of Google's tests have some level of flakiness [Google Testing Blog]. At Atlassian, test flakiness was responsible for as much as 21% of master build failures in the Jira Frontend repository [Atlassian]. If nobody on your team is measuring it, it is still there, spread across reruns and ignored red builds.
Keep per-test results for every run
Pipeline status is too coarse. A red build tells you that something failed; it does not tell you which test, or whether that test failed yesterday on a different change. Detection starts with storing one row per test per run: test name, commit, result, duration and the run it belongs to.
Many CI systems can collect this if your runner writes a standard report file, usually in the JUnit XML format (a common test report layout written in Extensible Markup Language):
- GitLab unit test reports show test results directly in merge requests and pipeline details, and require the JUnit XML format [GitLab].
- CircleCI collects data from XML files saved with the
store_test_resultsstep and uses it for insights such as the Tests tab [CircleCI]. - In Jenkins, the
junitstep from the JUnit plugin captures and processes test reports [Jenkins].
These views are a starting point. For flakiness you also need history across runs, so export the results somewhere you can query, even a simple table.
Rerun each failure once on the same commit
A single rerun turns a guess into a measurement. When a test fails, run it again on the same commit in the same environment and record both results.
- Fail, then fail: a reproducible failure. Triage it as a real problem.
- Fail, then pass: a flip. Record it against the test, even if the build ends green.
Do not rerun more than once. Each extra rerun lowers the chance you see the flip, and a hidden flip is the thing you are trying to detect. Retry or quarantine flaky tests covers what to do once a test is flagged.
For every failure, also compare against the test's last passing run. The useful diff covers the page, the network calls and the step that failed. If nothing relevant changed and the test passed on rerun, the failure is flaky. If the page or a response changed, you are looking at a real change, in the product or in the environment.
Flag tests by flip rate over a window
One flip can be bad luck. A pattern is a signal. Over a rolling window of runs, compute for each test how often its result flipped on a rerun or changed between consecutive runs of the same commit. Sort by that rate and review the top of the list regularly.
Useful fields next to the rate:
- the last commit where the test flipped,
- the step where it failed, if your runner reports steps,
- whether the failure was a timeout.
Timeouts deserve a separate column. In a study of flaky test fixes, many Async Wait flaky tests (54%) were fixed using waitFor [University of Illinois at Urbana-Champaign / ACM FSE]. A test that flips on a timeout is often waiting on the wrong condition, which is a test fix, not a product fix. What causes flaky tests goes deeper on the causes, and fixing flaky Playwright tests covers one framework in detail.
How AI Testing Platform fits
In AI Testing Platform, each failed test gets a triage card with the failing step, a screenshot, the page state, the network log and the diff from the last passing run (triage docs). It proposes a classification of app bug, test issue or environment, and the team decides (failure triage). Results for each run are also written to .aitp/last-run.json, and npx aitp report --last opens the last run report. The CircleCI integration is available and the Jenkins integration is in beta.