To reduce flaky tests, measure which tests flake, quarantine them with a named owner so they stop blocking merges, fix the root cause starting with timing and waits, and remove shared state between tests. Retries alone do not reduce flakiness. They hide it, so use them to detect flaky tests, not to close the problem.
Flaky tests are a measurable cost
A flaky test is a test that passes and fails on the same code with no change in between. It is easy to treat as a nuisance. The numbers from large engineering teams say otherwise.
Across Google's entire corpus of tests, there is a continual rate of about 1.5% of all test runs reporting a flaky result [Google Testing Blog]. About 84% of the pass-to-fail transitions Google observes involve a flaky test [Google Testing Blog]. In other words, most "this used to pass" signals were noise.
Atlassian reports the same pattern. Test flakiness was responsible for as much as 21% of master build failures in the Jira Frontend repository [Atlassian]. Approximately 15% of Jira backend repo failures are attributed to flaky tests, necessitating reruns that waste over 150,000 hours of developer time each year [Atlassian].
The cost is not only compute. Every flaky failure trains engineers to ignore red builds, and that habit is what lets real regressions through. The wider maintenance cost is covered in E2E test maintenance cost.
Measure flakiness before fixing it
You cannot reduce what you do not count. Start by recording, for every test, how often it failed and then passed on the same commit. A test that passes on a rerun without any code change is flaky by definition.
Keep the record per test, not per suite. A suite with one very unstable test needs a different response from a suite with many slightly unstable ones. Sort the list by how many pull requests each test blocked. That order is your work queue. How to collect the signal in continuous integration (CI) is described in detect flaky tests in CI.
Quarantine with an owner and a deadline
A known flaky test should stop blocking merges while it is being fixed. Quarantine means it still runs and still reports, but its result does not fail the build.
Automated quarantine works at scale. After replacing the manual approach of handling test failures with an automated flaky test suppression system, Slack dropped test job failures from 57% to less than 5% [Engineering at Slack].
Quarantine has one risk: it becomes permanent. Our recommendation is to give every quarantined test a named owner and a review date. If it is not fixed by then, delete it or rewrite it. A test that never counts is not protecting anything. The trade-off between the two approaches is discussed in retry or quarantine flaky tests.
Fix async waits first
Most flakiness has a small number of causes. In a study of 201 commits that likely fix flaky tests in 51 open-source projects, the top three categories of flaky tests are Async Wait, Concurrency, and Test Order Dependency [University of Illinois at Urbana-Champaign / ACM FSE].
Async Wait is the one to start with in end-to-end (E2E) tests. It happens when a test assumes an operation has finished, such as a page load, a network call or an animation, without checking. The fix is usually to wait for a condition instead of a fixed time. Many Async Wait flaky tests (54%) are fixed using waitFor [University of Illinois at Urbana-Champaign / ACM FSE].
In practice:
- Replace every fixed sleep with a wait on something visible, such as a heading, a button becoming enabled or a response arriving.
- Assert on the final state the user sees, not on an intermediate state.
- Give slow environments a longer timeout on the condition, not a longer sleep.
Remove shared state between tests
Order dependence and concurrency come from shared state. One test logs in and leaves a session behind. Another creates a record that a third test expects not to exist. Run them in a different order, or in parallel, and results change.
Each test should create the data it needs and not depend on what another test left behind. Use unique names for records created during a run. Reset or isolate browser state between tests. When two tests must share an account, make them run in sequence on purpose, and write down why. More causes are listed in what causes flaky tests.
How AI Testing Platform helps
AI Testing Platform gives every failed test a triage card with the failing step, the page state, the diff from the last passing run and a proposed classification: app bug, test issue or environment. The classification is a proposal, and a person confirms it. Locator and layout changes are healed and recorded for review, which removes one source of false failures, while a changed behaviour still fails the test. See the triage docs for the card fields.