Use both, for different jobs. A single retry is a diagnostic: it tells you whether a failure reproduces on the same commit. Quarantine is containment for a test you already know is flaky, and it only works when every quarantined test has an owner and a date by which it is fixed or deleted.
A retry answers one question
A flaky test is a test that passes and fails on the same code with no change in between. Its failures look like real regressions until someone checks. At Google, about 1.5% of all test runs reported a flaky result [Google Testing Blog], and about 84% of the pass-to-fail transitions they observed involved a flaky test [Google Testing Blog]. When most transitions from green to red are noise, a cheap way to separate noise from signal is worth having.
A retry is that cheap check. Run the failed test again, on the same commit, in the same environment. If it fails again, you have a reproducible failure and it goes to triage. If it passes, you have learned a fact: this test is non-deterministic. That fact is useful. It should be recorded, not thrown away with the red result.
Silent retries hide the signal
The failure mode of retries is a green build that nobody questions. A pipeline that retries until it passes and reports only the final result keeps the flaky test in the suite and moves the cost somewhere less visible. At Atlassian, approximately 15% of Jira backend repo failures are attributed to flaky tests, and the reruns they require waste over 150,000 hours of developer time each year [Atlassian]. Reruns are not free, even when they turn the build green.
Keep retries narrow:
- Retry a failed test at most once. More retries only lower the chance that you see the problem.
- Record both results. A fail followed by a pass marks the test as flaky, even if the build continues.
- Report the retry in the pull request, so the author and the reviewer both see it.
- Do not raise the retry count to get a pipeline through a release. That turns a known problem into an unknown one.
Quarantine contains a known problem
Quarantine means the test still runs, but its result no longer decides whether a change can merge. It is the right move once a test is confirmed flaky and a fix is not immediate. The rest of the suite stays a trustworthy gate, and the flaky test keeps producing data about how often it flips.
This works at scale. Slack replaced a manual approach to handling test failures with an automated flaky test suppression system and dropped test job failures from 57% to less than 5% [Engineering at Slack]. The gain came from handling known flaky tests by a consistent rule instead of by hand, one failure at a time.
Quarantine has a cost, and it should be stated plainly. A quarantined test protects nothing. A real regression in that flow will pass review while the test sits in quarantine. That is why the exit rules matter more than the entry rule.
Quarantine needs an owner and an exit date
A quarantine list without rules grows until it becomes a second, silent suite. Four rules keep it honest:
- Every quarantined test has a named owner, usually the team that owns the flow.
- Every quarantined test has a date by which it is fixed or deleted.
- The list is visible and reviewed on a fixed cadence, for example at each planning meeting.
- A test that passes its date is deleted. A test that nobody fixes is not protecting a flow, and keeping it only adds noise.
Fixing is often tractable. In a study of commits that fix flaky tests in open-source projects, the top three categories were Async Wait, Concurrency and Test Order Dependency [University of Illinois at Urbana-Champaign / ACM FSE]. Waits and ordering are usually under the test author's control. What causes flaky tests goes through each cause, and how to reduce flaky tests covers the fixes.
The procedure in practice
Put the two tools in sequence, so each one does the job it is good at:
- A test fails in continuous integration (CI). Retry it once.
- It fails again: treat it as a real failure and triage it. Do not quarantine a test that fails consistently; that is a broken test or a broken product.
- It passes on the retry: the build may continue, but the test is flagged as flaky and the flag is visible on the pull request.
- It is flagged again within a short window: quarantine it, assign an owner, set a date.
- The owner fixes the cause or deletes the test before the date.
Where the gate lives is a team decision. On GitHub, once required status checks are enabled, all required checks must pass before collaborators can merge into the protected branch [GitHub]. Keep quarantined tests out of the required checks and keep everything else in. How to detect flaky tests in CI covers how to find candidates before they reach the quarantine list.
How AI Testing Platform fits
In AI Testing Platform, a failed test gets a triage card with the failing step, the page state, the diff from the last passing run and a proposed classification: app bug, test issue or environment. The classification is a proposal, and the decision to retry, quarantine or fix stays with the team (failure triage, triage docs). When a step fails because an element moved or was renamed, the platform heals it and records the change for review, while a changed behaviour still fails the test. The platform reports a check on the pull request but does not block merges by itself; branch protection remains your rule.