A flaky test is a test that passes and fails on the same code, with no change in between. The most common causes are asynchronous waits that assume timing, concurrency, and dependence on the order tests run in [University of Illinois at Urbana-Champaign / ACM FSE]. In end-to-end (E2E) suites, environment problems such as slow networks, unstable third-party services and shared test data are the next place to look.
A flaky test gives different results on the same code
The definition matters because it separates flakiness from other failures. If a test fails every time after a change, it is not flaky. It is either catching a regression or it is out of date. If it fails sometimes on an unchanged commit, it is flaky.
Flakiness is common even in mature engineering organisations. Almost 16% of Google's tests have some level of flakiness associated with them [Google Testing Blog]. Across Google's entire corpus of tests, there is a continual rate of about 1.5% of all test runs reporting a flaky result [Google Testing Blog].
Our recommendation: before fixing a flaky test, label its cause. The categories below each need a different fix, and guessing wastes time.
Async waits are the first place to look
A useful reference on causes is a research paper that studied real fixes. In a study of 201 commits that likely fix flaky tests in 51 open-source projects, the top three categories of flaky tests are Async Wait, Concurrency, and Test Order Dependency [University of Illinois at Urbana-Champaign / ACM FSE].
Async Wait means the test assumed an operation had finished when it had not. A page is still loading. A request has not returned. A button is visible but not yet enabled. On a fast machine the test passes. On a busy continuous integration (CI) runner it fails.
The signature is a failure on a step that reads or clicks something that appears a moment later. The usual fix is to wait for a condition, not for a fixed time. Many Async Wait flaky tests (54%) are fixed using waitFor [University of Illinois at Urbana-Champaign / ACM FSE]. Framework specifics are covered in fix flaky Playwright tests.
Concurrency and test order create hidden coupling
Concurrency flakiness appears when two things run at the same time and touch the same resource. Two tests edit the same user profile. A background job updates a record while a test reads it. The result depends on which one finishes first.
Test order dependency is the sequential version of the same problem. Test B passes only if test A ran first and left data or a session behind. Run B alone, or after a different test, and it fails.
The signature of both is that the failure moves when you change parallelism or order. A test that always passes alone and sometimes fails in the full suite is likely coupled to another test. The fix is isolation: each test creates its own data, uses unique names and does not rely on state left by others.
The environment is part of the test
E2E tests run against a real browser, a real server and often real third-party services. Each of those can fail on its own:
- A staging server is slow under load from other runs.
- A payment sandbox or email service returns an error.
- A shared test account was changed by a person.
- The clock crosses midnight, a month end or a time zone boundary during the run.
- The browser runs out of memory on a crowded runner.
The signature is a failure that clusters in time or across unrelated tests. If many tests fail in the same minute, look at the environment before looking at any single test. Collecting that signal is described in detect flaky tests in CI.
A changed locator is a broken test, not a flaky one
One failure often gets called flaky when it is not. A developer renames a button or moves a field, and a test that found the element by an old selector fails. That failure is deterministic. It will fail every time until the test is updated.
Treating it as flakiness leads to the wrong fix. Rerunning does not help, and quarantine hides a test that just needs updating. Healing (re-finding an element after its locator changed) addresses this case directly, as long as it is limited to how an element is found and never extends to what the product does. The limits are described in the healing docs. What to do once you know the cause is in how to reduce flaky tests.
How AI Testing Platform separates causes
AI Testing Platform gives each failed test a triage card with the failing step, a screenshot, a snapshot of the page structure, the network log and the diff from the last passing run, plus a proposed classification: app bug, test issue or environment. The classification is a proposal for a person to confirm. Locator and layout changes are healed and recorded, and a behaviour change fails the test. More on the card is on the failure triage page.