Yes, run end-to-end (E2E) tests in continuous integration (CI) on every pull request, but not all of them. Run a small smoke suite that covers the flows you cannot ship broken, keep the full suite for after merge, and make the smoke run a required check only after it has proven stable.
Pull requests are where a failure is cheapest
An E2E test drives the application the way a user does: through a browser, against a deployed build, across every layer. When it fails on a pull request, the change that broke it is small, the author still has the context, and nothing has shipped. When the same failure appears after merge, someone has to bisect a batch of changes and find the author again.
That is the case for running on every pull request. The case against is time and noise, and both can be managed with scope.
Scope the pull request run to a smoke suite
A smoke suite is a short list of tests for the flows that must work in every release: sign in, the core action of the product, checkout or billing if there is one. It should finish while the author is still waiting for review, not after they have moved on.
A practical split:
- On every pull request: the smoke suite, run in parallel.
- After merge to the main branch: the full suite.
- On a schedule: the full suite across every browser you support.
Tags make the split cheap to maintain. Each test declares which suites it belongs to, and the pipeline selects by tag instead of by file list. How many E2E tests you need covers how to choose what goes into smoke.
Run the smoke suite against a deployment of the pull request itself when you have one. A shared staging server mixes changes from several branches, so a failure there may not belong to the change under test. E2E tests on Vercel preview deployments goes through that setup.
Stability comes before the merge gate
A required check that fails for no reason teaches people to ignore it or to click rerun. Flakiness is common enough to plan for: almost 16% of Google's tests have some level of flakiness [Google Testing Blog], and at Atlassian test flakiness was responsible for as much as 21% of master build failures in the Jira Frontend repository [Atlassian].
So add the check in two steps:
- Run the smoke suite on every pull request as an informational check. Watch it for a few weeks.
- Once its failures are real failures, make it required.
GitHub and GitLab both support this gate. On GitHub, after required status checks are enabled, all required checks must pass before collaborators can merge into the protected branch [GitHub]. On GitLab, a project can require a complete and successful pipeline before merge [GitLab]. Which checks are required is a team decision, and it should be written down.
Tests that turn out to be flaky leave the required set until they are fixed. Retry or quarantine flaky tests describes the rules for that, and detecting flaky tests in CI covers how to spot them early.
Report per test, not per pipeline
A single red pipeline status tells the author that something failed. It does not say which flow, which step or whether the failure is theirs. The pull request check should report each test by name with its own result, a link to the failing step and enough page state to decide without rerunning locally.
Two habits help:
- Keep test names readable. "Checkout with a saved card" is a better check name than a file path.
- Keep secrets out of the repository. Test credentials go into the CI secret store and are passed as environment variables.
A smoke run is one pipeline step
The pipeline step for the pull request run should be short enough to read in review. With AI Testing Platform, tests are plain-English steps in a YAML file (a plain-text configuration format) with optional tags, and one command runs a tagged suite in hosted browsers in parallel:
npx aitp run --tag smoke --cloudIn CI, the step looks like this, with the token and test credentials read from the secret store:
1- name: Run E2E
2 run: npx aitp run --tag smoke --cloud
3 env:
4 AITP_TOKEN: ${{ secrets.AITP_TOKEN }}
5 TEST_EMAIL: ${{ secrets.TEST_EMAIL }}
6 TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}
The exit code is 0 on pass, 1 on fail and 2 when healed steps wait for review. Healed means a step was re-found after an element moved or was renamed. Setting ci.failOnHealedPending: true in the config turns pending heals into a failing run (exit codes).
How AI Testing Platform fits
AI Testing Platform reports a pass, fail or healed status per test as a check on the pull request (CI integration). It does not block merges by itself: making the check required is your branch protection rule. Integrations are available for GitHub Actions and GitLab CI.