How to Fix Flaky Tests: From 18% to Under 3% in a Cypress Suite

A test suite is only useful if people believe it. On one project, about 18% of our automated test runs failed for reasons that had nothing to do with the code being tested. Re-running the pipeline usually turned it green, so that's what everyone learned to do. The problem with a suite that cries wolf is that eventually a real bug fails a test, someone clicks "re-run", and the bug ships.

Over the following months we brought that rate under 3% on a Cypress suite covering more than 300 critical user flows. None of the fixes were exotic. This article covers what flaky tests are, how to find them, the root causes behind almost all of them, and the specific changes that made the difference.

What is a flaky test?

A flaky test is a test that both passes and fails on the same code, without any change to the application or the test. Run it ten times on one commit and you get, say, eight passes and two failures.

That definition matters, because it rules out two things people often call "flaky":

• A test that fails consistently is not flaky. It's either catching a bug or it's broken.
• A test that fails because the environment was down is an infrastructure problem, even if the symptom looks the same.

Why flaky tests are so expensive

The cost isn't the re-runs. It's what they do to the team's behaviour:

• Red builds stop meaning anything. Engineers assume a failure is "just flakiness" and re-run, which is exactly when real regressions slip through.
• Feedback slows down. Every re-run adds minutes to every pull request, every day.
• Automation loses support. When a suite is unreliable, people start arguing for less automation instead of better automation.

Fixing flakiness is less about the tests and more about restoring trust in the pipeline.

Step 1: Measure before you fix

You can't fix what you can't see, and "the suite feels flaky" isn't something you can prioritise. Start by making flakiness measurable:

• Re-run failures on the same commit. If a test fails and then passes with no code change, log it as a flaky result. CI retries make this easy: a test that passed on retry is, by definition, flaky.
• Track results per test, not per build. A build-level pass rate hides the fact that most flakiness usually comes from a small number of tests.
• Define a flaky rate. We used: runs where at least one test passed only after a retry, divided by total runs. That gave us the 18% starting point and a number to drive down.
• Stress-test suspects. Run a single spec many times in a row. In Playwright, the built-in --repeat-each=20 flag does this; in Cypress, loop the spec in a CI job or script.

The result is a ranked list of offenders. Fixing the top ten usually removes most of the noise.

The root causes of flaky tests

Almost every flaky test I've investigated falls into one of these categories:

The fixes below map directly to these causes.

Fix 1: Use stable selectors

Selectors tied to layout or styling break whenever the page changes, and they often match a different element while the page is still rendering. We agreed with the developers on dedicated test attributes and treated them as a contract: they don't change unless the feature does.

// Brittle: depends on layout, breaks when a wrapper div is added
cy.get(".header > div:nth-child(2) button").click();

// Stable: a dedicated attribute that only exists for tests
cy.get("[data-cy=checkout-submit]").click();

Prefer, in order: dedicated test attributes, accessible roles and labels, then visible text. Avoid CSS classes, element positions and generated IDs.

Fix 2: Replace fixed waits with assertions

Fixed waits are the single most common cause of flakiness. cy.wait(2000) is too long on a fast run and too short on a slow one, so it makes the suite both slower and less reliable.

Instead, wait for the thing you actually depend on: a network response or a visible state.

// Flaky: hopes the orders have loaded after two seconds
cy.visit("/orders");
cy.wait(2000);
cy.get("[data-cy=order-row]").should("have.length", 3);

// Reliable: waits for the API call, then for the UI to reflect it
cy.intercept("GET", "/api/orders*").as("getOrders");
cy.visit("/orders");
cy.wait("@getOrders");
cy.get("[data-cy=order-row]").should("have.length", 3);

Cypress retries assertions like should("have.length", 3) automatically until they pass or time out. Lean on that: assert on the final state you expect, and let the framework handle the timing.

Fix 3: Make every test independent

If a test only passes when another test ran first, it will fail as soon as the order changes, tests run in parallel, or someone runs it on its own. Each test should create the state it needs and nothing else.

Logging in through the UI before every test is slow and adds another chance to fail. We moved login into a custom command that authenticates through the API and caches the session:

// cypress/support/commands.ts
Cypress.Commands.add("login", (email: string, password: string) => {
  cy.session([email, password], () => {
    cy.request("POST", "/api/auth/login", { email, password });
  });
});

// In a test
beforeEach(() => {
  cy.login("qa.user@example.com", Cypress.env("QA_PASSWORD"));
});

The same principle applies to test data: create it through the API in the test that needs it, and give it unique values so parallel runs never collide.

const email = `qa+${Date.now()}-${Cypress._.random(1e6)}@example.com`;
cy.request("POST", "/api/test-support/users", { email, plan: "premium" });

Fix 4: Control third-party services

Payment providers, email services and analytics scripts are outside your control, and their sandboxes are often slower and less reliable than production. For most end-to-end tests, stub them and test the integration separately in a smaller, dedicated suite.

cy.intercept("POST", "**/payments/charge", {
  statusCode: 200,
  fixture: "charge-success.json",
}).as("charge");

Stubbing also lets you test the failure paths that are hard to trigger for real, such as a declined card or a timeout.

Fix 5: Retries are a safety net, not a cure

Automatic retries hide flakiness instead of fixing it, but a small number of them keeps the pipeline usable while you work through the backlog. We enabled them in CI only:

// cypress.config.ts
import { defineConfig } from "cypress";

export default defineConfig({
  retries: { runMode: 2, openMode: 0 },
});

The rule that made this work: any test that needed a retry got a ticket, with an owner. Retries kept the pipeline green; the tickets made sure the root cause was still fixed. Without that rule, retries simply turn an 18% problem into an invisible one.

How to keep flakiness from coming back

Getting under 3% was one project. Staying there needed a few habits:

• Run the full suite on every pull request. We ran it through Jenkins and GitHub Actions, so a newly flaky test showed up within a day of being introduced, while the change was still fresh.
• Quarantine, don't delete. A flaky test moves to a separate job that doesn't block merges, gets an owner and a deadline, and comes back once it's fixed.
• Review tests like production code. Fixed waits, layout-based selectors and shared state are easy to catch in code review, and much cheaper to fix there.
• Watch the trend. Keep the flaky rate visible on a dashboard. A rising number is an early warning that the suite or the application is drifting.

The results

The most important change doesn't fit in the table: once a red build reliably meant a real problem, people stopped re-running pipelines and started reading the failures.

Frequently asked questions

What causes flaky tests?

Most flaky tests are caused by timing and asynchronous behaviour, brittle selectors, state shared between tests, colliding test data, unreliable third-party services, or differences between local and CI environments. Timing problems, especially fixed waits, are the most common.

How do you identify flaky tests?

Re-run failing tests on the same commit and record which ones pass on a retry, track results per test over time, and stress-test suspicious tests by running them many times in a row. A test that both passes and fails on unchanged code is flaky.

Should you retry flaky tests automatically?

A small number of automatic retries in CI is reasonable while you fix the underlying problems, but retries should never be the fix. Track every test that needs a retry and fix its root cause, otherwise retries hide real failures.

What is an acceptable flaky test rate?

There is no universal standard, but the goal should be that a failing build almost always means a real problem. In practice, keeping the rate in the low single digits and falling is a good target; anything higher and the team will start ignoring failures.

Key takeaways

• A flaky test passes and fails on the same code. Measure it per test before fixing anything.
• Most flakiness comes from timing, selectors, shared state, test data and third-party services.
• Replace fixed waits with assertions on network responses and visible state.
• Use dedicated test attributes as a stable contract with developers.
• Make every test independent: API login, API-created data, unique values.
• Use retries only as a safety net, and open a ticket for every test that needs one.
• Run the suite on every pull request so new flakiness is caught while it's cheap to fix.

Flaky tests often start with weak test design. For choosing what to test in the first place, read my guide to test case design techniques.

Is your team re-running pipelines more than it should? Let's talk.