Flaky Tests Persist Because Everyone Is Ignoring Them Rationally

You have done everything right.

You made the economic case for automation and got the investment approved. You distributed quality checks across the SDLC instead of piling them at the end. You replaced pyramid thinking with risk-weighted coverage. You stopped reporting a coverage percentage that was lying to you.

Six months later, your engineers have started ignoring test failures.

Not because they are careless. Because ignoring test failures became the rational choice. This article is about how that happens, why it happens to teams that know better, and why it is the final form of Test Debt.

What is flakiness?

A flaky test is a test that fails intermittently without any change to the code it covers. It sometimes passes and sometimes fails, with no consistent pattern. The most common root causes are timing issues in async operations, test-order dependencies, shared mutable state, and coupling to external services.

All of these are fixable. The fixable nature of the problem is not what makes it interesting. What makes it interesting is that teams fix very little of it, and teams with strong engineers who care about quality fix very little of it.

The reason is not the technical difficulty.

The scale

The numbers are worth stating clearly, because they establish what is actually at stake here:

At Google, approximately 16% of tests show some form of flakiness, and 84% of transitions from passing to failing involve a flaky test rather than a genuine regression.

At Microsoft, roughly 25% of test failures in large-scale CI systems are caused by flakiness, not actual code defects. The average time a developer spends per flaky test investigation: 30 minutes, before determining it was not a real failure.

Atlassian estimated 150,000 developer hours per year consumed by flaky test investigation before they built automated detection tooling.

Slack's mobile test failure rate reached 56.76% before they intervened. More than half of all test failures were noise.

These are not teams with poor engineering culture. They are among the best-resourced and most technically sophisticated organisations in the industry.

The rational inaction problem

Here is the core insight this article is built around: flaky tests are the only common engineering quality problem where every person in the chain responds correctly given their local information, and the outcome is still catastrophic.

The developer who sees a test failure and hits retry is not being careless. In their experience, 85% of unexplained failures are environment issues, not regressions. Investigating would take 30 minutes and probably turn up a race condition in test infrastructure that is not their code. Retry is the rational choice.

The QA engineer who logs the failure as "likely flaky, monitoring for pattern" is not dropping the ball. Without detection tooling, they cannot quickly distinguish a flaky failure from a real regression. Flagging and watching is the rational choice.

The infrastructure team that says "it is a test quality issue, not infrastructure" is not deflecting. Flaky tests usually are a test quality issue. Declining scope is the rational choice.

The engineering manager who does not put flakiness remediation on the sprint roadmap is not mismanaging. There are always user-facing features with higher business visibility. Flakiness does not appear in the product backlog with an owner or a deadline. Deferring is the rational choice.

Nobody made a bad decision, but nobody fixed anything as well.

This is the organisational failure mode this series has been describing in different forms since Article 1: costs that are invisible and diffuse; don't get addressed, because the people experiencing the cost in small increments are never the people with authority to fix it at the root.

The rational inaction framing applies most clearly to mid-to-large engineering organisations with specialisation between roles. In small teams of fewer than 10 engineers, the same person often occupies multiple positions in the chain described above, which creates different incentive dynamics. The argument is strongest for teams of 20 or more where there is clear separation between developer, QA, infrastructure, and management roles.

Why retry is the most expensive button in your CI interface

When a team normalises retrying flaky tests, they make a subtle but load-bearing decision: test failures no longer reliably indicate real problems.

That decision has a half-life. Once a team accepts that some failures are noise, the threshold for investigating failures drifts upward. A suite with 5% flakiness earns mild skepticism. A suite with 20% flakiness is treated as broken infrastructure. A suite with 50% flakiness is effectively ignored.

Slack documented their mobile test failure rate reaching 56.76% before they intervened. At that point, the pipeline provides negative value. It adds latency to every deployment while producing no useful signal about regressions. You have paid to build, maintain, and run a suite that has become pure noise.

The connection to Article 1 in this series is direct: Article 1 described the cost of having no test automation. A flaky, untrusted test suite is a different path to the same destination. You are making deployment decisions without a reliable test signal. The investment from Articles 1 through 4 has been eroded to zero. You are still paying 100% of the maintenance cost and receiving close to 0% of the coverage value.

The accountability structure that actually works

The research on teams that have genuinely reduced flakiness shows a consistent pattern. The technical fixes vary by codebase and language. The organisational fix does not vary.

At Microsoft, flaky tests are automatically assigned to the developer who owns the affected code. Not a team. A named individual. Developers with more than 10 open flaky test bugs have their PR merges blocked until they close them.

At Atlassian, the approach combined three elements: automated detection (so no manual effort required to identify flaky tests), immediate quarantine (the test is removed from the blocking suite while investigation is open, keeping the pipeline trustworthy), and a named owner assigned within 24 hours of identification.

Neither of these is primarily a technical solution. Both are accountability structures. They answer the question "whose problem is this?" with a specific name and a specific timeline, rather than leaving it as a shared responsibility that falls on no one.

The practical framework, regardless of team size:

Make the number visible first. Pull your CI data for the last 30 days. Find every test that failed and then passed on retry. Express that as a percentage of total runs. Most teams have never seen this number. It is the most important signal in their pipeline, and it is not surfaced by default in any major CI tool.

Quarantine, do not delete. A flaky test usually covers real behaviour. Removing it from the blocking suite preserves pipeline signal while investigation is open. Deleting it removes coverage. These are different decisions with different consequences.

Assign by name, not by team. "The QA team owns flakiness remediation" produces the same outcome as no ownership. "Jason owns these four tests, resolution expected by end of sprint" produces a different outcome.

Set a flakiness budget. Define a maximum acceptable rate; 1% is a reasonable starting threshold, and treat it the same way you treat error rate or latency: a metric with a threshold, a dashboard, and a named owner who is accountable when it is breached.

Assumption: The 1% flakiness budget threshold is based on commonly cited industry practice rather than a formally derived number. Google and Microsoft operate internal flakiness thresholds but do not publicly document the specific percentages. Teams with very large test suites (10,000 or more tests) may need to calibrate differently based on acceptable investigation volume.

Why AI detection changes the equation but not the root cause

Automated flaky test detection, including AI-based pattern recognition across CI runs, removes the manual work that was the excuse for not addressing flakiness earlier.

Without detection tooling, asking a team to track flakiness is asking them to do manual bookkeeping across every CI run. Under any reasonable delivery pressure, this does not happen.

With detection tooling, the identification is automatic. The question of whether to address it becomes purely organisational: who owns it and what happens if they do not.

This is why detection tools are necessary but not sufficient. Teams that deploy automated flaky test detection without the accountability structure around it often end up with a dashboard showing 12% flakiness that nobody acts on. The bottleneck was never the detection. It was the ownership.

Closing the series

This series started with a single argument: the cost of not investing in testing is real, measurable, and invisible until it is catastrophic.

Each article has described a different mechanism through which testing investment either fails to materialise or silently loses value:

Article 1 - The investment does not happen because nobody has made the cost of not having it visible (Article 1).
Article 2 - The investment happens too late in the cycle because the tooling for real-time quality feedback was not in place when the policy changed.
Article 3 - The investment covers the wrong things because the pyramid model was a cost heuristic that got mistaken for a structural law.
Article 4 - The investment measures the wrong things because a coverage gate creates the incentive to pass the gate rather than catch the bugs.
And then in this article, we spoke about how the investment erodes silently because flakiness is everyone's problem in a diffused sense and nobody's problem in an accountable sense.

The through-line is not tooling. The through-line is that each of these problems persists because the cost is invisible and the accountability is diffused. The moment you make the cost visible and assign the accountability to a specific person, the problem becomes tractable. Not easy, not fast, but tractable.

Article 3 ended with the observation that deciding what is risky is still a human judgment. Every article in this series has made a version of that same point. The investment decision is human. The timing decision is human. The prioritisation decision is human. The measurement decision is human. The accountability decision is human.

The tools have never been the bottleneck.

What we are trying to do at QApilot is collapse the gap between 'the cost exists' and 'someone sees it and can act on it.' That is true for coverage gaps, for flakiness, for untested risk paths. The tooling is there. The visibility is what has been missing.