The Hidden Economics of Flaky Tests

Fixing flaky tests is 90% of the effort. The last 10% is preventing them from being written again.

I’ve been binge-watching For All Mankind, the Apple TV show that imagines what would have happened if the global space race had never ended after the Soviet Union landed on the moon first. As someone who went to college planning to be an aeronautical engineer before switching to computer science major, I keep finding these fascinating cross-sections between aerospace and software. The show got me thinking about one particular parallel between the two industries, specifically the attitude both have around test failure. (In one episode, Apollo 24’s flight control computer fails during launch, and instead of just retrying the mission, NASA grounds everything, investigates the root cause, replaces the faulty component, and only then proceeds.)

Generally, in aerospace, a test failure means someone investigates until they understand the root cause. Standards like DO-178C require 100% code coverage with documented justification for any untested code. Non-deterministic test behavior is a certification blocker, not a backlog item.

In software, we’ve built an entire culture around ignoring flaky tests. I bet there is an xkcd comic around it somewhere. We rerun CI pipelines like it’s free. We disable tests that fail intermittently. We’ve normalized the idea that some percentage of our test suite is just noise, which is a polite way of saying we’ve collectively agreed to lie to ourselves about whether our code works.

I wanted to think through this openly here. What if we treated flaky tests not as inevitable annoyances but as economic decisions? What if we stopped being reactive (fixing individual tests) and started being proactive (preventing patterns from recurring)?

We’re making an economic decision every time we rerun a flaky test, and we’re making it blindly.

Roughly speaking, a single flaky test that fails 10% of the time costs a five-person team $450 per month (a very conservative estimate) in rerun time, context switching, and flow disruption. Research from Google found that flaky tests cost companies over 2% of coding time for a 50-person team, that’s an entire person-year lost annually to dealing with unreliable tests. Most teams have dozens of known flaky tests, which adds up to $300,000 per year in engineering overhead. This are very rough calculations and they’re just the rerun cost. The real damage is what happens to trust. When your test suite cries wolf often enough, people stop believing it. A real bug gets dismissed as “probably just flaky.”

Every flaky test presents an economic decision between fixing it, rewriting it, or deleting it. We make this decision constantly, using the most sophisticated algorithm known to engineering: whoever got annoyed by the flakiness that particular week gets to decide. That’s not a strategy, that’s just taking turns being mad.

In a presentation at the 2020 International Conference on Software Engineering, Microsoft researchers shared their experiences around a company-wide “fix or delete within two weeks” policy that reduced overall test flakiness by 18% in six months, gaining 2.5% developer productivity. But their policy was binary. They didn’t provide the economic framework for deciding which path to take.

I admit it’s not an easy decision tree to draw out. For example, the delete option is politically complicated and organizationally complex. Tests tied to compliance requirements, past incidents, or cross-team dependencies need broader approval than individual engineers can provide. You need organizational support to make these decisions, not just good math.

To me the formula for making that decision looks like this:

Ongoing cost = (CI reruns per week × team size × rerun time × hourly cost)

+ (false positives blockers × delay cost)

+ (context switching × flow disruption cost)

Fix cost = estimated debug hours × hourly cost

Rewrite cost = test rewrite hours × hourly cost

Delete cost = coverage gap risk × potential bug cost

If ongoing cost is $5,000 and fix cost is $500, you fix it. If fix cost is $3,000 and the test is duplicating coverage from three other tests, you delete it. If you’ve fixed it twice and it keeps coming back, the problem is probably the underlying code, not the test.

At a previous job, I started tracking every flaky test we fixed. Root cause, time spent debugging, whether it came back. After 30 flaky tests, patterns emerged.

40% were race conditions in async code, always fixed the same way by waiting for a specific condition instead of arbitrary timeouts. But we kept writing new tests with sleep() because it was faster in the moment.

25% were shared state between tests, where tests passed individually but failed when run together.

20% were environment dependencies, tests that worked locally but flaked in CI.

15% were actual production bugs, race conditions in real code that only surfaced under specific timing.

This taxonomy worked because the team had a relatively homogeneous tech stack and stable infrastructure. If your flaky tests come from third-party service outages or infrastructure issues outside your control, categorization gets harder. Sometimes the fix is different: better retries, circuit breakers, test isolation from external dependencies.

Once I had this taxonomy, the solution became clear. The team wasn’t facing 47 unique problems. We were facing five categories that manifested 47 times.

So I built a pre-commit hook that flagged patterns matching the flaky test taxonomy. If someone wrote a test with Thread.sleep() instead of the proper wait utility, the hook failed with a link to the three tickets where that pattern had caused flakiness before. A pretty simple proactive tooling solution.

This catches maybe 20% of flaky patterns, the obvious ones. Race conditions that only appear under CI load still get through, but catching 20% at code review is better than catching 0%.

What surprised me was that the hook also started catching bugs in production code. Those race conditions I found debugging flaky tests? They have signatures. The flaky test taxonomy became a code quality taxonomy. Win win!

Now, this was many years ago and everything I described took two months of disciplined tracking. Two months where I had organizational support to treat this as a priority instead of “work on features.” Admittedly that’s a luxury for most fast moving companies but, honestly, building systems requires organizational buy-in, time allocation, and cultural support. Individual engineers, no matter how experienced, don’t usually have that authority alone.

Today, AI assistance makes some of this easier, though. Tools like QMetry and LambdaTest are already using machine learning to calculate “flaky scores” and identify patterns in test execution history. But these tools are good at detection, finding tests that are already flaky. The leap to prediction, flagging new tests that will become flaky based on code patterns, is harder. That’s still research territory, not production-ready tooling. I give this 1-2 years before someone ships it, maybe less if the AI code assistant companies realize they’re sitting on gold.

The real opportunity is treating flaky tests as training data. Every flaky test you’ve ever fixed is a labeled example of fragile code patterns in your specific codebase. An LLM trained on your test history could flag suspicious patterns at code review time.

The pieces exist. CI/CD systems log every test run. The data is there. Someone needs to build the system that treats flaky tests as knowledge to extract rather than noise to ignore. (Psst! Please reach out if you’d like to geek out on this together)

The economic case is straightforward but even if you do the math, you still need to sell it to leadership, who wants features shipped, to product managers who see testing as overhead, and to teams who are already underwater.

So since I’m always yapping about “Step 1”, what’s that here →

Start small by picking your five flakiest tests, tracking them for a month, and calculating the actual cost.

Show the number to your manager and get buy-in for fixing those five with prejudice. Build the case iteratively instead of asking for two months upfront.

When it works, expand the taxonomy. When you have ten categories and prevention patterns, pitch the pre-commit hook. When you have data showing 20% prevention at code review, pitch the AI investment. Build credibility at each step.

This approach works best for teams with fairly established infrastructure and the authority to delete tests. It struggles when flakiness comes from outside your control, when you have a polyglot codebase, or when you lack organizational support for the initial investment.

Aerospace engineers don’t have flaky tests because they can’t afford them. Software engineers can afford them, but only if we’re making the fix/rewrite/delete decision with data instead of frustration. The answer isn’t to fix every flaky test the way aerospace does. It’s to build the knowledge graph of what makes tests flaky in your codebase, then use that knowledge to prevent the pattern from being written again.

Flaky tests as reactive fixes = 90% of the effort

Flaky tests as training data for prevention = the last 10%