Menu

Post image 1
Post image 2
1 / 2
0

We measured how much time our team wasted on flaky tests. The numbers were ugly

DEV Community·AGHL·27 days ago
#a7H9s3Ti
Reading 0:00
15s threshold

Like most teams using GitHub Actions, we’d gotten used to the ritual: push code, wait for CI, see a red build, re-run it, hope it passes this time. “It’s probably flaky” became the default response to any test failure — including real ones. We decided to actually measure the damage. Over 30 days on a single repo: 842 CI runs → 117 failures (13.9% failure rate) 31.5 developer hours spent investigating and re-running $426 in CI compute burned on re-runs that shouldn’t have been needed 1 regression shipped to production because a real failure was dismissed as “just flaky” The worst part? Nobody could tell us which tests were flaky. We had a vague sense — “that login test is weird” — but no actual inventory. And without an inventory, you can’t fix what you can’t see.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.

Read More