What is a flaky test?

It’s a test that sometimes fails, but if you retry it enough times, it passes, eventually.

Why flaky tests are a problem

Flaky tests are a problem for test suites because not only because they reduce the success rate of a test suite, but because they reduce confidence in the tests. The cost of this adds up quickly across an engineering team working on a project. A symptom of this is spending time re-running failed tests or incorrectly diagnosing a failed test as a flaky test.

What makes flaky tests hard to fix

Flaky tests produce a small signal over time, without a tool to track that signal it is incredibly hard to find a root problem or demonstrate progress. Sometimes this data gets captured as war-stories from veterans on the project or @retry-3 annotations around tests.

How TestRecall make this easier

Finding flaky tests should not require a superhero, and demonstrating increased stability of a test suite should be easy. TestRecall helps by tracking tests over time, making it easy to spot flaky tests - and measure the improvements you are making

flaky_test_statistics_img