Flaky Test, How to Prevent your Test Suit

Flaky Tests A Flaky test is one that has a non-deterministic outcome: it can pass sometime and fail other, for the same code, running the same test.

Why Flaky Test occurs?

If the application interacts with browsers, external services or has asynchronous behaviour, probably there will be flaky tests.

Flaky Test are costly

Flaky test are waste of time, causing unnecessary build failure. Also, they are big waste of developer time, their failure doesn’t indicate a bug, but they do require attention. Due to this entire pipeline of code slows down

This also reduces confidence in the test suite, if these are not fixed in time.

Approach for fixing test flakiness

• Run unstable test separately Test that fails is proving it’s value unless it’s flaky. Once you discover the flaky test, you still want them to run, remove them from CI Pipeline and execute them in separate jobs to avoid the disruption, investigate it as a part of paying down technical debt. It should be investigated to see if it is providing any actual value.
• Test in Isolation Writing focused test which do single thing is one of the effective ways to avoid them being flaky. Test which does multiple things increases the chances of failure and can make the test non-deterministic and testing features and issue in isolation helps.
• Write Helpful Errors Writing the custom error will help in reporting identifying the issues quickly and help in segregating the flaky test failure from actual failure. ( definitely for long term identification)
• Control the Environment Regularly do test run on controlled environment, which will be same for current test and future test. This ensures test always start with the same conditions. This also ensures having the good CI process and able to recreate the environment from scratch when required which is the good development process.
• Fixing the Async Wait Many Flaky tests I came across were using sleep statement to do waiting such kind of flaky test were fixed replacing the sleep method with waitFor condition.

#testAutomation #pipelines #bestPractices #qualitykoder https://www.qualitykoder.com/flaky-test-how-to-prevent-your-test-suite/