How AI/ML helps to classify failure reasons automatically?
How many hours do you spend analyzing automated test results every day? For sure, this time can be substantially decreased by automating the process of failure reasons analysis.?
That’s why the Zebrunner team decided to implement an AI and Machine Learning-based mechanism that categorizes failure reasons by stack traces in seconds. The great thing is that every team can train AI/ML on their own tests and their unique stack traces, and increase the classification accuracy to up to 100%!
In this article, learn how Zebrunner’s AI/ML allows you to get rid of routine time-consuming tasks and prioritize your activities!
TYPES OF FAILURE TAGS
Our AI/ML automatically recognizes the failure reasons and classifies them according to the following types:
Example: Automation scenario fails due to the mismatch of requirements and the actual application behavior.
Recommendation: Review and analyze such failures with top priority by experienced manual QA engineers.
Example: XPath for the tested element was updated, so automation engineers should change it as well in their codebase.
Recommendation: Review this failure by automation engineers to define if it’s desired or not, and react accordingly.
Example: Selenium Hub-related failures, running out of HDD space, problems with network and internet.
Recommendation: Review and plan fixes with your TestOps/DevOps and 3rd party tools support.
Example: A common category for recently created Zebrunner workspaces where the AI/ML model is still untrained; a unique non-reviewed failure detected.
Recommendation: Review and assign one of the reasons to retrain AI/ML and classify the problem next time.
HOW AI/ML WORKS
After your test is failed or skipped, Zebrunner’s AI/ML automatically detects the failure reason (BUSINESS ISSUE, LOCATOR ISSUE, INFRA ISSUE) with a certain level of accuracy (up to 100%).?
If the failure reason cannot be associated with any of the above-mentioned categories, or AI/ML requires additional training, the UNCATEGORIZED tag is added, which means the failure reason needs additional investigation and human actions.
The automatic classification is performed on a test finish, with a small delay possible. If during this delay, a user assigns the failure tag first, their assignment will always take precedence, while the system assignment will be ignored.
If the AI/ML accuracy rate is less than 90%, a special exclamation mark will appear beside the failure label. This means that AI/ML needs the user’s attention to check the correctness of the classification.
AI/ML TRAINING
AI/ML can and should be trained on the basis of failed tests and by real users. In order for AI/ML to begin the classification, at least 7 failures must be classified manually inside your Zebrunner workspace. The more tests you execute and assist to classify, the more accurate the automatic classification will be.
The AI/ML training is performed on the basis of the Like/Dislike mechanism.
If a user’s review is needed (the system marks the tag with the exclamation mark), they need to confirm the classification provided by AI/ML by clicking Like, or deny the classification by clicking Dislike.
If a user accepts the category assigned by AI/ML, they need to Save their choice. This action will be displayed in the AI/ML pop-up.
If a user dislikes the classification provided by AI/ML, they need to choose another failure reason category and press Save. Otherwise, the actions will be discarded and the classification will stay unrated.
Note: Even if AI/ML doesn’t require human intervention (when a failure is classified with over 90% accuracy level), or a user already voted with Like or Dislike, it’s still possible to change the assigned category to another one.
And, as on the screenshots above, all the assignments history is tracked together with the name of the user who chose the category, the assignment date, and time.?
Assignments provided by AI/ML are marked as System and contain the accuracy rate of its classification.
HOW TO START WITH ZEBRUNNER’S AI/ML
Want to see Zebrunner's AI/ML in action? Then don’t forget to visit our website and get started with Zebrunner just now: https://zebrunner.com/