Three Pillars of Chatbot Testing

Testing discipline – just like every other discipline in IT delivery – has evolved through multiple phases. From testing large monolithic applications in a waterfall delivery model to testing SaaS services developed in an agile fashion, testing fraternity has learnt several tools, models and processes along the way. The latest challenge they face is testing AI solutions! Rather than the software following a pre-defined pre-programmed logical construct, the software ‘self learns’ & evolves over time. Chatbots – that uses Natural language processing – pose a similar challenge. It's time to look at new toolsets for sure, but same established processes can still apply to test chatbot solutions, if right adaptions are made, below are some quick thoughts:

Test Data Management: AI and machine learning thrives on data! Chatbots are built on AI capability to understand natural language. More training data reinforces chatbot’s learning and improves prediction accuracy. The testing phase should build relevant test data to enable bot’s reinforcement learning. If the business uses a contact centre already, then the anonymised contact chat history would be a great source of building test data. Test data for chatbots must be composed of three sources – a) data from system of records (user profiles, product catalogues, etc), b) industry specific utterances to meet chat’s business goal & c) generic conversation pleasantries (greetings, feedback, small talk, etc). Use of utterance generator tools will also help in automated test data generation.

Test Driven Development: rather than taking conventional approach, where engineers write unit tests first & then build code, for chatbots, these tests should be written as end to end dialogs as black box tests. Again, using the existing contact chat history can help in defining the ‘would-be’ conversations. Once the tests are written as realistic scenarios, the developers can follow the ‘Red-Green-Refactor’ path to implement TDD. The test cases should also include flavours of boundary conditions (which can be many for conversational interfaces).

Test Automation: after the test cases are established & test data is setup, automating the test execution should be really straightforward. Simple selenium-based frameworks that use excel sheets as test data/flow database, can be an obvious choice. Test automation for chatbots is valuable when end to end chatbot usage flow including bot design, execution, analytics, re-training bots & validating bot post retraining, is automated. Randomising user utterances during automated test, will help in testing context awareness & context switching capability of chatbot. 

Apart from these adaptions, other best practices of ‘shift left’ testing & continuous non-functional testing should also be applied for successful chatbot delivery & assured business outcomes.