A point of view on Testing Cognitive Solutions

IBM is at the forefront of cognitive technologies and the organization is bringing in new ways to effectively use IBM’s artificial intelligence engine “Watson” into the realm of our day-to-day work. One of a “first of a kind” use-case we are working on is adapting IBM Watson to tutor college students through natural language dialog without any human intervention. Evidently, the major “heavy-lifting” is to train IBM Watson with the corpus of knowledge needed, but a large part of the training effort is to ensure that IBM Watson is tested thoroughly with all the possible combinations and traversals of the dialogue. This article is about how to improve the efficacy of cognitive chat-bots like these.

We have developed an automated testing framework which we are using for testing the IBM Watson Tutor. The e-learning titles are IBM Watson conversation enabled and use deep-learning mechanism to understand student intents, questions, provide hints, clear misconceptions & doubts and help students get tutored by IBM Watson. The challenge was to generate enough testing scenarios, cases, and data dynamically and in near real-time to simulate a conversation between IBM Watson and the student and validate both the product, the content and metadata involved. This framework is helping save large manual testing effort and helping in increasing the test coverage substantially.

This framework has three key components -

1.   Test Data Generator Service: It takes the contents (with which the cognitive tutor is trained) as input and generates the entire expected conversation between IBM Watson and student. Simulates all possible conversation routes that the student and IBM Watson can take based on the domain and very effective in understanding the bottle-neck of the conversation. Leverages NLP for creating the response.

2.   Test Data Validator Service: It simulates the chat between IBM Watson and student through REST API calls and tracks the failure steps, so that problem areas can be quickly identified and rectified.

3.   Test Automation Orchestrator Service: Performs the orchestration of the micro-services and integrates with the run-time environment. Provides various options to users to define and orchestrate the test-cases, so that the user can choose to run a or the full-set, depending on which phase the testing is being done. 

The framework in micro-service based and scalable. Using similar approaches in other domains (remote process automation, deep-learning etc.) can possibly help the training effort and bring in a judicious mix of training vs. testing data.

The views expressed are entirely my own and have no bearing on IBM's position