How do you test AI solutions with users?

The key aspect to consider during user testing is that the business problem is kept in focus. It is possible that all user may or may not appreciate all the factors associated with an AI solution. Therefore, it will be essential that issues or observations reported by users are analyzed and then triaged appropriately, whether through tweaks to algorithm/system interface or via training/education to users.

I feel fortunate to have the unique perspective to engage multiple Fortune 10 companies on their application of UserTesting as it pertains to AI outcomes. I agree with Atika on keeping the business problem in focus. I would like to share my perspective on also being really mindful of the application of those insights. During my time helping leaders build new product development life cycles that include UserTesting insight applications, I saw a large degree of variance in companies yielding value. Create feedback loops around measuring ROI and customer engagement from the provided insights. Be prepared to pull the tread beyond the initial scope of questioning.

A good way to come up with user acceptability testing methods (UAT for short) is to ask an AI! If you're working in GPT, ask Claude to generate a UAT checklist for you, or vice versa. Try asking an AI to roleplay as an expert in ___ field and to come up with testing measures for you and personas for target users. Personas are super powerful in AI.

It is essential that testing and training data are distinct and differentiated. I cannot emphasize this enough. If you test with the same data you train on, your models will all score 100% and you will fail to improve.

User tests can take different forms and serve different purposes. For example: - Model accuracy: Compare historic data (and how humans have handled it) vs how the model performs to determine additional improvement potential for the model. - Process changes: Is the documentation clear? Is the user experience intuitive and easy to follow? Will it actually safe time/ generate more revenue/ etc. - Troubleshooting: Do users know where to find additional information, how to diagnose issues, whom to inform, or how to resolve common issues by themselves? The type of user test depends on the type of the project/ implementation you want to verify.

Testing AI solutions with users requires a tailored approach to address the complexities of these systems. To leverage user feedback effectively, it's crucial to prioritize and implement changes based on user input, evaluating their impact on both user experience and AI performance. Iterative testing and refinement are often necessary to align your AI solution with user needs and expectations fully. This continuous improvement cycle helps create AI systems that not only function well but also provide a seamless and satisfying experience for users.

In my view, it is crucial that users know that they are testing AI upfront. There have been some instances where users have been testing and training AI without their knowledge. This kind of practice has ethical implications.