What are the best ways to visualize AI prediction uncertainty?

I come from a bioepidemiology background, so for me, 95% confidence intervals are most helpful. I also like gradient based confidence intervals a lot.

As an economists, I am keen on simple numbers covering the probabilities of error and how those are spread between false-positives and negatives however many would struggle to understand )5% confidence intervals. So my advice is, if you have the time, to focus on the situational experience at hand. If this is about a voice assistant in a passenger car: How often do people interact with one another, and give commands for the car, what % will those be correctly acted upon and how often will the car interrupt normal conversation, due to false positives? It helps paint the picture and might also bring new PoVs on the CX side.

I tend to agree with what is written by others. I wish, however, to offer another approach: Bayesian statistics offers an evaluating distribution of scores, this is a new domain that is called BNN (Bayesian neural network). the idea is that by designing the NN in a more sophisticated way, We will be able to generate a confidence. interval around our performances, This approach offers a standard statistical language that the clever people from the counterparty will be able to discuss coherently and allow convincing with the quality of the models. If we work on tabular data, try calculate its DAG using packages such as BNN to create explainability. It won replace uncertainty analysis but it is undoubtedly a complementary tool

What's the best visual in any scenario? One that's easy for your user to read! Don't just pick charts those that are intuitive to you (the creator) or pack in more info (data density). Error bars build upon the bar chart, one of the most intuitive visuals - since comparing heights is easy for most users. This works particularly well in the financial industry since it has some similarities with the candlestick chart, a popular one amongst users.

Imagine a doctor is using AI for medical image diagnosis. Heatmaps can help identify where AI is certain about the diagnosis and where it's not sure. This helps focus on areas needing closer examination. It strengthens trust in AI technologies and also assists doctors in making better decisions.

The What-If Tool (WIT) is an open-source visual interface from TensorFlow, designed to interactively explore the uncertainty of models. It showcases uncertainty by generating counterfactuals, revealing borderline decisions, and highlighting fluctuating predictions based on feature alterations. It does this by keeping user in the loop. Look out for such what-if tools that are available across libraries to demystify the opaque nature of AI and the uncertainty of its predictions.

There are different types of uncertainties that should be considered in neural networks. One is Aleotoric uncertainty which is present due to noise present in observed sensor data and other processes which add noise to the input. The other is Epistemic uncertainty which is present in the model from the training process and can be dealt with by using additional training data. There is scientific literature which discusses methods of quantification of uncertainty in neural networks. These methods can be used to quantify the uncertainty in models and can be added to the project documentation to convey this information to the end users.