I know why the model is doing that, but how do I change it?

Rather than just helping a user understand why a model behaves the way it does, help them change the behaviour.

There’s currently a lot of discussion about how we can make the behaviour of AI models more transparent. However, this often focuses on letting the user see how a model works or why it behaves the way it does. The user generally has little control over the actual working of the model, or the process by which it was created. The concept of scrutability addresses this problem. Judy Kay defines scrutability in user modelling for personalisation. Kay states that scrutable user models are those where the user has control over the information stored about them, as well as the analysis that is performed on that information. Although this definition is provided in the context of personalisation, its principles can be applied for AI applications.

A natural way to give users control is to allow them to remove certain features from the dataset. However, we can also provide control over data processing steps. For example, in the EDA (Exploratory Data Analysis) phase of a project we may reduce the number of features through statistical tests. One aspect of this is multicollinearity testing, where we identify groups of features that are so highly correlated with each other that we only need to keep one and can discard the rest. A data scientist will try to make the decision of which features to discard based on a statistical measure e.g. retaining the feature with the most predictive power. However, a user may have a different view as to which features they would like to retain. For example, a domain expert may determine that a particular feature will often not be available at the time a prediction is being made.

Another way to give users control is in the model selection process. Data scientists will often try multiple models on a dataset to determine which gives the best performance. Performance can be measured with a variety of metrics e.g. R-squared for regression, accuracy for classification etc. Once a model has been selected, it will often be refined through a process called hyperparameter tuning. This is where settings within the model are optimised to improve its performance. The choice of metric is obviously crucial in this process, since it will determine which model is chosen and how that model is tuned. Data scientists have heuristics available to them for choosing metrics. For example, when classifying an imbalanced set, accuracy can be misleading. This is because a classifier that always predicts the majority class can achieve a high accuracy score. In this scenario, a metric such as AUC is better. However, data scientists may not always have the full picture. The choice of metric is essentially a statement of how the model should behave, which will often require domain knowledge. For example, consider the case of a doctor who is employing a model to determine whether a patient has cancer. The doctor may require that the model be highly accurate when predicting that a person does not have cancer, even if it is not as accurate when predicting that the person does have cancer.

Any ideas as to how we can make our AI models more scrutable? Leave them in the comments below.