How do you use human feedback in Machine Learning?

Humans have their own processor called the brain, so using human feedback in ML is an opportunity to insert a natural feature engineering layer into your model, adding features that will be pre-processed by human brains. These selected features will improve the diversity, robustness, and generalization of the ML models.

One uses the so-called feedback in a fruitless attempt to overcome the stochasticity in ML - or, in other words, trying to jiggle the generalization problems. As long as there is a stochastic model, there is no guarantee that the outcomes are always deterministic or that they make sense.

Additionally it is also important to address the kind and amount of human feedback being leveraged during ML practices. For example, using only a subset of individuals to provide feedback could lead to data bias, or during hyper parameter tuning, if too much human feedback is involved then the models maybe overfit and may not generalise well.

Well, so let's stop and ask ourselves: what are the **three** ML components? SL, UL, and RL. What does the feedback do? Improves the UL (adjust the classes), cleans up the SL, and improves the RL by cleaner data or by rule adjustment (policy and/or reward). So, after one localizes the effect of the feedback, one has the method of collection. No, there is no magic recipe.

Here is an example of how human feedback in ML can be used: Problem: A bank's credit scoring model inaccurately predicts loan defaults, leading to financial losses. Human Feedback in ML Solution: Loan officers review the model's predictions and provide feedback based on their expertise. This feedback is used to retrain the model, improving its ability to predict defaults. Evaluating the Impact: The model's performance is measured using metrics like accuracy and AUC-ROC. Financial outcomes, such as reduction in loan defaults and increase in approved loans, are also tracked to assess the impact of the feedback.

A very important model evaluation metric is to quantify the difference between models trained with and without the human feedback layer. Analyzing deeper model metrics such as Feature Importance or prediction errors for each class in a classification model is critical to understanding the influence of Human Feedback. This metrics evaluation procedure is fundamental to obtaining interpretability, transparency, trust, and accountability.

Supervised ML in legal document review relies heavily on the labeling efforts of humans in order to make predictions of document relevancy. Since humans aren’t machines, the quality of the labels may vary depending on their interpretation of the labeling instructions, concept drift (evolving understanding of the case), or even unrelated factors (being fatigued, disengaged, etc). Ideally, you’d want a team of reviewers who make consistent decisions not only individually, but also collectively as a group.

Obtaining human feedback at scale by contributor numbers is challenging because of human bias and variability, but we'd be fooling ourselves to say that such aren't already in the training data from the start. Getting at that necessary scale by way of additional time is also challenging, because people lose motivation, get bored, and lose accuracy. Scale solutions must recognize and handle that human fluctuation and engage the feedback providers. An unorthodox but promising opportunity lies in the rise of Game-Based Learning, in which the digital medium allows granular control over prompts, ubiquitous feedback collection points, and all while keeping the participant engaged and exercising great effort in the activity if done well.

A common misconception is that humans are sovereign individuals. Humanity is a social construct. The individual context we value only exists within a social collective. There are a number of biases that influence human perceptions. We must be careful in considering the extent of social on individual context when using it as human feedback to train ML models. Individuals tend to seek out confirmation bias - information that confirms their existing beliefs or expectations, leading to selective attention and filtering of information. Another example is the availability heuristic, where people base judgments on the ease with which information comes to mind. Recent memories are accessed first to optimize brain processing.

Another key challenge to using human feedback in ML in my opinion would be ‘domain knowledge’. The understanding of the domain space of the problem is crucial to the success of any human feedback. Several times humans involved in the formulation of ML solutions are not domain experts and are scientists or engineers who have knowledge about the ML algorithm but no the problem space itself, hence results may not always be favourable due to errors made in the upstream. Involving subject matter experts could help mitigate such errors and prevent propagation of any flaws.

Human feedback increases the users' sense of ownership over the models as it brings the end user closer to the developer and exposes the importance of humans in the ML model development loop. Human feedback can also be seen as an excellent way to insert human knowledge and experience into data models, inserting a positive "bias", as it derives from the experience of many years of each individual.