You should always care about bias in your AI projects

Implementing these best practices to combat bias in AI is crucial for every project, big or small. This will pave the way for your business's long-term success.

Taylor Swift’s Era’s Tour has broken records, selling out 146 stadium shows across 5 continents so far. She took the crown as Spotify’s most streamed artist in 2023, with over 105 million monthly listeners. While she does have some catchy songs, her record-breaking numbers have raised questions about another possible force at play: popularity bias in music recommendation algorithms.

Bias is a pervasive theme in our AI love story that we know all too well. ?Spotify has come under criticism that its recommendation algorithms unfairly promote already popular music, creating barriers for independent and emerging artists. Many other, seemingly harmless, applications of AI in predictive modelling have been shown to have serious implications when the model produces skewed results or systematic errors. Credit decisions, job applicant screening, and patient care prioritization are all recent areas where AI has discriminated against groups or individuals based on characteristics like race, gender, and socio-economic status. While predictive modelling has been around for decades, the need to address the AI bias risk is urgent.

This is not only a social responsibility - preventing AI bias in your predictive modelling also protects your organization's bottom line.

No matter how small or innocuous the use-case for your predictive model, the steps in this article need to be diligently applied in your organization to mitigate bias. These tools and processes need to be baked into your organization from the beginning of your data and AI practice. Following best practices for even an experimental proof-of-concept will cement a culture of responsibility that will carry through into larger projects with real-world, sensitive implications. This is not only a social responsibility – preventing AI bias in your predictive modelling also protects your organization’s bottom line. The consequences of bias, like issues with accuracy or regulatory challenges can have large cost-implications.

This article mostly references AI in predictive modelling, chosen due to its widespread use. However, it’s worth noting that Large-Language Models (LLMs) are also predictive models that learn structure, patterns, and nuances from historical data. When given a prompt, they predict the most probable next word or sequence of words based on their training set of natural language data. Therefore, the principles of AI bias management are no different whether you are building an LLM or any type of predictive modelling application.

Step 1 – Acknowledge and understand potential bias.

Acknowledging the bias risk may seem like an obvious step. In practice, many organizations fail to formalize this risk within their core operating principles and employee training materials. Furthermore, this responsibility rarely expands outside of data science teams. To be truly effective, organization wide acknowledgement and executive buy-in is required.

The importance of organization wide acknowledgment can be demonstrated through the popular Amazon case. Amazon developed an AI recruiting tool to streamline their recruiting process and save HR time spent manually reviewing resumes. The model sought to find and predict who will be high performing candidates based on a training set containing 10 years of historical CV data. The tool was eventually shut down after it learned to systemically downgrade female candidates because of the historical dominance of men in the software industry.? What’s notable is that the HR department served as both suppliers of the training data and as the tool’s end-users. While the HR team was primarily concerned with the efficacy of the model and the time-savings it provided, the AI team, though potentially aware of the bias risk, was not directly involved in defining candidate selection criteria or evaluating the implications of the results. This example makes it clear why recognition of bias risk needs to expand beyond the data and AI teams. Predictive AI involves all aspects of a modern organization.

The AI team, though potentially aware of the bias risk, was not directly involved in defining candidate selection criteria or evaluating the implications of the results.

To address AI bias, it’s important to understand the sources from which it arises. Common categories include data bias: incomplete or non-representative training data, algorithmic bias: prejudices introduced during model design, or cognitive bias: inherent human biases affecting developers or end-users.

Step 2 – Diversify your data.

Data bias can be prevented by using large and diverse data sets to train your predictive model. Training data should be representative of the real-world in which the predictions will be used and capture the full range of applicable conditions. For example, if you are training a model on sales data that will be used globally, be sure to include all countries where your organization does business.

Racial bias, for instance, is a large risk in healthcare settings, where there is traditionally more data available from Caucasian patients. Consider a team looking to detect the likelihood of skin cancer. They have well-documented images and data from dermatology clinics all located in European countries. They might be a start-up seeking investment or an internal team in a large organization, either way, they want results fast. Proving their model works as intended will get them more funding and recognition. Soon, even though the team was only seeking to build a proof-of-concept, external pressures push their diagnosis tool to quickly reach the market. Once it’s operational, it becomes clear that the model is dangerously inaccurate for patients with darker skin tones, due to the overrepresentation of Caucasian patient images.

This generalized example reflects true stories common in healthcare related modelling. It shows that, even though the perception of risk may seem low during experimentation, external factors, or turnover within the team, can cause lack of data diversity to be overlooked. Implementing structured processes to assess data needs upfront can help prevent this issue. Gaps in data should be clearly identified and can addressed with data augmentation. If there is limited data available, synthetic data generation can be quite powerful in offsetting data bias. After data bias risk is mitigated, processes should also be in place to flag potential algorithmic bias. ?

?

Step 3 – Implement algorithmic checks and balances

There are a range of technical tools and operational practices that can be used to verify fairness and limit algorithmic bias. The first step is to define the metrics by which you will be measuring fairness. Researchers have come up with different statistical metrics to accomplish this such as demographic parity, equal opportunity, and predictive equality. Predictive equality looks for an equal number of false positives as false negatives across different groups. Another interesting technique is counterfactual fairness. In this method, potentially sensitive attributes such as race or gender are reversed, and the output is monitored for consistency.

The final application may be more sensitive than initially intended.

Much of the onus falls on the developers who need to carefully choose their algorithmic approaches with fairness in mind. Several tools are available to analyze training data and the model’s predictions to identify potential disparities. IBM’s AI Fairness 360, Google’s What-If Tool, Microsoft’s Fairlearn, and Scikit-Fairness are all robust toolkits that can be used throughout the AI project lifecycle. Another concept worth exploring is explainable AI. This is when the algorithms are designed to display the reasoning behind their decision making.

Implementing these tools will ensure checks and balances are followed with every application, no matter the scope. The final application may be more sensitive than initially intended, especially when new development teams get involved and build on top of existing models. On top of that, the legal landscape is constantly evolving and having these processes in place helps your business be prepared. Staying at the forefront of AI bias risk management will also lend your company authority to advocate for, and shape industry wide standards. ?

National Institute of Standards and Technology’s (NIST) AI Risk Management Framework (RMF) provides some useful standards for identifying and managing risk. The EU’s Artificial Intelligence Act (AIA) also categorizes risks, using categories: unacceptable, high, limited, and minimal. While categorizing risk is an important step, without additional checks and balances, developers may be cautious to experiment and explore new use-cases. This could stifle creativity be more costly to address later in your company’s AI journey.

?

Step 4 - Continuous monitoring and evaluation

The AI lifecycle extends far beyond initial data collection, development, and tuning. Real world data and context can shift over time. New biases can appear, and existing biases can be exacerbated.

Your business’s reputation can be negatively affected from a poorly performing AI model, regardless of the severity.

The toolkits mentioned to check algorithmic and data bias can also help to continuously monitor the model’s performance in the real world. Processes should be in place to regularly assess behaviour and prevent “drift”, which is change in model accuracy over time due to changing conditions. These tools can automatically trigger alerts when drift is detected, but scheduled, manual reviews should also be practiced. Even for simple applications, regular reviews and re-evaluation with new data will help align the model with new ethical standards and legal frameworks.

The importance of continuous monitoring and evaluation was demonstrated in February of this year when Air Canada’s AI powered customer service chatbot provided a customer with false information regarding a bereavement refund. The lawsuit made headlines when the customer sued Air Canada asking them to honour the chatbot’s statements. Though the final judgement only cost the airline around $800, the implications on Air Canada’s brand and the public opinion of its AI risk management capabilities carry longer term costs and risk stifling future innovations. While perhaps it deemed chatbots to be a low-risk and high value use-case, investment in continuous monitoring and evaluation may have flagged this issue earlier. Your business’s reputation can be negatively affected from a poorly performing AI model, regardless of the severity.

AI decisions should not replace human judgement, but it should serve to augment and inform it. The concept of human-in-the-loop (HITL) is another powerful framework where human intervention helps the AI model learn and improve over time. Applying this human oversight and regular reviews with stakeholders can only help build transparency and trust around your AI practice.

?

Step 5 – Foster a culture of responsibility and awareness

Intentionally creating a culture of AI responsibility is a challenging task, and trying to change beliefs and ways of doing things that are already baked into employee mindsets is even more difficult. Soon, all companies will use AI and it will touch all areas of your organization. Cultural priorities need to start with the leadership team. Leadership should set clear expectations for ethical considerations and standards with which to align all AI predictive modelling efforts. Executive sponsorship is needed to ensure resources, time, and funding are sufficiently allocated to meet the desired goals.

Cultures, like AI models, are subject to drift over time.

Comprehensive training and continuous learning on the latest developments in AI ethics and bias mitigation should be encouraged and available to all employees. Finally, sharing best practices with the public and other leaders can raise trust in AI across the industry.

Cultures, like AI models, are subject to drift over time. Regular practice assessments and recognition for employees and projects that have successfully addressed bias will help reinforce and maintain the established culture.

Parting Thoughts

The above steps: acknowledging risk, ensuring diverse data, implementing algorithmic checks, continuous monitoring, and fostering a culture of responsibility are foundational to your AI and predictive modelling practice. Thought that list is by no means exhaustive, the need is urgent and should be followed regardless of the perceived severity of your application. It’s all our responsibility to stay current in the field and regularly engage in critical discussions with our teams and with other businesses, with the goal of learning new and evolving best practices

Whether recommending songs or maximizing sales revenue, we all benefit from fair and unbiased AI predictions. Put the measures in place now, so your business can focus on creating and innovating in the new era of AI.