What Most People Get Wrong about AI & Bias

When I taught and created content for a course in the NYU SPS Human Capital Management program, I wanted to help graduate students prepare for what it means to understand, evaluate, and implement rapid changing technology inside of organizations. One of the most common misconceptions surrounding the topic is how AI algorithms encode and perpetrate cultural biases. Unsurprisingly, it is a very complex problem.?

But first, what is Machine Learning??

Machine Learning (ML) is a field of Artificial Intelligence (AI) where computers learn to make predictions or decisions from data without being explicitly programmed, using algorithms and statistical models. It enables computers to improve their performance on tasks as they gain experience from the data they process.

A fallacy I often hear in companies is, “AI or ML models are biased."

Well, not exactly. Often, headlines refer to bias in AI in a vague and uncertain way. In the famous case of Google, where in 2015 software engineer Jacky Alciné pointed out that the image recognition algorithms in Google Photos were classifying his black friends as “gorillas.” To summarize this occurrence, the algorithm that Google created did not know how to identify people or animals correctly, resulting in a devastating outcome. (Should Google have tested this, yes. Was the actual algorithm biased, no. We’ll explain).?

First, you need to understand how ML models learn and how data scientists or engineers “train them”:

• Getting Data: First, they gather the information (data) we need to teach the model.
• Cleaning Data: They make sure the data is neat and tidy, like fixing missing values or making it easier for the computer to understand.
• Splitting Data: They divide the data into different parts: one for teaching (like a textbook), one for checking how well they're teaching (like a quiz), and one for final testing (like a big exam).
• Creating Hints: Sometimes, they create helpful hints or clues for the computer to learn better from the data.
• Learning from Data: they let the computer learn from the data, just like studying from the textbook.
• Adjusting Learning: they help the computer get better by making some adjustments in how it learns.
• Checking Progress: they check how well the computer is doing by giving it quizzes (validation) and making changes as needed.
• Final Exam: they test the computer's knowledge on a big exam (test set) to see how well it learned.
• Using What's Learned: If the computer does well, they can use it to make predictions or do useful things.
• Keeping an Eye: they keep an eye on how well the computer is doing over time and update it if needed.

As you can see, there are many phases where things can go awry . . .

Think of it like teaching a computer to recognize pictures of cats. You gather cat pictures, make sure they're clear, split them into learning, practice, and testing groups, pick a good way for the computer to learn, let it study, and see how well it can identify cats in new pictures. If it's good, you can use it to find cats in photos. And you need to check if it's still good at it as time goes on.

Let's review another famous case where things went terribly wrong. Amazon had invested in building their own recruiting AI tool, only to find out that the algorithm learned that women often did not get the job in the male dominated technology industry. Therefore, it started to give female applicants lower scores, which often removed them from the process altogether.?

Often, people confuse the teams who built them or the algorithm itself with the biased patterns that the algorithm begins to display in the outcomes, patterns or predictions it suggests. Algorithms in Machine Learning and AI models are not in and of themselves biased, they learn from patterns where many times, the insights and patterns reveal that there is a bias in the insight or a mistake in the calculation.

In the case of Amazon, it learned a biased pattern, where female applicants statistically were unlikely to get a job offer. In the case of Google, the algorithm incorrectly categorized attributes as humans or animal, which was disturbing, but this was not a pattern that it was trained on, the machine made assumptions that the engineers were unaware of until tested.?

But looking at these two examples, it’s important to understand, how do AL or ML models work? Let us explain in a more specific way, as understanding the anatomy of these algorithm is important to have critiques of where AI or ML can present bias and ethical dilemmas:

Two of most commonly used model “types” used today are:

(1) Neural network models: these types of algorithms perform best with unstructured data; i.e. image? recognition and classifying images into categories, i.e. pictures of handwritten numbers and classifying them into the correct respective categories.?

Neural networks are a class of machine learning algorithms inspired by the structure and functioning of the human brain's neural networks. They consist of interconnected nodes, also known as artificial neurons or units, organized in layers: inspired by biological neural networks, where the behavior of neurons are controlled using many parameter. Each connection between nodes has an associated weight, and these weights are adjusted during the training process to optimize the network's performance. Neural networks are good at handling complex and non-linear relationships in data. They excel in tasks such as:

• Pattern Recognition: Neural networks can learn to recognize patterns and features in data, making them useful for image and speech recognition, handwriting recognition, and object detection in computer vision.
• Natural Language Processing (NLP): Neural networks are used in various NLP tasks like language translation, sentiment analysis, text generation, and chatbots, as they can capture intricate semantic and contextual relationships in language.
• Classification: They can classify data into different categories or classes, such as email spam detection, medical diagnosis, and sentiment analysis

The complexity with neural networks is that they will come to a decision about something, i.e. classification of an image, i.e. categorizing a picture, but scientists and engineers do not always know why this decision was made. While scientists and mathematicians who create these algorithms understand the actual quantification and calculations, when the algorithms branch and make a decision about an image for example, it is really difficult to know when, where and why the algorithm made the decision it did. Hence, neural networks are more prone to making decisions and recognizing patterns that can result in a bias– making them more complex and thus, potentially dangerous.?

(2) Decision Tree models: this type uses structured data, a common example would be identifying home prices by combining labeled attributes to recognize patterns that result in price, for example,? location, square footage, age of home, etc.?

For example, XGBoost (Extreme Gradient Boosting) is a powerful and widely used machine learning algorithm designed to solve various types of problems, particularly structured tabular data. XGBoost builds a strong predictive model by combining the predictions of multiple weaker models, typically decision trees.

Applications of XGBoost are particularly effective with structured data, where features have well-defined meanings and relationships. This makes it a popular choice for many Kaggle competitions and real-world business problems involving tabular data.

With the examples of Google and Amazon, you can see that the google image example was most likely using a neural network algorithm and Amazon was most likely using an XGBoost algorithm (just my guess), both presented a devastating outcome, but for different reasons.?

As business leaders, it is vital that we know enough detail in order to diagnose, understand and prevent these outcomes from impacting humans, their careers, hiring decisions etc.?

How to prevent bias in your organization’s algorithms:

1. Make sure the datasets that are sourced, curated and used are intentionally diverse.?
2. Audit and request that all internal and third party technology teams provide the following when contemplating adding in AI into your business processes (in addition to other privacy due diligence): (a) details about their test data sets and how they have tested?them (b) the model or algorithmic type?and how they have trained or not trained the model (depending on type) (c) what variables the model uses and to what degree each variable is weighted. For example, related to hiring, if the model more heavily indexed on education vs years of experience. Often, engineers/data scientists control what information is fed into the models. Machine Learning models typically decide how to weight the importance of this data on their own in the training process– hence, the need to understand how the model works, as the machine can determine importance and it may not be the variable we want to index on.
3. Create an ethics committee: as AI becomes more integrated into business operations, ethical considerations and potential biases must be addressed. The authors highlight the need for multidisciplinary teams to ensure responsible AI development and deployment, balancing technological advancements with ethical decisions.?
4. Test before moving anything into production: if case studies on AI have proven anything, it is that testing must be robust before considering using AI in the real world. Testing still doesn’t guarantee you will get everything correctly, but by creating a diverse team where you can generate use cases, you can write use cases (tests) that will quiz the algorithm/model for any potential bias outcomes or insights.?

Additional Resources on AI and Ethics:

• https://hbr.org/2020/10/a-practical-guide-to-building-ethical-ai
• https://hbr.org/2018/08/3-questions-about-ai-that-nontechnical-employees-should-be-able-to-answer?ab=at_art_art_1x4_s01
• https://hbr.org/2023/07/what-smart-companies-know-about-integrating-ai?ab=HP-magazine-text-1
• https://hbr.org/2022/09/ai-isnt-ready-to-make-unsupervised-decisions
• https://hbr.org/2022/07/why-you-need-an-ai-ethics-committee

A special thank you to Anna A. Tavis, PhD for many conversations, support, and mentoring over the years as I've taught courses in the program. Additional thank you to Beverly Tarulli, Ph.D. for her curriculum review and mentorship.

*This article only represents my personal thoughts and perspective.