AI Bias and Bad Prompts

A major cause of bad responses in AI is the fact that we don’t know how to ask for what we want. Sometimes we don’t even know what we want. I go over this in a lighthearted way when I discuss how hard it was to get DALL-E to illustrate a joke. AI works well when you’re trying to get to a specific correct answer. It doesn’t work great when the command is “Draw a picture that makes me laugh” or “Rewrite this so it’s better but still in my voice”. I would know. I’ve tried both of those a lot.

Additionally, a generative AI tool is never going to tell you that your question is bad; it’s just going to guess the right answer. Sometimes this is particularly useful. I’ve asked ChatGPT “What is an Asian made car that’s like zippy?” (Miata) or “What’s a word for ‘around everywhere’ that starts with an A?” (Ubiquitous). It got the right answer even though my question was incredibly vague in the case of the former and getting key facts wrong in the case of the latter.

But issues arise when we ask bad questions in areas that actually matter. If I ask ChatGPT to recommend a fun song, it’s going to decide on a definition of fun and use that to make a recommendation. I can either accept that recommendation (even if it might not be the best possible choice for me), or I can go back with some more specific requirements. It’s going to do the exact same thing if I ask it to recommend a good mathematician for an open position but with much higher stakes.

Let’s say I do ask an AI tool to write a description of a great mathematician because I’m a hiring manager looking to fill an open position. There is a lot of data that it could use. It can look through job postings and see what other companies are looking for in a role that requires a mathematician, though this is likely not going to be the job title. It might define mathematician to be a person with a higher degree in math, rather than a specific job title. It can then look through LinkedIn and see what some of the shared characteristics of mathematicians are that have worked for a long time or who have a higher number of followers or who have high level positions. It might look up biographies of famous mathematicians to see what commonalities they have. It might also look at TV shows or books or other media in which a mathematician protagonist is featured.

The AI tool will come up with a list of characteristics that describe a great mathematician. Some of these qualities that it would get from the above training might include characteristics like ‘good at problem solving’, ‘logical’, ‘knowledge of mathematical theories’, ‘man with glasses’. Most of us will agree with the first three qualities but have a problem with the last one. But all four are pretty logical assumptions.

I don’t know where the stereotype that bad eyesight equals good at math comes from, but you have to admit that it is prevalent. However, we can easily understand why it might have the gender requirement. If the tool is trained on historic job postings, any from before 1964 might actually specify that they’re only looking for a man. Famous historical mathematicians are likely to be men as well. While women’s representation in STEM fields is increasing, I don’t think that it’s yet at parity. If you’re looking for a mathematician, there is a good chance that the best candidate is a man.

This seems icky to me. But the AI tool’s ‘job’ is to find the characteristics that are most likely to result in a great mathematician. It doesn’t draw a distinction between the fact that ‘logical’ is inherent to the definition of mathematician and ‘man’ is the result of systematic oppression that would not let women in these prestigious jobs. From the AI’s perspective, finding a great candidate for this position might require both inherent skill and the ability to navigate the patriarchy. It’s going to use both criteria unless it is told not to.

You may have noticed that I switched from using ChatGPT when I write about my experience to ‘an AI tool’ when I write about the situation with the hypothetical HR bot. That is because the major tools on the market, including ChatGPT, are explicitly taught not to give answers that could be considered prejudiced. One of the first things people seem to do with generative AI tools is to try to make them say something racist, and that inevitably makes the news. But a smaller, in-house developed AI tool may not have these explicit prohibitions and there might be some unforeseen loopholes. You can ask ChatGPT to describe a perfect job applicant all day long and it will never use a physical characteristic or even gendered language. Even so, there are still some issues with the major market tools. If you ask ChatGPT to draw a perfect job applicant, it has no choice but to use physical characteristics and will almost always draw a white man with glasses for a STEM job.

This is a real issue that we are dealing with right now. New York City has recently put into effect a law that limits the use of AI in hiring decisions for exactly these reasons. I don’t know how effective this law will be, but I’m hopeful. I think there needs to be a lot more work done on the part of both regulators and innovators to understand how AI models work. I think that a good first step for all involved would be to work hard to make sure that the questions we ask are?precise.