The Way we use Technology Amplifies Injustice & Inequality
With the advancement of automated technology we are relying on computer programs to act as moral and ethical agents on behalf of humans. Algorithms determine where you can live and work, they decide if you are good or bad at your job and they are even used to determine whether you could be a potential criminal or not.
Algorithms and programs have been given an immense amount of power and trust in our world — but are they worthy of it? Are algorithms really blind to the prejudices and biases of the world? Can they be trusted to make infallible decisions that can make or break someone’s life?
Can Technology Really Be Biased?
Computer programs are after all just crunching numbers and cannot be influenced and corrupted as people can be. However, because of the immense amount of trust placed in algorithms they become all the more dangerous when they are wrong. Algorithms can and do display bias. They are not perfect and treating them as if they are has dangerous consequences that effect many people’s lives.
While I understood that bias could fester in computer programs I never really understood how harsh the effects of bias could be. I personally thought that the worst that could happen is that sometimes Siri won’t understand what I’m saying sometimes or a Google search result might be racially insensitive once in a while.
But while that is awful, it isn’t the end of the world right? It’s not actually effecting my life in a negative way — so it probably isn’t that bad. WRONG. I can’t believe that even after majoring in Computer Science and interning at companies like Google and Microsoft I did not realize how dangerous the consequences of technology could be.
Self-driving cars, for instance are more likely to crash into pedestrians with darker skin tones because object recognition systems are better at recognizing people with lighter skin tones. Your life could literally be at stake because Object Recognition Software is biased and does not recognizing people with darker skin-tones as humans. Another study shows that machine learning algorithms that help recruiters find relevant candidates based on job descriptions are gender biased. These algorithms determine whether or not someone is a good candidate for a job or not, and because of this bias accomplished women are not being considered at the same level as their male counterparts.
The Way we use Technology Amplifies Inequality
What’s even worse is that these algorithms are often used by the government and legal systems. Consider COMPAS — a risk scoring algorithm that scores the likelihood of an individual to engage in recidivist crime (the likelihood that someone will commit crime again). COMPAS is widely used in the American criminal justice system in pretrial and sentencing and is often a deciding factor when determining whether a criminal should get a harsher sentence or not. The inner workings of COMPAS are not available to the public, it is used in massive scale, and as it will be shown extremely unfair.
ProPublica, a non-profit team of investigative journalists analyzed criminal records and corresponding COMPAS scores to find that COMPAS’ predictions had double the false positive rate for black defendants than white defendants and double the false negative rate for white defendants than black defendants.
This means that COMPAS is more likely going to falsely score a candidate as someone who will recidivate if they are black. At the same time COMPAS is also more likely going to falsely score someone as low risk of recidivation if they are white. COMPAS amplifies racism and contributes to mass incarceration of black people in America.
There’s dozens of algorithms like COMPAS that are used to make life changing verdicts about us. The fact that algorithms display bias is pretty bad, but the way we consider them to be absolute truths and provide no way to question them makes them a whole lot worse.
What Happens When The Numbers Are Wrong?
Cathy O’ Neil describes in length how certain algorithms are often unquestioned and unaccountable. The verdicts of these algorithms are taken to be the truth. The algorithms are scaled to make judgements on large masses of people, and the worst part is that very few individuals are privy to understanding how these algorithms work. O’ Neil calls these algorithms which are “opaque, unfair and massively scalable” are known as weapons of math destruction (WMD).
The most upsetting and damaging consequence of WMDs is that their results are so difficult to prove wrong. In 2007 Washington D. C’s mayor Adrian Fenty was determined to improve under performing schools. The education reformers he hired theorized that the issue was the teachers, so they decided to employ this teacher assessment tool called IMPACT to weed out the bad teacher. IMPACT turned out to be a WMD, and Sarah Wysocki became a victim of one of IMPACTs false verdicts. Wysocki, in spite of being revered by the school’s principal and students’ parents, got a very low IMPACT score and was fired. Wysocki students’ standardized test scores were artificially inflated the year before by another teacher. Since Wysocki refused to inflate the scores again her students’ scores reduced on average. Wysocki theorized that this was the reason her IMPACT score was low. She had a strong case so she decided to appeal the score and clear up the confusion.
While WMDs are deaf to charms and threats, they are also deaf to logic. Unless there is iron clad proof that a WMD is faulty and an irrefutable amount of evidence is presented to show that the algorithm is wrong the verdict of a WMD is often taken to be the unwavering truth. Simply, presenting suggestive countervailing evidence will not suffice. O’ Niel articulates this sentiment of inequality the best — “The human victims of WMDs are held to a far higher standard of evidence than the algorithms themselves.”. As expected, when Wysocki challenged her score, the district responded by saying that there was nothing that they could do and the numbers did not lie — they told her she was treated fairly.
Fortunately for Wysocki, she was quickly able to find another job elsewhere. However, not all victims of WMDs are this lucky.
How Do The Numbers Start Lying?
Placing an unquestioned amount of trust in machines and algorithms has egregious consequences. It can fuel the inequalities already present in the world and amplifies their consequences. These algorithms were created to avoid human issues like bias and prejudice, but somehow end up further perpetuating them. Why does this even happen? How do numbers and math start lying?
All algorithms are based on mathematical models. A mathematical model is an abstract mathematical representation of a process, device or concept; it uses a number of variables to represent inputs, outputs and internal states, and sets of equations and inequalities to describe their interactions. In simpler terms mathematical models are ways to simplify processes and concepts in order to represent them mathematically.
While mathematical models are consistent and will always generate the same answer for the same scenario, they are still susceptible to the bias manifested in decisions made by their creators. Who decides how these models are created? Who decides what factors and parameters should be included in a mathematical model? Humans do. Humans decide which parameters need to be considered and how complicated decisions are simplified and represented mathematically. Something everyone can agree on is that all humans are biased or blindsided in some way or the other.
At this point it may be tempting to just say that algorithms are not biased — it is just the creators who are. There are many articles and blog posts out there that claim exactly this: algorithms are simply biased because humans are. While individual human bias and blind spots are a contributor to bias manifesting in technology, the issue is not as simple as that. Bias can be created in many more ways than just someone not including the right parameters in a mathematical model and not simplifying the real world in an accurate way.
Bias, in fact is a required function in any kind of predictive algorithm. The observation that bias is implicit and a required factor to determining any kind of algorithmic prediction making was made decades ago by Dietterich and Kong. All algorithms are biased to show results that support or help the purpose they were created for. In order to create a mathematical model, one needs to consider what the purpose of the model is.
Prioritizing Money Over Morals
In most highly scaled and pervasive models the goal is to either cut costs or increase revenue. For instance, a credit card company would want to predict a customer’s creditworthiness, but, “creditworthiness” is a very abstract and vaguely defined concept. When translating this into something that can be computed a company must decide whether it wants maximize profit or increase the number of loans that get repaid.
The problem is that revenue and business goals are at the heart of these decisions and ethics and fairness are often a mere afterthought to avoid bad press and publicity. This causes algorithms that help determine creditworthiness create scores that will help make choices which will eventually increase revenue. These scores do not care about ensuring fairness and equality among the people they score because this was not the purpose the model was made for. They may reflect racial biases and other unethical in-differences of the world but because their purpose is to help generate revenue, as long as the purpose of profit is satisfied there is no reason for credit card companies to change these models.
Since growth in revenue is often the determining factor for a lot of widely used algorithms in the industry, these ethically questionable WMDs are often considered to be extremely successful. Placing revenue at the center and not having conversations about ethics amplify the effects of WMDs.
Note: There’s a lot of other ways bias can occur — a lot of it relating to the kind of data we use and how we process it. I won’t go into those details here but I’ve included some resources below.
We Need To Stop Assuming Programs Are Perfect
What happens after the models are created? How does one know whether a program is working or not? How does someone determine whether or not their program is fair and ethical? Once models are created, they are tested. Bias can and does manifest during these testing cycles as well. Usually when an algorithm does not seem to reflect reality it is tested and tinkered with until it represents reality more accurately.
Feedback is important to understand whether or not a model works. A model that works really well in certain population and demographic might not work that well at predicting results accurately in another population. Diabetic patients, for example, exhibit different symptoms based on ethnicity. HbA1c levels, which are used to monitor diabetes also manifest in different ways across ethnicity and genders. Thus, a singular model would not suffice to accurately predict diabetes in patients even if all populations were equally represented in the training set.
The danger occurs when imperfect mathematical models are treated as perfect and their verdicts are treated as irrefutable truth. When mathematical models are not questioned, they both define and justify the truth — resulting in pernicious feedback cycles. This ends up amplifying and perpetuating bias both within the model as well as in reality. Take the example of IMPACT and COMPAS — the WMDs mentioned in the earlier sections. One of the reasons these models became incredibly dangerous and powerful was that there was no way to oppose their verdicts. Since there was no way to oppose their verdicts the models were assumed to be accurate and true, thus further perpetuating inaccurate scores.
IMPACT resulted in teachers being penalized for lower average standardized test scores even if they were being more ethical, and applauded teachers with higher standardized test scores even when they doctored the tests. Similarly, COMPAS perpetuated racism by falsely rating people of color with a greater likelihood to recidivate which caused black people to get harsher sentences which then actually put them in scenarios in which they would recidivate.
What Now?
Some of the most dangerous consequences of computer algorithms occur completely because of how humans choose to use them. The way society treats and uses ranking systems and scores as always absolute verdicts, the way problems are phrased, the way that ethics is often not on the table are all major contributors to both algorithms displaying bias as well as the extent to which they effect human lives.
Educating both users and creators of technology about how technology is influencing people’s lives is incredibly important. A model becomes a WMD when its inner workings are unclear and unknown to most of the public, when it is employed at large scale and does not accept negative feedback, and when it is unfair. To prevent more WMDs from being created we need to increase transparency, reduce the amount of trust we place in these models by providing a way for “victims” of these models to argue against their score and to finally prevent injustices by bringing ethics into the conversation.
About Me
I’m not a researcher or expert in the field of ethics or computer science. I’m just a computer science student who was given an assignment to write a paper about the negative effects of computing technology.
During my research for the paper I was absolutely appalled to see how technology could be used to make damaging decisions and amplified inequality. Even though I am a computer science major and had significant industry experience, I did not really understand how bias could manifest in technology or how awful the effects of bias could be. There’s very little awareness about this issue and there’s a lot that needs to be done to prevent it. This is the least I could do to bring more attention and awareness about the issue.
For more of my illustrations, writing and thoughts follow me:
Dasani Madipalli (@dasani_decoded) * Instagram photos and videos
Dasani Madipalli - Incoming PM - Microsoft | LinkedIn
Further Reading
I used a lot of different sources to write this post but if you want to learn more I’d recommend Reading the following two books:
- Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy by Cathy O’Niel
- Algorithms of Oppression: How Search Engines Reinforce Racism by Safiya Noble
Resources & References
Calem, Paul. Firestone, Simon. And Wachter, Susan. “Credit impairment and housing tenure status”, Journal of Housing Economics, Vol 19, Iss 3. 2010. Pp 219–232
Ajunwana, Ifeoma. Friedler, Sorelle, etc. “Hiring By Algorithm: Predicting and Preventing Disparate Impact”. 2016.
Chang, Serina. Zhong, Ruiqi. Adams, Ethan, et. “Detecting Gang-Involved Escalation on Social Media Using Context”. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018. Pp 46–56.
Logg, Jennifer. Minson, Julia. Moore, Don. “Algorithm appreciation: People prefer algorithmic to human judgement”.Organizational Behavior and Human Decision Processes. Vol 151. 2019. Pp 90–103.
Wu, Jane. Paeng, Erin. Linder, Kari, etc. “Artificial Intelligence for Human-Robot Interaction”. AAAI Publications Fall Symposium Series. 2016. https://www.aaai.org/ocs/index.php/FSS/FSS16/paper/viewPaper/14118
Paul Robinette, Wenchen Li, Robert Allen, Ayanna M. Howard and Alan R. Wagner. “Overtrust of Robots in Emergency Evacuation Scenarios”, ACM/IEEE International Conference on Human-Robot Interaction. 2016.
Karen Hao, “Self-driving cars may be more likely to hit you if you have dark skin”, The Download. March 1, 2019. https://www.technologyreview.com/the-download/613064/self-driving-cars-are-coming-but-accidents-may-not-be-evenly-distributed/
De-Artega, Maria, Romanov Alexey. Wallach Hannah, etc. “Bias in Bios: A Case Study of Semantic Representation Bias in a High Stakes Setting”. Association for Computing Machinery. 2019. https://www.microsoft.com/en-us/research/uploads/prod/2019/01/bios_bias.pdf
O’ Niel, Cathy. “Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy”. New York Crown. 2016. Introduction.
Shadowen, Ashley. “Ethics and Bias in Machine Learning: A Technical Study of What Makes Us ”Good””. CUNY Academic Works. 2017.
Bhargava, Rahul. “The Algorithms Aren’t Biased, We Are”. MIT Media Lab (Medium). 2018. https://medium.com/mit-media-lab/the-algorithms-arent-biased-we-are-a691f5f6f6f2
Hao, Karen. ”This is how AI bias really happens — any why it’s so hard to fix”. MIT Technology Review (Medium). February 2019. https://www.technologyreview.com/s/612876/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix/
Suresh, Harini and Guttag, John. “A Framework for Understanding Unintended Consequences of Machine Learning”. Association for the Advancement of Artificial Intelligence. 2019.
Dastin, Jeffrey. “Amazon scraps secret AI recruiting tool that showed bias against women”. Reuters (Business News). October 2018. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G
Noble, Safiya. “Algorithms of Oppression: How Search Engines Reinforce Racism”. New York University Press. 2018.
Hao, Karen. “Why AI is a threat to democracy — and what we can do to stop it”. MIT Technology Review. February 2019. https://www.technologyreview.com/s/613010/why-ai-is-a-threat-to-democracyand-what-we-can-do-to-stop-it/
Campolo, Alex. Sandfilippo, Madelyn. Whittaker, Meredith. Crawford, Kate. “AI Now 2017 Report”. AI Now Institute. 2017.
William Frey, Patton Desmond, Gaskell Michael. “Artificial Intelligence and Inclusion: Foremerly Gang-Involved Youth as Domain Experts for Analyzing Unstructured Twitter Data”. SAGE Journals. 2018. https://journals.sagepub.com/doi/10.1177/0894439318788314
William Frey, Patton Desmond, Gaskell Michael. “Artificial Intelligence and Inclusion: Foremerly Gang-Involved Youth as Domain Experts for Analyzing Unstructured Twitter Data”. SAGE Journals. 2018. https://journals.sagepub.com/doi/10.1177/0894439318788314
AI Fairness 360 Open Source Toolkit. IBM Research Trusted AI. https://aif360.mybluemix.net/
Raji, I and Buolamwini, J. “Actionalbe Audition: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products.” Conference on AI, Ethics, and Society. 2019.