What is an algorithm prison?

Our world is getting increasingly connected and automated, which sounds great from a customer experience point of view. We go to a new business, and with a few clicks, they can access our profile and provide hassle-free services. However, this is all hunky dory until we hit an algorithm prison.

Imagine a person who just got out of jail and has a terrible credit score. They cannot get a job because they have a bad credit score, and they cannot improve their credit score because they cannot get a job. Society is designed to not offer any second chances. As long as you are on upward mobility, things are rosy; the moment you stumble, then you can plummet quite quickly in society.

This phenomenon was dramatized in the episode "Nosedive" from the Netflix show Black Mirror. The episode's plot revolves around social credit scores, where people with higher scores get privileged social access. The lead character is desperate to increase her score to be eligible for a house in a posher neighborhood. She had some unfortunate incidents, and her social score keeps plummeting throughout the episode, and there is no way out for her.

Algorithm Prison

"Algorithm prison" is a term that metaphorically describes a situation where individuals or systems become trapped or constrained by the outcomes or decisions generated by algorithms, often without the ability to understand or challenge those outcomes. It implies a scenario where algorithms are rules or instructions followed to solve a problem or make decisions and exert significant control over various aspects of life or society, sometimes leading to unintended consequences or injustices.

Like the actual prison, inside the "algorithm prison," individuals feel powerless, have limited agency, and are restricted by the automated decisions made by algorithms, such as those used in predictive policing, automated hiring systems, or credit scoring. It highlights concerns about transparency, accountability, and fairness in algorithmic decision-making processes, where biases or errors in the algorithms can perpetuate systemic inequalities or unjust outcomes.

The term underscores the importance of critically examining and regulating the use of algorithms in various domains to ensure they serve societal interests and values rather than perpetuating harmful or discriminatory practices. I firmly believe that AI is for Humans and Humans are not for AI. It also emphasizes the need for transparency, accountability, and ethical considerations in designing, deploying, and monitoring responsible algorithmic systems.

What do you think could be done to break algorithms prisons?

If you like, please subscribe to the FAQ on Data newsletter and/or follow Fawad Qureshi on LinkedIn.