Hidden Bias: How Might Healthcare Systems Achieve AI Equity?

When AI Falls Short: The Ethical Gaps in Healthcare Systems

Picture a device used in hospitals and homes to measure our health. Now, imagine this: the same device might not work as well for some people simply because of their skin color. This isn’t a hypothetical scenario—it’s the reality of pulse oximeters, and it highlights the ethical challenges we face as we integrate AI into healthcare.

What is Ethical AI?

Ethical AI refers to the development and deployment of AI systems that align with human values and ethical principles. In healthcare, this means ensuring AI is fair, transparent, accountable, and inclusive. Key principles include:

• Fairness: AI should work equally well for all patients, regardless of race, gender, or socioeconomic status.
• Transparency: AI decisions should be explainable, avoiding "black box" algorithms that obscure how conclusions are reached.
• Accountability: Clear mechanisms must be in place to address errors or biases in AI systems.
• Inclusivity: AI must be designed with diverse datasets and input from marginalized communities.

AI has the power to change medicine for the better, but it can also create unfair gaps in care. This edition looks at why we need ethical AI in healthcare, starting with the example of pulse oximeters and their bias. We’ll look at how AI can create unfairness and share ways to make sure AI helps everyone, not just a few.

The Pulse Oximeter Problem: A Case Study in Bias

Pulse oximeters, devices that measure blood oxygen levels, have been found to provide less accurate readings for patients with darker skin tones. A 2020 study in?The New England Journal of Medicine?revealed that pulse oximeters were?three times more likely to miss low oxygen levels in Black patients compared to White patients. In some cases, oxygen levels were overestimated by?up to 8%?for patients with darker skin, leading to delayed or inadequate treatment.

In January 2025, the FDA proposed new guidelines to improve the accuracy of pulse oximeters for all skin tones. This move underscores the urgent need to address racial bias in medical devices, especially as AI is increasingly integrated into these tools.

Why does this matter?

• Clinical Impact: Inaccurate readings can delay critical care, particularly for patients with conditions like COVID-19 or heart failure.
• Trust Erosion: Patients and clinicians may lose confidence in AI-enhanced devices if biases are not addressed.
• Regulatory Pressure: Governments are tightening oversight to ensure medical devices are equitable.

How AI Can Make Bias Worse

The pulse oximeter issue is just one example of how AI can exacerbate bias. In October 2024, 13 hospitals in Philadelphia discontinued the use of race-based algorithms in clinical decision-making. These algorithms, which adjusted risk scores based on race, were found to?systematically underestimate the health needs of Black patients.

This decision echoes findings from a 2019 study published in?Science, which revealed that a widely used commercial algorithm exhibited significant racial bias. The study found that?Black patients were significantly sicker than White patients at the same risk score, yet they were?less likely to be flagged for high-risk care management. Specifically, the algorithm’s bias meant that only?17.7% of Black patients?were identified for additional help, compared to what would have been?46.5%?if the algorithm were unbiased.

The bias in both cases arose because the algorithms relied on proxies like healthcare costs or race, which fail to account for systemic inequities. For example, less money is often spent on Black patients due to unequal access to care, leading algorithms to underestimate their health needs.

Implications for Healthcare Leaders:

• Patient Care: Biased AI can lead to unequal treatment and worse outcomes for marginalized groups.
• Reputation: Hospitals and clinics risk losing trust if their tools are perceived as unfair.
• Regulatory and Legal Risks: Non-compliance with ethical AI standards could result in fines or lawsuits.

How to Make AI Fair: Steps Forward

To address these challenges, healthcare systems must adopt robust ethical frameworks and practical strategies. Here are some actionable steps:

1. Diverse Datasets: Use data that represents all patient populations, including racial, gender, and socioeconomic diversity. Partner with underserved communities to collect inclusive data.
2. Explainable AI (XAI): Implement AI systems that provide clear explanations for their decisions, avoiding "black box" models. Use tools like SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations) to make AI decisions transparent.
3. Rigorous Testing: Test AI systems across diverse populations to identify and correct biases. Establish ongoing monitoring to ensure AI tools remain equitable over time.
4. Regulatory Compliance: Adhere to guidelines from organizations like the FDA, WHO, and CDC. Advocate for stronger policies to ensure AI fairness in healthcare.

The Human Touch: Ensuring Equity and Empathy

While AI can enhance healthcare delivery, it cannot replace the human element. A December 2024 article in?Vanity Fair?highlighted how some insurance companies are using AI to automate claim denials, leaving patients feeling like mere numbers. This underscores the importance of maintaining empathy and equity in AI-driven healthcare.

What can hospitals and clinics do?

• Training: Educate healthcare professionals to recognize and mitigate AI bias. For example, workshops on interpreting AI outputs and identifying potential biases.
• Patient Advocacy: Ensure patients understand how AI is used in their care and provide avenues for feedback.
• Collaboration: Foster partnerships between clinicians, technologists, and ethicists to design AI tools that prioritize patient well-being.

What’s Next: Building a Fairer Future

Achieving AI equity in healthcare requires a multi-stakeholder approach. Here’s how we can move forward:

1. Emerging Technologies: Explore AI tools that prioritize fairness, such as federated learning (which trains models on decentralized data without compromising privacy). Invest in AI systems that continuously learn and adapt to reduce bias over time.
2. Policy Initiatives: Support legislation that mandates transparency and accountability in AI systems. Advocate for funding to research and address AI bias in healthcare.
3. Financial Impact: Biased AI can lead to?increased costs?and?wasted resources?due to misdiagnoses or inefficient care. Investing in equitable AI can improve outcomes and reduce long-term healthcare expenses.

Conclusion: Let’s Work Together

AI has the potential to revolutionize healthcare, but it also carries significant risks. The pulse oximeter disparity and race-based algorithms are stark reminders that we must prioritize?fairness, transparency, and inclusivity?in AI systems. By working together—clinicians, technologists, policymakers, and patients—we can build a future where AI serves?all equitably.

Let’s ensure AI in healthcare is not just?innovative, but also?just, inclusive, and human-centered.

Next Edition Preview

In our next edition,?"Human-Centered AI – Bridging the Gap Between Technology & Clinicians", we’ll explore how AI can?support, not replace, clinicians. We’ll highlight real-world examples of AI tools that enhance physician workflows, discuss strategies to build trust between clinicians and AI, and examine the challenges of integrating AI into healthcare.

Stay tuned for insights on bridging the gap between cutting-edge technology and the human expertise that defines great patient care.

?? Innovating Together for Real-World Impact

?? #AIEquity #EthicalAI #HealthcareFairness #AIinHealthcare #HealthTech #MedicalEthics #InclusiveAI #PatientCare #HealthInnovation #AIforGood #FutureOfHealthcare