5 Lessons in Governing Clinical AI

The AI Centre is a leader in clinical artificial intelligence for healthcare. Our staff are breaking down barriers and forging new paths for the integration of innovative technology into our healthcare systems. As part of our commitment to sharing and collaborating in our learnings with our peers, Robin Carpenter shares his top five lessons in governing #ClinicalAI.?

Robin is the ethics and governance lead at the AI Centre and has been working tirelessly to ensure that all these ground-breaking AI technologies are subject to robust ethical governance. He is passionate about transparently communicating our data governance and ethics models. The healthcare AI industry is changing at an incredible rate, and ethics and governance policies must keep pace with this.?

Lesson 1: leave your ego at the door

In early 2020 I walked into the AI Centre, ready to hit the ground running. My background is drug trials, the area of research with the strictest regulation, and I’d spent the last decade identifying “good” research from “bad”. Surely the healthcare AI industry could not be much different (spoiler alert: it was).?

The AI industry has created new questions, demanded new standards, and is changing so unbelievably rapidly. I was humbled and forced to learn quickly.?

?

Lesson 2: your peers will hear about your work even if you do not share it

To help wrap my head around the organisation I asked to see a mission statement and was given something simple: we create AI that improves healthcare. Since then our mission has grown and evolved but this remains at the essence of what we do.?

Our mission had ethical intentions, which is an important start, but ethical principles only matter as much as they are put in practice. For example, would it be okay to create AI that improved cancer detection whilst only meeting minimum transparency standards? How about if it was ground-breaking but created by an untrustworthy organisation? Or if there had been no involvement with practicing oncologists? How we carried out research – our governance structure – mattered as much as why we carried out research.??

Enforcing robust, ethical governance meant making decisions to protect our ideals – but doing what you ought to do often creates more work than falling back on what you simply can do. Interestingly, even though our decisions and their repercussions were not being shared publicly, we still gained a reputation for putting research quality first, both in NHS departments and beyond.?

Our decisions and process had become public – without us actively having to share them. It is important for our reputation to resonate with the external individuals and bodies that could prohibit our research – which is often the case in healthcare research due to limited resources. My experience is that they actually prioritise our approval. I have never experienced that before!?

?

Lesson 3: understand the different paradigms in your space

Data is massively political; it can describe a person, be shared with strangers, and generate power – so it is no wonder that people talk about how it is used. In politics people pick sides and, in this industry, the sides are quality versus speed of production.?

The truth is an AI consortium, such as the AI Centre, cannot rely simply on ethical staff. We have principles, yes, but we also have a system that holds us accountable. The main group that holds us accountable is our Data Allocation Committee (DAC), which has been reviewing AI research since I started at the AI Centre. The DAC could have been established in one of two ways:??

Option 1: the quick, legal route:??

• If projects stay within the law, we pass them.?

Option 2: the slower, ethical route:??

• We bring in members of the public and give them the power to change and reject projects.?
• We bring in clinicians to question how beneficial a project would be to the NHS.??
• We create mechanisms that highlight the limitations of the AI.?

Many players of the healthcare technology industry would expect option 1 because it would be the fastest win. In my opinion, it is important to learn about different industry norms and manage stakeholder expectations early on. Be clear that clinical AI research takes a while, that shortcuts don’t work in the long term, and that good AI practices are constantly evolving. We are fortunate that our stakeholders at the AI Centre are united under common values and as such, recognised the benefit of quality over speed. We went for option 2.?

?

Lesson 4: leadership defines culture, culture defines teams, teams create products

The clinical AI industry is new for everyone, even those of us specialising in it, and every opportunity for learning produces value.??

When it came to my own short-sightedness and learning opportunities, I experienced an unanticipated level of support from my managers. Leadership with a clear mission and strong values are essential in instilling the same in their team, and I was impressed by the trust that my managers had in me to pursue ethical governance.?

An unexpected impact of this was that it gave greater depth to my relationship with my team and our work. Mistakes provide opportunity for learnings and test your objectives. You must defend your principles and prove their worth with better pipelines, better solutions, and better AI.??

?

Lesson 5: your pipeline is everchanging

At the conception of the AI centre, we had a pipeline to prototype AI. It quickly became apparent that there was a need for this to expand and cover the whole lifecycle, from product conception to deployment to decommission.??

Over the last few years, we have had difficult and honest conversations with both leadership and the public. On our journey the pipeline has changed a lot - from best practice to public perception and feedback - and I guarantee in the next few months it will change again. To succeed in this ecosystem, you must learn to regularly update your pipeline to adapt to its changing needs.??

?

If you are interested in learning more about our governance policies, you can reach out to us at [email protected].??