ECR 2024, Vienna: To AI or not to AI? That’s the question.

I always enjoy visiting Vienna. On my free afternoon, I was able to snack on a bratwurst from a street vendor, ate a Wiener Schnitzel for dinner, walked through the historic center, and enjoyed a slice of Sacher chocolate cake. In addition, I also saw an opera performance at night at the Vienna Opera House which was awesome.

This year's annual European Radiology Meeting (ECR) drew close to 20,000 visitors, which is about half of the attendance of its annual US counterpart (RSNA) in Chicago. Its theme was the future of medical imaging, i.e. the next generation of Radiology, of which AI took center stage. There were about 50 AI vendors represented, which is about half of what we could see in Chicago. Unlike prior years, most of the AI vendors had clearance to market their product in the European market, (CE mark) and quite a few had the US FDA clearance as well, although many of them that were exhibiting had not (yet) entered the US market.

This year's big question is whether AI technology is ready to be deployed, whether there is a sufficient business case and/or clinical evidence justifying it, and more importantly, is the healthcare user community ready to adopt it. The presentations and discussions at the conference did not give yet a clear "yes or no" answer. Vendors presented an optimistic picture showing their many deployments worldwide and the impact that their AI technology has made, and most of that is true. However, as can be expected, it is like buying a car, every car salesman or woman will tell you that their car is the best, but it takes a test drive to find out if it meets expectations and requirements.

These are the most important challenges with AI deployment today that I have observed:

·?????? AI workflow integration: AI vendors haven’t quite figured out how the AI fits optimally in the clinical workflow. Several AI companies displayed their results in a separate viewer and nice GUI with a dashboard allowing physicians to correct the AI findings and perform QA analysis. Many of them have bells and whistles that are over-engineered: a physician does not want a separate viewer or have to do anything different when using the AI, he or she wants it to be invisible and hidden in the background.

·?????? AI infrastructure integration: AI requires changes in routing tables, and status exchanges between the many parts that are involved which could require additional middleware and training of support teams and local IT and PACS administrators. For example, a study might need to be “cached” and only displayed on a worklist till an AI result comes back from the AI application. This could take some time if the study needs to be de-identified and sent to the cloud for processing before the result comes back. One user told me that his PACS would remove a study from a radiology reading worklist after the AI result sent as a Secondary Capture for this study was received by the PACS requiring another “complete” action by the technologist.

·?????? AI lack of onboarding with vendors: If you are a new AI vendor and like to integrate it with either a PACS vendor or an AI platform, I wish you good luck. I heard from several new entrants that they have a hard time getting any face-time with the onboarding teams at those companies which creates a chicken-and-egg problem: If you are not available on their marketplace or AI platform, you can’t get in front of a customer.

·?????? AI result encoding and standards use are sub-optimal: The AI output is often poorly implemented. Most AI algorithms create a so-called DICOM Secondary Capture (a copy of the image with burned-in annotations) which is done as pretty much every PACS supports this, therefore it provides a least common denominator. Some vendors create a DICOM Encapsulated PDF which is supported by most PACS systems as well. The problem with these implementations is that it is inefficient (creating another copy) but more importantly, the findings are not available in a format that could be easily ingested by a reporting system. A much better solution is exchanging only the image annotations in the form of a DICOM Presentation State (GSPS, modality PR) or even better, as a DICOM Structured Report (SR) containing the encoded findings. The recommendation by the IHE committee is to use a standard template identified as TID 1500. However early implementations of this SR have shown so many different implementations that the receivers have to customize the code for each different AI creator. Additional recommendations by standards committees or a “de facto” standard creation by one of the major vendors for the SR is an urgent issue to be addressed.

·?????? The AI data input is often “dirty”: Routing and selecting the right images to be processed by the AI is a challenge. Unfortunately, the DICOM metadata (aka “header”) is not always reliable. This is especially true if there is little control over the source of the data, e.g. if it originates from multiple sites and/or modalities. Study and Series descriptions, Codes for anatomy and modality can be missing, inconsistent, or even wrong. An AI algorithm that processes Mammo X-rays will likely flag an error if it receives an Ultrasound, but it has no idea if it missed a study that was misidentified and simply did not get routed to the AI. There is a need for the AI software to inspect the pixel data to make sure that the metadata reflects the image. For example, a stroke AI looking at CT images of the head might not want to have the scout views routed to its application. A few vendors are working on this but more work needs to be done.

·?????? Security issues have to be addressed: The FDA has recently strengthened the security requirements of medical software. A security plan has to be submitted as part of the FDA filing, and penetration testing has to be done to prove that the system is robust and will withstand cyber-attacks and test results submitted. In addition, the hospital IT department will perform a security assessment which might take a minimum of 6 months if not more. It is not unusual for a hospital to require the AI to be plugged into an existing AI platform, this has the advantage that a lot of the vetting of the application is done by the AI platform vendor. Using an AI platform will benefit both the hospital as well as the AI vendor as there are only a handful of popular platforms to be integrated with vs potentially thousands of individual hospitals.

·?????? Privacy mechanisms have to be implemented: Many of the AI applications are hosted in the cloud and connected using a public network. The hospital is ultimately responsible for maintaining patient privacy which is typically achieved by anonymizing the patient data. Anonymizing is non-trivial as it often has to be configured depending on the application Some AI algorithms require different patient characteristics to be non-anonymized. In addition, there could be “hidden” PHI in the company's private header Attributes, hidden in free text fields and/or user-entered text fields, or even part of the pixel data. The effectiveness of the anonymization has to be tested using actual hospital test data. The anonymization can be done by either an AI platform or PACS AI marketplace or by the algorithm itself. In the latter case, the anonymization might need to be run on-premises.

·?????? AI Return on investment is still a big barrier. When there is no reimbursement from insurance when using the AI the investment might be hard to justify from a financial perspective. This is slowly changing but there are only a few use cases right now that have an actual AI procedure code associated that can be used to get paid. A source of funding could be insurers, which happened in the UK where the NHS committed ￡21m (about $26.5m) last year to deploy AI. In contrast, imagine HHS in the US allocating $130m (which is about the same amount per citizen), this is politically just not going to happen. This shows how the deployment opportunities vary significantly based on the country and its specific healthcare provider structure.

·?????? Incentives and potential benefits for AI are different for each region: Europe has about the same amount of countries (50) as there are states in the continental US (49). Imagine each US state having a different language, some more than one, and more than half (29) having a different currency. But most importantly for each state, the healthcare ranges from being almost free as paid for by the government to a combination of public and private. To illustrate its different landscape, a recent initiative of the NHS is to have all of the radiology scans reported within 4 weeks. In my experience, most US hospitals have a turnaround time of 30 minutes if not sooner, and most outpatient clinics 24 hours. AI for the UK to detect critical findings can therefore have a major impact just based on long turn-around time which would not apply to the US. Many more differences make AI deployment more or less attractive depending on the region.

·?????? Vendor longevity is a concern: Will an AI provider be around in a few years? The initial excitement by investors who were pouring money into AI startups is waning, A few companies reported a second round of financing between $5m and $10m, but overall, the initial excitement is gone. AI companies are consolidating, merging, and being acquired. There will certainly be several companies in the exhibit halls today that you might not see anymore over the next few years.

·?????? How to determine the AI efficacy? It takes a test drive to find out if a particular AI solution meets the expectations, however, a physician does not have the time and effort to compare multiple solutions. Unfortunately, there are few resources and papers available to find out how different vendors compare and perform clinically. Performance is characterized using training data sets that vary. As an analogy, it would be comparing a car’s performance in its off-road driving capability with another one that has only been tested on highway driving. There is no question that the lack of transparency creates “angst” for physicians as they have no yardstick to determine which algorithm to select.

·?????? What does the AI miss? Missing incidental findings is still a concern if an AI algorithm is used to eliminate negative findings such as when used for screening. Imagine using an AI application that is trained only to detect fractures and therefore misses another condition that could be critical. A solution would be to use an algorithm that looks for multiple conditions, for example, some vendors claim to test for 50+ conditions in a chest radiograph. But how many findings are enough knowing that there are rare conditions that only would be detected by having a human eye look at the image?

In conclusion, assuming that a particular AI deployment is well integrated into the hospital infrastructure, seamlessly embedded in the physician workflow, is clinically superior among its peers, meets security and privacy requirements, and can be justified with a sound business case, it makes sense to deploy it. However, in many cases, one or more of these conditions might not quite (yet) be satisfied in which case the question of “to AI or not to AI” might be answered negatively.

Herman Oosterwijk, "the AI-Guy"