Making sense of the FDA AI/ML List

This text explains why it is not easy to interpret and contextualize the list of 510k clearances for AI/ML products. To do this, one needs to have a lot of domain knowledge, such as: the SAMD regulatory process, the market for predictive CDS and imaging AI, the principles of product strategy, and the data cleansing/pruning techniques.

Some aspects of the list that need to be considered are:

The list is based on someone's or group's analysis of the 510k clearances in the CDRH's records. Not all 510k submissions have a clear indication that they use machine learning in their design and function. The FDA says that the list is "[… is] primarily based on information provided in the summary descriptions of their marketing authorization document."

• Implications: The list may include some products that are not really AI/ML products, but rather enable or interface with AI applications. For example, some of the PACS and PACS viewer products listed may integrate AI applications into radiologist workflow but are not AI/ML products themselves. Similarly, some of the device accessory products may have some ML processing in their offerings, but are not necessarily AI "products" per se.

The list is not up to date, as it is derived from public records (e.g.: the CDRH 510k portal), which have a lag of several months. The current publication only covers AIML submissions until the end of July 2023.

• Implications: The list may not reflect the current market situation, as there may be many more submissions cleared since then and before the list is released by FDA (end of October/early November). This gap may understate some of the faster growing vendors. For example, this year, Qure.AI and Annalise.ai may have more clearances than what the list shows.

The list shows the 510k clearances that the Agency has approved for the submissions from manufacturers. However, this does not mean that the list contains all the 'products' that are available in the market. A manufacturer may obtain a 510k clearance for a certain product or capability, but may not actually sell or use it afterwards.

• Implications: Therefore, there may be some 510k clearances on the list that do not correspond to any product or capability that is being sold or used!

One way that medical device manufacturers can expand the capabilities of their products is by filing and obtaining clearance for a new indication from the FDA. This means that the product can perform a new function that was not previously approved. For example, let's look at the imaging AI product codes. Some of these products (QAS, QBS, etc.) can analyze an imaging study and alert the relevant users that the AI model detected something on that study that needs their attention. However, these products cannot mark the location of the finding on the image (e.g.: with a box, an arrow, or a label). To do that, they need a different clearance for a "Computer Aided Detection" and/or Diagnosis product code (e.g.: MYN, Q…). This would allow them to offer detection and localization of a finding, which might be more useful for the users of the software (e.g.: radiologists). When they get this clearance, they can update their existing product (e.g.: Joe Co's AI Lesion-o-Matic) with the new functionality. This is not a new product, but an upgrade of the same product. In contrast, some large vendors would not update their existing applications (e.g.: GE AW, Siemens Syngo) but rather replace them with new ones. This is primarily due to product lifecycle considerations, but it also logically doesn’t make sense to keep the older version in the catalog if the same functionality exists in the newer version of the Head App, especially now that there are ML-enabled segmentations.

• Implications: Because many of the products on the list are for expanded claims, one can count the 'products' available to the market.

This is a list of 510k product clearances that grant the manufacturer the ability to start marketing and selling the specific product to the US healthcare market. BTW, the manufacturer can only market and promote the arising product described by the 510k clearance letter, per the indications and constraints granted by the particular product and its product code.

• Implications: For instance, an AI application that has been approved as a Computer Aided Prioritization software should not be marketed by the manufacturer as a tool for disease classification, diagnostic suggestions, and so on.

Given that this list is compiled annually by or for the FDA, there’s a possibility that the filtering and search criteria may vary from year to year. Consequently, products cleared in previous years may not be included in the 2021 or 2022 lists (even though they were cleared in 2018, 2019, or 2020), but may appear in subsequent lists. This was observed in the 2023 list.

• Implications: There may be annual fluctuations in the baseline numbers of cleared submissions that incorporate AI/ML.

The manufacturer’s name listed on the 510k may vary from one submission to another. This could be intentional due to changes in the corporate legal entity, or unintentional due to typos or minor variations. For instance, if you were to sort, list, filter, and count the list, most analysis tools would identify “AI Is Great Ltd Beijing Co” and “AI is Great Ltd. Beijing Co.” as two separate manufacturers, even though they are the same entity. This could occur within the same year or across different years. Large companies often have multiple legal entities and submit 510ks under these various entities. For example, “GE Healthcare” may appear under 12 different names in the manufacturer column, such as “GE Medical Systems”, “GE Healthcare LLC”, “GE Precision Health”, and so on.

• Implications: Exercise caution and diligence when analyzing or interpreting analyses of the list, particularly when it involves aggregations of manufacturers and products.

The CDRH evaluates new submissions in ‘panels’ that best represent the clinical and technical characteristics of the intended application/product resulting from the clearance. However, this categorization is not always straightforward for the agency. Occasionally, it may be appropriate for a product to be reviewed by a panel different from its healthcare application area. This is particularly true for dental products on the AI/ML list. Since dental algorithms utilize data captured by X-Ray systems, they are reviewed by the Radiology panel at CDRH, despite being part of a different healthcare sector.

• Implications - There may be an overrepresentation of certain products in specific clinical domains, such as Radiology, due to the inclusion of dental products.

Product Code Categorization: The primary product code assigned to the submission may change from the manufacturer’s initial submission to clearance and then post-market. This is reflected in the database by the presence of “subsequent product codes”. For instance, many of the 西门子医疗 AI Rad Companion submissions have a primary product code as a modality type or accessory (e.g., JAK for CT), but then a subsequent product code of “QIIH” (automated radiologic processing). Understanding how, why, and when the subsequent codes were introduced is typically not straightforward without a FOIA request to review the documentation. The introduction of subsequent codes can be requested by the manufacturer (even in the original submission), added after initial clearance, introduced by the Agency, and also applies when a product (like patient monitors) aggregates many sensors and thus physiologic data types.

• Implication: Be aware of these additional tags and the function of the resulting product when categorizing the AI/ML application as a type of AI product.