Radiologists More Accurate Detecting Breast Cancer With AI Than Without AI

This is the first study to compare AI’s performance in breast cancer screening alone vs. with assistance from a human radiologist.?

In a large scale study in Germany researchers demonstrated that radiologists working with AI were more accurate detecting breast cancer than radiologists working without AI, and vice versa - the AI was more accurate when working with a radiologist than when working independently. The study was led by Vara, a German company, in collaboration with radiologists at the Essen University Hospital in Germany and the Memorial Sloan Kettering Cancer Center in New York. Vara's AI has been used by radiologists in Germany for two years and is used in 30% of Germany’s breast cancer screening centers. Vara's AI software is also used to screen for breast cancer in Mexico and Greece.

This is the first study to compare AI’s performance in breast cancer screening alone vs. with assistance from a human radiologist. This type of human/AI collaboration is expected to improve accuracy of diagnosis, help detect breast cancer earlier, and improve survival rates. A paper on the study entitled Combining the strengths of radiologists and AI for breast cancer screening: a retrospective analysis was published in The Lancet Digital Health on July 1, 2022.

Study Overview

• The researchers tested two approaches and compared the performance using the same dataset of 1,193,197?mammograms that were collected from 8 screening sites in Germany.
• Approach #1 - The AI worked independently to analyze mammograms.
• Approach #2 - The AI automatically distinguished between scans that it assessed as "normal" and "not normal". The AI referred the "not normal" scans to a radiologist. The radiologist reviewed the "not normal" scans before looking at the AI’s assessment. The AI would then issue a warning if it detected cancer when the doctor did not.
• The results from both approaches were compared with the decisions that radiologists made on 82,851 mammograms sourced from screening centers.

3 Possible Screening Pathways

Image source: Combining the strengths of radiologists and AI for breast cancer screening: a retrospective analysis

Training the Neural Network

1. Vara trained the AI with data from over 367,000 mammograms
2. The data included radiologists’ notes, original assessments, and information on whether the patient ultimately had cancer.
3. The objective was to train the AI how to classify the scans as one of the following

• confident normal
• not confident (unknown)
• confident cancer

Subgroup Performance

Subgroup performance on sensitivity at exemplary configuration on external-test data. Image source: Combining the strengths of radiologists and AI for breast cancer screening: a retrospective analysis

Findings

• The AI correctly classified 63% of the scans as "confident normal".
• 3/4 of the screening studies didn’t need to be reviewed by a radiologist and improved overall accuracy.
• Approach #1 (AI in standalone mode) achieved a sensitivity of 84% and a specificity of 89% but was less accurate than the average unaided radiologist.
• Approach #2 (a radiologist and AI working together) was 2.6% more accurate at detecting breast cancer than a doctor working alone, and fewer false positives.
• The simulated decision-referral approach significantly improved upon radiologist sensitivity by 2.6 percentage points and specificity by 1.0 percentage points surpassing radiologist performance.
• The decision-referral approach also yielded significant increases in sensitivity for a number of clinically relevant subgroups, including subgroups of small lesion sizes and invasive carcinomas. (see subgroup performance image)
• Sensitivity of the decision-referral approach was consistent across 8 screening sites and 3 different device manufacturers. (see subgroup performance image)

Conclusions

• The decision-referral approach leverages the strengths of both the radiologist and AI, demonstrating improvements in sensitivity and specificity surpassing that of the individual radiologist and of the standalone AI system.
• The decision-referral approach has the potential to improve the screening accuracy of radiologists, is adaptive to the requirements of screening, and could allow for the reduction of workload ahead of the consensus conference, without discarding the generalized knowledge of radiologists.
• The researchers confirmed consistent and improved performance of the decision-referral approach across clinically relevant subgroups as well, including those presenting as challenging cases for radiologists.
• An AI model, if deployed in clinical practice, also has the potential to be further improved by undergoing training on newly incoming data, ensuring that performance on all subgroups does not degrade.

Dataset Partitions

Subscribe, Comment, Join Group

I'm interested in your feedback - please leave your comments.

To subscribe to the AI in Healthcare Milestones newsletter please click here.

To join the AI in Healthcare Milestones Group please click here.

Copyright ? 2022 Margaretta Colangelo. All Rights Reserved.

This article was written by Margaretta Colangelo. Margaretta is a leading AI analyst who tracks significant milestones in AI in healthcare. She consults with AI healthcare companies and writes about some of the companies she consults with. Margaretta serves on the advisory board of the AI Precision Health Institute at the University of Hawai?i?Cancer Center @realmargaretta