Prospective clinical evaluation of automatic lesion assessment in patients with multiple sclerosis

Purpose: To perform a real-world clinical validation of a commercial AI tool for automatic MRI assessment in multiple sclerosis (MS) patients, evaluating its impact on assessment time, workflow, and accuracy in detecting new and enlarging lesions. Methods: We prospectively enrolled MS patients undergoing routine follow-up MRI from September-December 2024. Current and prior MRI examinations were anonymized and assessed independently by four neuroradiologists with and without AI assistance (mdbrain v.4.11.0). Assessment times were recorded, and radiologists completed utility questionnaires. Lesion quantification was compared between radiologist alone, radiologist with AI, and AI alone. Performance metrics including sensitivity, specificity, and predictive values were calculated case-level for detecting new and enlarging lesions. Results: The cohort included 112 MS patients scanned on 8 different MRI scanner models with varying protocols. Mean assessment time was reduced by 27 s when using AI versus without (p = 0.317). Radiologists found AI helpful in 87% of cases and reported difficulties in 11%. AI obtained negative predictive values of 0.89 for detecting new lesions when comparing to assessment without AI. Positive predictive values were low (0.35–0.65) due to false positive tendencies. Conclusion: We prospectively validated an AI tool for MS MRI follow-up in a real-world setting. It showed modest, non-significant time savings and low positive predictive value, limiting research use. High negative predictive value supports triaging potential. Radiologists found the AI tool helpful for lesion counting and detecting small new lesions. Findings highlight the need for thorough clinical evaluation, especially in areas lacking definitive ground truth.