TY - JOUR
T1 - Scanner agnostic large-scale evaluation of MS lesion delineation tool for clinical MRI
AU - Hindsholm, Amalie Monberg
AU - Andersen, Flemming Littrup
AU - Cramer, Stig Præstekjær
AU - Simonsen, Helle Juhl
AU - Askløf, Mathias Gæde
AU - Magyari, Melinda
AU - Madsen, Poul Nørgaard
AU - Hansen, Adam Espe
AU - Sellebjerg, Finn
AU - Larsson, Henrik Bo Wiberg
AU - Langkilde, Annika Reynberg
AU - Frederiksen, Jette Lautrup
AU - Højgaard, Liselotte
AU - Ladefoged, Claes Nøhr
AU - Lindberg, Ulrich
N1 - Copyright © 2023 Hindsholm, Andersen, Cramer, Simonsen, Askløf, Magyari, Madsen, Hansen, Sellebjerg, Larsson, Langkilde, Frederiksen, Højgaard, Ladefoged and Lindberg.
PY - 2023
Y1 - 2023
N2 - INTRODUCTION: Patients with MS are MRI scanned continuously throughout their disease course resulting in a large manual workload for radiologists which includes lesion detection and size estimation. Though many models for automatic lesion segmentation have been published, few are used broadly in clinic today, as there is a lack of testing on clinical datasets. By collecting a large, heterogeneous training dataset directly from our MS clinic we aim to present a model which is robust to different scanner protocols and artefacts and which only uses MRI modalities present in routine clinical examinations.METHODS: We retrospectively included 746 patients from routine examinations at our MS clinic. The inclusion criteria included acquisition at one of seven different scanners and an MRI protocol including 2D or 3D T2-w FLAIR, T2-w and T1-w images. Reference lesion masks on the training (n = 571) and validation (n = 70) datasets were generated using a preliminary segmentation model and subsequent manual correction. The test dataset (n = 100) was manually delineated. Our segmentation model https://github.com/CAAI/AIMS/ was based on the popular nnU-Net, which has won several biomedical segmentation challenges. We tested our model against the published segmentation models HD-MS-Lesions, which is also based on nnU-Net, trained with a more homogenous patient cohort. We furthermore tested model robustness to data from unseen scanners by performing a leave-one-scanner-out experiment.RESULTS: We found that our model was able to segment MS white matter lesions with a performance comparable to literature: DSC = 0.68, precision = 0.90, recall = 0.70, f1 = 0.78. Furthermore, the model outperformed HD-MS-Lesions in all metrics except precision = 0.96. In the leave-one-scanner-out experiment there was no significant change in performance (p < 0.05) between any of the models which were only trained on part of the dataset and the full segmentation model.CONCLUSION: In conclusion we have seen, that by including a large, heterogeneous dataset emulating clinical reality, we have trained a segmentation model which maintains a high segmentation performance while being robust to data from unseen scanners. This broadens the applicability of the model in clinic and paves the way for clinical implementation.
AB - INTRODUCTION: Patients with MS are MRI scanned continuously throughout their disease course resulting in a large manual workload for radiologists which includes lesion detection and size estimation. Though many models for automatic lesion segmentation have been published, few are used broadly in clinic today, as there is a lack of testing on clinical datasets. By collecting a large, heterogeneous training dataset directly from our MS clinic we aim to present a model which is robust to different scanner protocols and artefacts and which only uses MRI modalities present in routine clinical examinations.METHODS: We retrospectively included 746 patients from routine examinations at our MS clinic. The inclusion criteria included acquisition at one of seven different scanners and an MRI protocol including 2D or 3D T2-w FLAIR, T2-w and T1-w images. Reference lesion masks on the training (n = 571) and validation (n = 70) datasets were generated using a preliminary segmentation model and subsequent manual correction. The test dataset (n = 100) was manually delineated. Our segmentation model https://github.com/CAAI/AIMS/ was based on the popular nnU-Net, which has won several biomedical segmentation challenges. We tested our model against the published segmentation models HD-MS-Lesions, which is also based on nnU-Net, trained with a more homogenous patient cohort. We furthermore tested model robustness to data from unseen scanners by performing a leave-one-scanner-out experiment.RESULTS: We found that our model was able to segment MS white matter lesions with a performance comparable to literature: DSC = 0.68, precision = 0.90, recall = 0.70, f1 = 0.78. Furthermore, the model outperformed HD-MS-Lesions in all metrics except precision = 0.96. In the leave-one-scanner-out experiment there was no significant change in performance (p < 0.05) between any of the models which were only trained on part of the dataset and the full segmentation model.CONCLUSION: In conclusion we have seen, that by including a large, heterogeneous dataset emulating clinical reality, we have trained a segmentation model which maintains a high segmentation performance while being robust to data from unseen scanners. This broadens the applicability of the model in clinic and paves the way for clinical implementation.
KW - automatic segmentation algorithm
KW - clinical applicability
KW - clinical dataset
KW - heterogeneous dataset
KW - multi-scanner
KW - multiple sclerosis
KW - white matter lesions (WML)
UR - http://www.scopus.com/inward/record.url?scp=85161041404&partnerID=8YFLogxK
U2 - 10.3389/fnins.2023.1177540
DO - 10.3389/fnins.2023.1177540
M3 - Journal article
C2 - 37274207
SN - 1662-4548
VL - 17
JO - Frontiers in Neuroscience
JF - Frontiers in Neuroscience
M1 - 1177540
ER -