Forskning
Udskriv Udskriv
Switch language
Region Hovedstaden - en del af Københavns Universitetshospital
E-pub ahead of print

Generalizability of machine learning for classification of schizophrenia based on resting-state functional MRI data

Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

Harvard

APA

CBE

MLA

Vancouver

Author

Cai, Xin-Lu ; Xie, Dong-Jie ; Madsen, Kristoffer H ; Wang, Yong-Ming ; Bögemann, Sophie Alida ; Cheung, Eric F C ; Møller, Arne ; Chan, Raymond C K. / Generalizability of machine learning for classification of schizophrenia based on resting-state functional MRI data. I: Human Brain Mapping. 2019.

Bibtex

@article{dc0f4cd1e7194f40ba13337d512b3720,
title = "Generalizability of machine learning for classification of schizophrenia based on resting-state functional MRI data",
abstract = "Machine learning has increasingly been applied to classification of schizophrenia in neuroimaging research. However, direct replication studies and studies seeking to investigate generalizability are scarce. To address these issues, we assessed within-site and between-site generalizability of a machine learning classification framework which achieved excellent performance in a previous study using two independent resting-state functional magnetic resonance imaging data sets collected from different sites and scanners. We established within-site generalizability of the classification framework in the main data set using cross-validation. Then, we trained a model in the main data set and investigated between-site generalization in the validated data set using external validation. Finally, recognizing the poor between-site generalization performance, we updated the unsupervised algorithm to investigate if transfer learning using additional unlabeled data were able to improve between-site classification performance. Cross-validation showed that the published classification procedure achieved an accuracy of 0.73 using majority voting across all selected components. External validation found a classification accuracy of 0.55 (not significant) and 0.70 (significant) using the direct and transfer learning procedures, respectively. The failure of direct generalization from one site to another demonstrates the limitation of within-site cross-validation and points toward the need to incorporate efforts to facilitate application of machine learning across multiple data sets. The improvement in performance with transfer learning highlights the importance of taking into account the properties of data when constructing predictive models across samples and sites. Our findings suggest that machine learning classification result based on a single study should be interpreted cautiously.",
keywords = "generalizability, machine learning, reproducibility, schizophrenia spectrum disorders",
author = "Xin-Lu Cai and Dong-Jie Xie and Madsen, {Kristoffer H} and Yong-Ming Wang and B{\"o}gemann, {Sophie Alida} and Cheung, {Eric F C} and Arne M{\o}ller and Chan, {Raymond C K}",
note = "{\circledC} 2019 The Authors. Human Brain Mapping published by Wiley Periodicals, Inc.",
year = "2019",
month = "10",
day = "1",
doi = "10.1002/hbm.24797",
language = "English",
journal = "Human Brain Mapping",
issn = "1065-9471",
publisher = "John/Wiley & Sons, Inc. John/Wiley & Sons Ltd",

}

RIS

TY - JOUR

T1 - Generalizability of machine learning for classification of schizophrenia based on resting-state functional MRI data

AU - Cai, Xin-Lu

AU - Xie, Dong-Jie

AU - Madsen, Kristoffer H

AU - Wang, Yong-Ming

AU - Bögemann, Sophie Alida

AU - Cheung, Eric F C

AU - Møller, Arne

AU - Chan, Raymond C K

N1 - © 2019 The Authors. Human Brain Mapping published by Wiley Periodicals, Inc.

PY - 2019/10/1

Y1 - 2019/10/1

N2 - Machine learning has increasingly been applied to classification of schizophrenia in neuroimaging research. However, direct replication studies and studies seeking to investigate generalizability are scarce. To address these issues, we assessed within-site and between-site generalizability of a machine learning classification framework which achieved excellent performance in a previous study using two independent resting-state functional magnetic resonance imaging data sets collected from different sites and scanners. We established within-site generalizability of the classification framework in the main data set using cross-validation. Then, we trained a model in the main data set and investigated between-site generalization in the validated data set using external validation. Finally, recognizing the poor between-site generalization performance, we updated the unsupervised algorithm to investigate if transfer learning using additional unlabeled data were able to improve between-site classification performance. Cross-validation showed that the published classification procedure achieved an accuracy of 0.73 using majority voting across all selected components. External validation found a classification accuracy of 0.55 (not significant) and 0.70 (significant) using the direct and transfer learning procedures, respectively. The failure of direct generalization from one site to another demonstrates the limitation of within-site cross-validation and points toward the need to incorporate efforts to facilitate application of machine learning across multiple data sets. The improvement in performance with transfer learning highlights the importance of taking into account the properties of data when constructing predictive models across samples and sites. Our findings suggest that machine learning classification result based on a single study should be interpreted cautiously.

AB - Machine learning has increasingly been applied to classification of schizophrenia in neuroimaging research. However, direct replication studies and studies seeking to investigate generalizability are scarce. To address these issues, we assessed within-site and between-site generalizability of a machine learning classification framework which achieved excellent performance in a previous study using two independent resting-state functional magnetic resonance imaging data sets collected from different sites and scanners. We established within-site generalizability of the classification framework in the main data set using cross-validation. Then, we trained a model in the main data set and investigated between-site generalization in the validated data set using external validation. Finally, recognizing the poor between-site generalization performance, we updated the unsupervised algorithm to investigate if transfer learning using additional unlabeled data were able to improve between-site classification performance. Cross-validation showed that the published classification procedure achieved an accuracy of 0.73 using majority voting across all selected components. External validation found a classification accuracy of 0.55 (not significant) and 0.70 (significant) using the direct and transfer learning procedures, respectively. The failure of direct generalization from one site to another demonstrates the limitation of within-site cross-validation and points toward the need to incorporate efforts to facilitate application of machine learning across multiple data sets. The improvement in performance with transfer learning highlights the importance of taking into account the properties of data when constructing predictive models across samples and sites. Our findings suggest that machine learning classification result based on a single study should be interpreted cautiously.

KW - generalizability

KW - machine learning

KW - reproducibility

KW - schizophrenia spectrum disorders

UR - http://www.scopus.com/inward/record.url?scp=85073996466&partnerID=8YFLogxK

U2 - 10.1002/hbm.24797

DO - 10.1002/hbm.24797

M3 - Journal article

JO - Human Brain Mapping

JF - Human Brain Mapping

SN - 1065-9471

ER -

ID: 58060515