Forskning
Udskriv Udskriv
Switch language
Rigshospitalet - en del af Københavns Universitetshospital
Udgivet

Expert sampling of VR simulator metrics for automated assessment of mastoidectomy performance

Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

Harvard

APA

CBE

MLA

Vancouver

Author

Bibtex

@article{b9855d322b7148dea72cc32d59d623d3,
title = "Expert sampling of VR simulator metrics for automated assessment of mastoidectomy performance",
abstract = "OBJECTIVE: Often the assessment of mastoidectomy performance requires time-consuming manual rating. Virtual reality (VR) simulators offer potentially useful automated assessment and feedback but should be supported by validity evidence. We aimed to investigate simulator metrics for automated assessment based on the expert performance approach, comparison with an established assessment tool, and the consequences of standard setting.METHODS: The performances of 11 experienced otosurgeons and 37 otorhinolaryngology residents. Participants performed three mastoidectomies in the Visible Ear Simulator. Nine residents contributed additional data on repeated practice in the simulator. One hundred and twenty-nine different performance metrics were collected by the simulator and final-product files were saved. These final products were analyzed using a modified Welling Scale by two blinded raters.RESULTS: Seventeen metrics could discriminate between resident and experienced surgeons' performances. These metrics mainly expressed various aspects of efficiency: Experts demonstrated more goal-directed behavior and less hesitancy, used less time, and selected large and sharp burrs more often. The combined metrics-based score (MBS) demonstrated significant discriminative ability between experienced surgeons and residents with a mean difference of 16.4{\%} (95{\%} confidence interval [12.6-20.2], P < 0.001). A pass/fail score of 83.6{\%} was established. The MBS correlated poorly with the final-product score but excellently with the final-product score per time.CONCLUSION: The MBS mainly reflected efficiency components of the mastoidectomy procedure, and although it could have some uses in self-directed training, it fails to measure and encourage safe routines. Supplemental approaches and feedback are therefore required in VR simulation training of mastoidectomy.LEVEL OF EVIDENCE: 2b Laryngoscope, 129:2170-2177, 2019.",
keywords = "Adult, Aged, Clinical Competence, Education, Medical, Continuing, Education, Medical, Graduate, Educational Measurement, Female, Humans, Male, Mastoidectomy, Middle Aged, Otolaryngology/education, Simulation Training/methods, User-Computer Interface, Virtual Reality",
author = "Andersen, {Steven Arild Wuyts} and Mikkelsen, {Peter Trier} and S{\o}rensen, {Mads S{\o}lvsten}",
note = "{\circledC} 2019 The American Laryngological, Rhinological and Otological Society, Inc.",
year = "2019",
month = "9",
doi = "10.1002/lary.27798",
language = "English",
volume = "129",
pages = "2170--2177",
journal = "The Laryngoscope",
issn = "0023-852X",
publisher = "JohnWiley & Sons, Inc",
number = "9",

}

RIS

TY - JOUR

T1 - Expert sampling of VR simulator metrics for automated assessment of mastoidectomy performance

AU - Andersen, Steven Arild Wuyts

AU - Mikkelsen, Peter Trier

AU - Sørensen, Mads Sølvsten

N1 - © 2019 The American Laryngological, Rhinological and Otological Society, Inc.

PY - 2019/9

Y1 - 2019/9

N2 - OBJECTIVE: Often the assessment of mastoidectomy performance requires time-consuming manual rating. Virtual reality (VR) simulators offer potentially useful automated assessment and feedback but should be supported by validity evidence. We aimed to investigate simulator metrics for automated assessment based on the expert performance approach, comparison with an established assessment tool, and the consequences of standard setting.METHODS: The performances of 11 experienced otosurgeons and 37 otorhinolaryngology residents. Participants performed three mastoidectomies in the Visible Ear Simulator. Nine residents contributed additional data on repeated practice in the simulator. One hundred and twenty-nine different performance metrics were collected by the simulator and final-product files were saved. These final products were analyzed using a modified Welling Scale by two blinded raters.RESULTS: Seventeen metrics could discriminate between resident and experienced surgeons' performances. These metrics mainly expressed various aspects of efficiency: Experts demonstrated more goal-directed behavior and less hesitancy, used less time, and selected large and sharp burrs more often. The combined metrics-based score (MBS) demonstrated significant discriminative ability between experienced surgeons and residents with a mean difference of 16.4% (95% confidence interval [12.6-20.2], P < 0.001). A pass/fail score of 83.6% was established. The MBS correlated poorly with the final-product score but excellently with the final-product score per time.CONCLUSION: The MBS mainly reflected efficiency components of the mastoidectomy procedure, and although it could have some uses in self-directed training, it fails to measure and encourage safe routines. Supplemental approaches and feedback are therefore required in VR simulation training of mastoidectomy.LEVEL OF EVIDENCE: 2b Laryngoscope, 129:2170-2177, 2019.

AB - OBJECTIVE: Often the assessment of mastoidectomy performance requires time-consuming manual rating. Virtual reality (VR) simulators offer potentially useful automated assessment and feedback but should be supported by validity evidence. We aimed to investigate simulator metrics for automated assessment based on the expert performance approach, comparison with an established assessment tool, and the consequences of standard setting.METHODS: The performances of 11 experienced otosurgeons and 37 otorhinolaryngology residents. Participants performed three mastoidectomies in the Visible Ear Simulator. Nine residents contributed additional data on repeated practice in the simulator. One hundred and twenty-nine different performance metrics were collected by the simulator and final-product files were saved. These final products were analyzed using a modified Welling Scale by two blinded raters.RESULTS: Seventeen metrics could discriminate between resident and experienced surgeons' performances. These metrics mainly expressed various aspects of efficiency: Experts demonstrated more goal-directed behavior and less hesitancy, used less time, and selected large and sharp burrs more often. The combined metrics-based score (MBS) demonstrated significant discriminative ability between experienced surgeons and residents with a mean difference of 16.4% (95% confidence interval [12.6-20.2], P < 0.001). A pass/fail score of 83.6% was established. The MBS correlated poorly with the final-product score but excellently with the final-product score per time.CONCLUSION: The MBS mainly reflected efficiency components of the mastoidectomy procedure, and although it could have some uses in self-directed training, it fails to measure and encourage safe routines. Supplemental approaches and feedback are therefore required in VR simulation training of mastoidectomy.LEVEL OF EVIDENCE: 2b Laryngoscope, 129:2170-2177, 2019.

KW - Adult

KW - Aged

KW - Clinical Competence

KW - Education, Medical, Continuing

KW - Education, Medical, Graduate

KW - Educational Measurement

KW - Female

KW - Humans

KW - Male

KW - Mastoidectomy

KW - Middle Aged

KW - Otolaryngology/education

KW - Simulation Training/methods

KW - User-Computer Interface

KW - Virtual Reality

U2 - 10.1002/lary.27798

DO - 10.1002/lary.27798

M3 - Journal article

VL - 129

SP - 2170

EP - 2177

JO - The Laryngoscope

JF - The Laryngoscope

SN - 0023-852X

IS - 9

ER -

ID: 59291729