Research
Print page Print page
Switch language
The Capital Region of Denmark - a part of Copenhagen University Hospital
Published

Reliable Assessment of Surgical Technical Skills Is Dependent on Context: An Exploration of Different Variables Using Generalizability Theory

Research output: Contribution to journalJournal articleResearchpeer-review

  1. The Positivism Paradigm of Research

    Research output: Contribution to journalComment/debateResearchpeer-review

  2. Embracing Multiple Stakeholder Perspectives in Defining Trainee Competence

    Research output: Contribution to journalJournal articleResearchpeer-review

  3. A Night on Trauma

    Research output: Contribution to journalJournal articleResearchpeer-review

  4. In reply to walsh

    Research output: Contribution to journalComment/debateResearchpeer-review

  1. Education in the placement of ultrasound-guided peripheral venous catheters: a systematic review

    Research output: Contribution to journalReviewResearchpeer-review

  2. Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA)

    Research output: Contribution to journalJournal articleResearchpeer-review

  3. Current Status of Handheld Otoscopy Training: A Systematic Review

    Research output: Contribution to journalReviewResearchpeer-review

  4. Identification of Cellular Voids in the Human Otic Capsule

    Research output: Contribution to journalJournal articleResearchpeer-review

View graph of relations

PURPOSE: Reliable assessment of surgical skills is vital for competency-based medical training. Several factors influence not only the reliability of judgments but also the number of observations needed for making judgments of competency that are both consistent and reproducible. The aim of this study was to explore the role of various conditions-through the analysis of data from large-scale, simulation-based assessments of surgical technical skills-by examining the effects of those conditions on reliability using generalizability theory.

METHOD: Assessment data from large-scale, simulation-based temporal bone surgical training research studies in 2012-2018 were pooled, yielding collectively 3,574 assessments of 1,723 performances. The authors conducted generalizability analyses using an unbalanced random-effects design, and they performed decision studies to explore the effect of the different variables on projections of reliability.

RESULTS: Overall, 5 observations were needed to achieve a generalizability coefficient > 0.8. Several variables modified the projections of reliability: increased learner experience necessitated more observations (5 for medical students, 7 for residents, and 8 for experienced surgeons), the more complex cadaveric dissection required fewer observations than virtual reality simulation (2 vs 5 observations), and increased fidelity simulation graphics reduced the number of observations needed from 7 to 4. The training structure (either massed or distributed practice) and simulator-integrated tutoring had little effect on reliability. Finally, more observations were needed during initial training when the learning curve was steepest (6 observations) compared with the plateau phase (4 observations).

CONCLUSIONS: Reliability in surgical skills assessment seems less stable than it is often reported to be. Training context and conditions influence reliability. The findings from this study highlight that medical educators should exercise caution when using a specific simulation-based assessment in other contexts.

Original languageEnglish
JournalAcademic medicine : journal of the Association of American Medical Colleges
Volume95
Issue number12
Pages (from-to)1929-1936
Number of pages8
ISSN1040-2446
DOIs
Publication statusPublished - Dec 2020

    Research areas

  • Clinical Competence, Humans, Learning Curve, Orthopedic Procedures/education, Reproducibility of Results, Simulation Training, Temporal Bone/surgery

ID: 60286350