Print page Print page
Switch language
The Capital Region of Denmark - a part of Copenhagen University Hospital

Reliable Assessment of Surgical Technical Skills Is Dependent on Context: An Exploration of Different Variables Using Generalizability Theory

Research output: Contribution to journalJournal articleResearchpeer-review

  1. The Positivism Paradigm of Research

    Research output: Contribution to journalJournal articleResearchpeer-review

  2. Embracing Multiple Stakeholder Perspectives in Defining Trainee Competence

    Research output: Contribution to journalJournal articleResearchpeer-review

  3. A Night on Trauma

    Research output: Contribution to journalJournal articleResearchpeer-review

  4. In reply to walsh

    Research output: Contribution to journalComment/debateResearchpeer-review

  1. Anesthesiologists’ airway management expertise: Identifying subjective and objective knowledge gaps

    Research output: Contribution to journalJournal articleResearchpeer-review

  2. Evidence of Mobile Applications in Otolaryngology Targeted at Patients

    Research output: Contribution to journalEditorialResearchpeer-review

View graph of relations

PURPOSE: Reliable assessment of surgical skills is vital for competency-based medical training. Several factors influence not only the reliability of judgments but also the number of observations needed for making judgments of competency that are both consistent and reproducible. The aim of this study was to explore the role of various conditions-through the analysis of data from large-scale, simulation-based assessments of surgical technical skills-by examining the effects of those conditions on reliability using generalizability theory.

METHOD: Assessment data from large-scale, simulation-based temporal bone surgical training research studies in 2012-2018 were pooled, yielding collectively 3,574 assessments of 1,723 performances. The authors conducted generalizability analyses using an unbalanced random-effects design, and they performed decision studies to explore the effect of the different variables on projections of reliability.

RESULTS: Overall, 5 observations were needed to achieve a generalizability coefficient > 0.8. Several variables modified the projections of reliability: increased learner experience necessitated more observations (5 for medical students, 7 for residents, and 8 for experienced surgeons), the more complex cadaveric dissection required fewer observations than virtual reality simulation (2 vs 5 observations), and increased fidelity simulation graphics reduced the number of observations needed from 7 to 4. The training structure (either massed or distributed practice) and simulator-integrated tutoring had little effect on reliability. Finally, more observations were needed during initial training when the learning curve was steepest (6 observations) compared with the plateau phase (4 observations).

CONCLUSIONS: Reliability in surgical skills assessment seems less stable than it is often reported to be. Training context and conditions influence reliability. The findings from this study highlight that medical educators should exercise caution when using a specific simulation-based assessment in other contexts.

Original languageEnglish
JournalAcademic medicine : journal of the Association of American Medical Colleges
Issue number12
Pages (from-to)1929-1936
Number of pages8
Publication statusPublished - Dec 2020

ID: 60286350