Forskning
Udskriv Udskriv
Switch language
Region Hovedstaden - en del af Københavns Universitetshospital
Udgivet

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

Publikation: Bidrag til tidsskriftReviewForskningpeer review

  1. The Positivism Paradigm of Research

    Publikation: Bidrag til tidsskriftKommentar/debatForskningpeer review

  2. Embracing Multiple Stakeholder Perspectives in Defining Trainee Competence

    Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

  3. A Night on Trauma

    Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

  1. Simulation-based VATS resection of the five lung lobes: a technical skills test

    Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

  2. Assessing competence in cochlear implant surgery using the newly developed Cochlear Implant Surgery Assessment Tool

    Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

  3. Effect of 3D-Printed Models on Cadaveric Dissection in Temporal Bone Training

    Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

Vis graf over relationer

PURPOSE: Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance.

METHOD: In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability.

RESULTS: Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment.

CONCLUSIONS: G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

OriginalsprogEngelsk
TidsskriftAcademic medicine : journal of the Association of American Medical Colleges
Vol/bind96
Udgave nummer11
Sider (fra-til)1609-1619
Antal sider11
ISSN1040-2446
DOI
StatusUdgivet - 1 nov. 2021

ID: 65443043