Research
Print page Print page
Switch language
Rigshospitalet - a part of Copenhagen University Hospital
Published

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

Research output: Contribution to journalReviewResearchpeer-review

  1. In reply to walsh

    Research output: Contribution to journalComment/debateResearchpeer-review

  2. Construct Validity of the Reporter-Interpreter-Manager-Educator Structure for Assessing Students' Patient Encounter Skills

    Research output: Contribution to journalJournal articleResearchpeer-review

  1. Assessing competence in cochlear implant surgery using the newly developed Cochlear Implant Surgery Assessment Tool

    Research output: Contribution to journalJournal articleResearchpeer-review

  2. Effect of 3D-Printed Models on Cadaveric Dissection in Temporal Bone Training

    Research output: Contribution to journalJournal articleResearchpeer-review

  3. Use of simulation-based training of surgical technical skills among ENTs: an international YO-IFOS survey

    Research output: Contribution to journalJournal articleResearchpeer-review

  4. 3D-Printed Models for Temporal Bone Surgical Training: A Systematic Review

    Research output: Contribution to journalReviewResearchpeer-review

View graph of relations

PURPOSE: Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance.

METHOD: In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability.

RESULTS: Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment.

CONCLUSIONS: G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

Original languageEnglish
JournalAcademic medicine : journal of the Association of American Medical Colleges
Volume96
Issue number11
Pages (from-to)1609-1619
Number of pages11
ISSN1040-2446
DOIs
Publication statusPublished - 1 Nov 2021

    Research areas

  • Analysis of Variance, Checklist/methods, Clinical Competence/statistics & numerical data, Concept Formation, Databases, Factual, Education, Medical/statistics & numerical data, Educational Measurement/methods, Humans, Models, Statistical, Practice Guidelines as Topic/standards, Reproducibility of Results

ID: 65443043