A New Procedure-Specific Assessment Tool is Superior to Objective Structured Assessment of Technical Skills in Evaluating Orthopaedic Technical Skills: A Cadaveric Simulation Validation Study

BACKGROUND: Competency-based medical education relies on assessment to evaluate performance and guide learning. This study compared the validity evidence and psychometric performance of the Objective Structured Assessment of Technical Skills (OSATS), adapted for distal radius fracture fixation, with a procedure-specific instrument-the Distal Radius Fracture Assessment of Technical Expertise (DRF-RATE)-for assessing technical performance during volar locking plate (VLP) fixation of distal radius fractures (DRFs).

METHODS: First-year orthopaedic residents (n = 12) and practicing trauma or hand surgeons (n = 13) performed standardized VLP fixation on a cadaveric DRF model. Anonymous video recordings were independently assessed by 2 orthopaedic trauma surgeons and 2 hand surgeons using either OSATS or DRF-RATE. Validity evidence was gathered in accordance with the Messick contemporary validity framework.

RESULTS: Internal consistency reliability was good-to-excellent for OSATS (α = 0.82-0.98) and DRF-RATE (α = 0.95). Interrater reliability was weaker for OSATS (r = 0.59-0.69) than for DRF-RATE (r = 0.78), though both demonstrated systematic rater bias. OSATS discriminated between novices and experienced surgeons (p = 0.02), whereas DRF-RATE demonstrated even stronger discrimination (p < 0.001). Contrasting-groups analysis revealed higher misclassification rates with OSATS (21%-40%) than DRF-RATE (11%-13%), indicating superior accuracy for the latter.

CONCLUSIONS: The DRF-RATE demonstrated stronger and more comprehensive validity evidence than the modified OSATS, with better reliability, higher discrimination and classification accuracy, and overall superior psychometric performance compared with OSATS. Its procedural granularity and structured design address key limitations of OSATS and support detailed, high-quality formative feedback for distal radius fracture fixation.