Testing basic competency in knee arthroscopy using a virtual reality simulator: exploring validity and reliability

BACKGROUND: Diagnostic knee arthroscopy is a common procedure that orthopaedic residents are expected to learn early in their training. Arthroscopy requires a different skill set from traditional open surgery, and many orthopaedic residents feel less prepared for arthroscopic procedures. Virtual reality simulation training and testing provide an opportunity to ensure basic competency before proceeding to supervised procedures in patients.

METHODS: Twenty-six physicians (thirteen novices and thirteen experienced arthroscopic surgeons) were voluntarily recruited to perform a test consisting of five arthroscopic procedures on a knee arthroscopy simulator. Performance was evaluated by obtaining predefined metrics from the simulator for each procedure, and z-scores, describing suboptimal performance, were calculated from the metrics. The intercase reliability of the simulator metrics was explored by calculating an intraclass correlation coefficient. Finally, a pass-or-fail standard was set with use of the contrasting groups method, and the consequences of the pass-or-fail standard were explored.

RESULTS: One procedure was excluded from the final test because of a lack of validity. The total Z-scores for the four procedures included in the final test showed an intercase reliability of 0.87 (95% confidence interval, 0.78 to 0.93). The total mean z-score (and standard deviation) was 38.6 ± 27.3 points for the novices and 0.0 ± 9.1 points for the experienced surgeons (p < 0.0005). The pass-or-fail standard was set at a total z-score of 15.5 points, resulting in two of the novices passing the test and a single experienced surgeon failing the test.

CONCLUSIONS: By combining four procedures on a virtual reality arthroscopy simulator, it was possible to create a valid, reliable, and feasible test of basic arthroscopic competency and to establish a credible pass-or-fail standard.

CLINICAL RELEVANCE: The simulation-based test and pass-or-fail standard could aid in assessing and ensuring basic competency of future orthopaedic residents before proceeding to supervised procedures in patients.