TY - JOUR
T1 - Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA)
AU - Engberg, Morten
AU - Lönn, Lars
AU - Konge, Lars
AU - Mikkelsen, Søren
AU - Hörer, Tal
AU - Lindgren, Hans
AU - Søvik, Edmund
AU - Svendsen, Morten Bo
AU - Frendø, Martin
AU - Taudorf, Mikkel
AU - Russell, Lene
N1 - Copyright © 2021 Lippincott Williams & Wilkins, Inc.
PY - 2021/10/1
Y1 - 2021/10/1
N2 - BACKGROUND: Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE.METHODS: This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard.RESULTS: Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed.CONCLUSION: Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts.LEVEL OF EVIDENCE: Diagnostic test, no or poor gold standard, level V.
AB - BACKGROUND: Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE.METHODS: This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard.RESULTS: Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed.CONCLUSION: Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts.LEVEL OF EVIDENCE: Diagnostic test, no or poor gold standard, level V.
UR - http://www.scopus.com/inward/record.url?scp=85116808410&partnerID=8YFLogxK
U2 - 10.1097/TA.0000000000003338
DO - 10.1097/TA.0000000000003338
M3 - Journal article
C2 - 34225347
SN - 2163-0755
VL - 91
SP - 663
EP - 671
JO - The journal of trauma and acute care surgery
JF - The journal of trauma and acute care surgery
IS - 4
ER -