Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool

Leizl J Nayahangan; Jonathan Lawaetz; Michael Strøm; Louise de la Motte; Peter Rørdam; Bo C Gottschalksen; Nikolaj F Grøndal; Martin Græbe; Jes Sandermann; Brian L Pedersen; Lars Konge; Jonas Eiberg; Collaborators in Denmark

doi:10.1016/j.ejvs.2020.01.021

Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool

Leizl J Nayahangan, Jonathan Lawaetz, Michael Strøm, Louise de la Motte, Peter Rørdam, Bo C Gottschalksen, Nikolaj F Grøndal, Martin Græbe, Jes Sandermann, Brian L Pedersen, Lars Konge, Jonas Eiberg, Collaborators in Denmark

16 Citations (Scopus)

Abstract

OBJECTIVE: The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard.

METHODS: Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting. All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score.

RESULTS: Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 ± 12 and 25.6 ± 6, respectively (p = .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed.

CONCLUSION: Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt.

Original language	English
Journal	European journal of vascular and endovascular surgery : the official journal of the European Society for Vascular Surgery
Volume	59
Issue number	5
Pages (from-to)	767-774
Number of pages	8
ISSN	1078-5884
DOIs	https://doi.org/10.1016/j.ejvs.2020.01.021
Publication status	Published - May 2020

Keywords

Aortic Aneurysm, Abdominal/surgery
Clinical Competence
Humans
Vascular Surgical Procedures/methods
Training
Simulation
Open AAA
Assessment
Validation study

Access to Document

10.1016/j.ejvs.2020.01.021

Cite this

Nayahangan, L. J., Lawaetz, J., Strøm, M., de la Motte, L., Rørdam, P., Gottschalksen, B. C., Grøndal, N. F., Græbe, M., Sandermann, J., Pedersen, B. L., Konge, L., Eiberg, J., & Collaborators in Denmark (2020). Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool. European journal of vascular and endovascular surgery : the official journal of the European Society for Vascular Surgery, 59(5), 767-774. https://doi.org/10.1016/j.ejvs.2020.01.021

Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool. / Nayahangan, Leizl J ; Lawaetz, Jonathan; Strøm, Michael et al.

In: European journal of vascular and endovascular surgery : the official journal of the European Society for Vascular Surgery, Vol. 59, No. 5, 05.2020, p. 767-774.

Research output: Contribution to journal › Journal article › Research › peer-review

Nayahangan, LJ , Lawaetz, J, Strøm, M, de la Motte, L , Rørdam, P, Gottschalksen, BC, Grøndal, NF, Græbe, M, Sandermann, J, Pedersen, BL , Konge, L , Eiberg, J & Collaborators in Denmark 2020, 'Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool', European journal of vascular and endovascular surgery : the official journal of the European Society for Vascular Surgery, vol. 59, no. 5, pp. 767-774. https://doi.org/10.1016/j.ejvs.2020.01.021

Nayahangan LJ , Lawaetz J, Strøm M, de la Motte L , Rørdam P, Gottschalksen BC et al. Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool. European journal of vascular and endovascular surgery : the official journal of the European Society for Vascular Surgery. 2020 May;59(5):767-774. https://doi.org/10.1016/j.ejvs.2020.01.021

@article{1a0290288783418d9593d3ce88582f42,

title = "Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool",

abstract = "OBJECTIVE: The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard.METHODS: Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting. All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score.RESULTS: Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 ± 12 and 25.6 ± 6, respectively (p = .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed.CONCLUSION: Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt.",

keywords = "Aortic Aneurysm, Abdominal/surgery, Clinical Competence, Humans, Vascular Surgical Procedures/methods, Training, Simulation, Open AAA, Assessment, Validation study",

author = "Nayahangan, {Leizl J} and Jonathan Lawaetz and Michael Str{\o}m and {de la Motte}, Louise and Peter R{\o}rdam and Gottschalksen, {Bo C} and Gr{\o}ndal, {Nikolaj F} and Martin Gr{\ae}be and Jes Sandermann and Pedersen, {Brian L} and Lars Konge and Jonas Eiberg and {Collaborators in Denmark}",

year = "2020",

month = may,

doi = "10.1016/j.ejvs.2020.01.021",

language = "English",

volume = "59",

pages = "767--774",

journal = "European Journal of Vascular Surgery",

issn = "1078-5884",

publisher = "W.B./Saunders Co. Ltd",

number = "5",

}

TY - JOUR

T1 - Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool

AU - Nayahangan, Leizl J

AU - Lawaetz, Jonathan

AU - Strøm, Michael

AU - de la Motte, Louise

AU - Rørdam, Peter

AU - Gottschalksen, Bo C

AU - Grøndal, Nikolaj F

AU - Græbe, Martin

AU - Sandermann, Jes

AU - Pedersen, Brian L

AU - Konge, Lars

AU - Eiberg, Jonas

AU - Collaborators in Denmark

PY - 2020/5

Y1 - 2020/5

N2 - OBJECTIVE: The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard.METHODS: Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting. All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score.RESULTS: Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 ± 12 and 25.6 ± 6, respectively (p = .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed.CONCLUSION: Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt.

AB - OBJECTIVE: The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard.METHODS: Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting. All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score.RESULTS: Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 ± 12 and 25.6 ± 6, respectively (p = .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed.CONCLUSION: Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt.

KW - Aortic Aneurysm, Abdominal/surgery

KW - Clinical Competence

KW - Humans

KW - Vascular Surgical Procedures/methods

KW - Training

KW - Simulation

KW - Open AAA

KW - Assessment

KW - Validation study

U2 - 10.1016/j.ejvs.2020.01.021

DO - 10.1016/j.ejvs.2020.01.021

M3 - Journal article

C2 - 32089508

VL - 59

SP - 767

EP - 774

JO - European Journal of Vascular Surgery

JF - European Journal of Vascular Surgery

SN - 1078-5884

IS - 5

ER -

Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool

Abstract

Keywords

Access to Document

Fingerprint

Cite this