Artificial intelligence-based action recognition and skill assessment in robotic cardiac surgery simulation: a feasibility study

To create a deep neural network capable of recognizing basic surgical actions and categorizing surgeons based on their skills using video data only. Nineteen surgeons with varying levels of robotic experience performed three wet lab tasks on a porcine model: robotic-assisted atrial closure, mitral stitches, and dissection of the thoracic artery. We used temporal labeling to mark two surgical actions: suturing and dissection. Each complete recording was annotated as either "novice" or "expert" based on the operator's experience. The network architecture combined a Convolutional Neural Network for extracting spatial features with a Long Short-Term Memory layer to incorporate temporal information. A total of 435 recordings were analyzed. The fivefold cross-validation yielded a mean accuracy of 98% for the action recognition (AR) and 79% for the skill assessment (SA) network. The AR model achieved an accuracy of 93%, with average recall, precision, and F1-score all at 93%. The SA network had an accuracy of 56% and a predictive certainty of 95%. Gradient-weighted Class Activation Mapping revealed that the algorithm focused on the needle, suture, and instrument tips during suturing, and on the tissue during dissection. AR network demonstrated high accuracy and predictive certainty, even with a limited dataset. The SA network requires more data to become a valuable tool for performance evaluation. When combined, these deep learning models can serve as a foundation for AI-based automated post-procedural assessments in robotic cardiac surgery simulation. ClinicalTrials.gov (NCT05043064).