Towards an AI-driven registry for postoperative complications: a proof-of-concept study evaluating the opportunities and challenges of AI models

OBJECTIVES: Postoperative complications (PCs) require substantial resources to manage and are cumbersome to monitor. Artificial intelligence (AI), particularly natural language processing (NLP), offers a potential solution by automating and streamlining these processes, but perceived PC rates may differ depending on model optimisation strategies. This study aimed to develop a mock-up AI-driven automated registry for PCs. We hypothesised that using NLP to obtain longitudinal overviews of key quality metrics is feasible, but that optimisation strategies impacted the observed rate of PCs.

METHODS: We analysed 100 505 surgical cases from 12 Danish hospitals between 2017 and 2021. Previously validated NLP models were applied to detect seven types of PCs, using two different threshold settings: a set of thresholds optimised for positive predictive value (precision), referred to as F-score of 0.5, and a set of thresholds optimised for sensitivity, referred to as F-score of 2. Trends in PC rates over time were assessed, and hospital-level variations were examined using logistic regression models.

RESULTS: The NLP models detected 8512 or 15 892 PCs, depending on threshold selection, corresponding to total PC rates of 9.14% and 17.1%, respectively. Most PCs showed stable or increasing trends over time, regardless of threshold setting. Regression analyses demonstrated that threshold selection significantly influenced findings, impacting hospital comparisons.

CONCLUSION: We demonstrate that NLP can be used for automated PC detection. However, threshold selection and additional performance metrics must be carefully considered.