Increased Risks for Random Errors are Common in Outcomes Graded as High Certainty of Evidence

OBJECTIVE: To assess the risk for random errors in outcomes graded as high certainty of evidence (CoE).

STUDY DESIGN AND SETTING: We randomly selected 100 Cochrane reviews with dichotomous outcomes rated as high CoE using GRADE (Grading of Recommendations Assessment, Development, and Evaluation). To detect increased risks for random errors, two investigators independently conducted Trial Sequential Analysis (TSA) employing conventional thresholds for type I (α = 0.05) and type II (β = 0.10) errors. We dually re-graded all outcomes with increased risks for random errors and conducted multivariate logistic regression analyses to determine predictors of increased risks for random errors.

RESULTS: Overall, 38% (95% confidence interval: 28% to 47%) of high CoE outcomes had increased risks for random errors. Outcomes assessing harms were more frequently affected than outcomes assessing benefits (47% vs. 12%). Re-grading of outcomes with increased random errors showed that 74% should have been downgraded based on current guidance. Regression analyses rendered small absolute risk differences (p = 0.009) and low number of events (p = 0.001) as significant predictors of increased risks for random errors.

CONCLUSION: Decisionmakers need to be aware that outcomes rated as high CoE often have increased risks for false-positive or false-negative findings.