Machine Learning and Artificial Intelligence for Supporting Systematic Reviews: A Systematic Review of Recent Methodological Developments and Recommendations for Implementation

OBJECTIVES: Machine learning (ML) and artificial intelligence (AI) may enable automation of components of systematic reviews. These methods need evaluating to ensure continued robustness of systematic reviews and appropriate integration into existing workflows. The objective of this systematic review was to identify recent evidence of the use, performance, and implementation of ML/AI to support systematic reviews.

METHODS: We searched Embase, Medline, and the Cochrane Library to identify articles published since 1 January 2022. We included any study type that evaluated the performance of ML/AI tools and explored the real-world implementation of these tools. We synthesized results using a staged approach, and this abstract summarizes findings from published literature reviews.

RESULTS: We included 20 articles corresponding to 19 literature reviews. These reviews described the use of ML/AI tools (n = 5), implementation evaluation of ML/AI (n = 3), performance evaluation of ML/AI tools (n = 5), identification of available ML/AI tools (n = 3), and ML/AI-assisted methods (n = 3). Most reviews, particularly those assessing performance, considered screening (n = 14) or data extraction (n = 14); limited evidence was found on filtering, searching, deduplication, snowballing, quality appraisal, and qualitative synthesis. Common performance metrics include recall, precision, and F1 scores; prioritization of recall optimization was recommended. Typically, ML/AI tools are trained on titles and abstracts obtained mainly from Medline and tend to focus on randomized controlled trials. Barriers to the uptake of ML/AI include lack of regulatory agency guidance, costs, training requirements, user friendliness of tools, and transparency concerns.

CONCLUSIONS: Further research is needed on the best approaches to maximize recall of ML/AI tools. There is a need to expand the data sources used for training ML/AI tools. Given high expectations for ML/AI, performance assessment against current practice, rather than perfection, may be appropriate to enable efficiency and increased uptake of these methods.