OBJECTIVES: We address three questions: (1) Can SLRs and HTA value dossiers be fully automated with AI? (2) What combination of AI and human expertise produces the highest quality SLRs and HTAs? (3) Can AI add extra value to the dossier?

METHODS: We conducted a retrospective comparative study to evaluate the full end-to-end automation of a SLR relevant to a HTA submission. We also explored various configurations of AI and human collaborations. Our experimental design comprised a pipeline of AI models integrated into a web platform, powering the entire SLR process from research question formulation to manuscript creation: (1) crafting the research question, (2) devising search strategy and query, (3) conducting PRISMA-compliant screening, (4) extracting relevant data, (5) synthesizing evidence, (6) writing the manuscript. We assessed the impact of varying levels and types of AI involvement on the quality of: decisions, evidence, and writing.

RESULTS: The optimal workflow combined AI and human expertise. In this hybrid approach, 85.7% of extracted data perfectly matched the manual version, with the remaining 14.3% closely aligned.The fully automated workflow achieved perfect extraction 77.1% of the time, close matches 14.5%, and incorrect extractions 8.3% of the time. Both versions were free from hallucinations. Qualitative evaluation of the fully AI-written document showed interesting results and a surprising level of performance in some areas, but lacked the detail and nuance of the partly automated and manual manuscripts in others.

CONCLUSIONS: While full automation of SLRs and HTAs showed promise, the thoughtful integration of AI with expert input proved most effective. We believe this work provides valuable insights for HTA sponsors and reviewers in evaluating the future potential of AI in HTA and SLR processes, while acknowledging the continued importance of human expertise in the field.