OBJECTIVES: The automation of evidence generation in market access of new medicines using generative artificial intelligence (GenAI) is promising. This study explores GenAI, specifically GPT-3.5, GPT-4 Turbo, and GPT-4o via a private Azure OpenAI instance, for automating title and abstract (TIAB) screening in systematic literature reviews (SLRs). The focus was on evaluating the performance of these models in classifying studies based on inclusion/exclusion criteria.

METHODS: Two main methods were evaluated. Firstly, prompts were used to extract data and further prompting used to include/exclude based on the extracted data. Secondly, hierarchical prompting based on two sub-approaches were tested; complex prompting using all the inclusion/exclusion criteria, and a simpler approach using natural language constructed prompts (NLCP) that considered the inclusion reasons alone. In both cases 50 abstracts per dataset were used to engineer the prompts before these prompts were applied to the remaining dataset. Model performance was measured by conflict rate with human reviewers' decisions.

RESULTS: Results indicated that the use of NLCP with GPT-4 Turbo and GPT-4o models provided the best outcomes for TIAB screening. This approach yielded an average of 10% conflict rates with human reviewers across all datasets. While both GPT-4 and GPT-4o demonstrated high accuracy, GPT-4o offered a significant advantage in terms of speed and cost, being twice as fast and more economical compared to GPT-4 Turbo.

CONCLUSIONS: The study finds that employing NLCP for TIAB classification is more effective than using complex inclusion/exclusion criteria or extraction-driven prompting. Furthermore, despite the similar accuracy between GPT-4 Turbo and GPT-4o, the latter's efficiency in processing time and cost makes it a preferable choice for automating SLRs. This research underscores the potential of GenAI in streamlining the literature review process, thereby contributing to the acceleration of market access of new medicines.