OBJECTIVES: EU HTA Regulation’s Joint Clinical Assessments (JCA) are likely to require health technology developers (HTDs) to conduct a large number of comparative clinical analyses within a tight timeframe (≤100 days). Large language models (LLMs) have previously demonstrated proficiency for extracting data from text, therefore, the purpose of this study was to explore whether LLMs could also be leveraged to extract data from tables and figures. Such automation could help HTDs meet JCA requirements.

METHODS: Python was utilized to screen publications and identify pages containing relevant tables and figures. A deep learning model was then used to extract tables and figures into separate images and GPT-4o employed to identify the type of figure/table and label accordingly (e.g. patient characteristics, forest plot). Labelled images of 13 figures and 5 tables were submitted to two LLMs (Claude 3 Opus and GPT-4o) for data extraction. The data extracted from each table or figure was assessed for inclusiveness and accuracy.

RESULTS: The deep learning model achieved 100% accuracy when extracting tables and figures from relevant pages in a publication and saving them as images; this was inclusive of situations where tables/figures were split across multiple pages. GPT-4o and Claude achieved 100% accuracy when extracting data from the images of tables/figures, including in cases where figures comprised multiple subfigures. Only in instances where the text size in the image was very small (<6pts), and not easily readable by a human, were LLMs unable to extract data and became susceptible to hallucinations.

CONCLUSIONS: The results show that a combination of Python, deep learning and LLMs can be used to automatically extract images of tables and figures from within publications. Using these images, LLMs have also demonstrated the ability to extract data accurately. Such automation has the potential to significantly reduce burden on HTDs preparing for JCA submissions.