OBJECTIVES: The JCA process uses the Patient, Intervention, Comparator, Outcome (PICO) framework and requests that each EU member state put forward their PICO requirements. This could potentially introduce many PICO sets that need consideration within a JCA submission. Thus, it would be beneficial to have an automated process able to quickly determine which PICO sets align with a registrational trial’s PICO. We have investigated whether large language models (LLMs) can determine the alignment of JCA PICO populations, as predicted by clinical experts, with the population of a target registrational trial, using a case study in patients with relapsed refractory multiple myeloma (RRMM).

METHODS: Twenty predicted JCA PICO populations were identified for patients with RRMM and ≥1 prior line. We used a modal approach and provided prompts and contextual information to two LLMs (Claude 3 Opus, GPT-4) accessing their APIs through Python. Alignment was defined as “Full” (trial population = JCA population), “Partial (subgroup)” (trial population subgroup of JCA population), “Partial (overlap)” (trial population overlaps with JCA population), “None” (no overlap). Accuracy of alignment categorization for the populations was determined by comparing the LLM outputs to alignment categorization by clinical experts.

RESULTS: Human classification of the alignment of the 20 populations was partial (subgroup) (“PS”) for three and partial (overlap) (“PO”) for 17. Claude was correct for 18/20, with 2 misclassifications (Full instead of PO; PO instead of PS). GPT was also correct for 18/20, with 2 misclassifications (both PO instead of PS). Potential ambiguity in the population definition for the two populations was likely to have caused the mis-categorization.

CONCLUSIONS: If appropriate context is provided, LLMs are capable of understanding complex epidemiological concepts and categorizing the alignment of two populations. Thus, LLMs can be used to automate this categorization of PICOs within the JCA process.