Advancing Patient Health Through Regulatory-Grade Real-World Evidence Generation
Josh Gagne, PharmD, ScD, Johnson & Johnson, Global Epidemiology, Cambridge, MA, USA; Paul Coplan, ScD, MBA, Johnson & Johnson, MedTech Epidemiology & Real-World Data, New Brunswick, NJ, USA; Kourtney Davis, PhD, MSPH, Johnson & Johnson, Innovative Medicine, Global Epidemiology, Horsham, PA, USA; Hong Qiu, MD, PhD, Johnson & Johnson, Global Epidemiology, Titusville, NJ, USA; and Patrick Ryan, PhD, Johnson & Johnson, Observational Health Data Analytics, Titusville, NJ, USA
Real-world evidence (RWE) generation is at an important inflection point. While real-world data (RWD) have long been used to inform regulatory decisions related to medical product safety, RWE generated from the same underlying data sources is being increasingly used to inform decisions about drug, biologic, and medical device effectiveness.
"Having health authorities recognize the value of RWE, formalize its use through clear guidance, and sponsor initiatives to foster its use to inform regulatory decisions are important steps forward in realizing the full potential of RWD for advancing patient health."
Signaling a heightened consideration for RWE, the US Food and Drug Administration (FDA) released new industry guidance on August 30, 20231 and December 21, 20232,3 on the use of RWD and RWE for regulatory decisions for drugs and biologics. This adds to the 20174 guidance on the use of RWE to support regulatory decision making for medical devices, which was recently updated and circulated for comments on December 19, 2023.5 The European Medicines Agency (EMA), National Medical Products Association (NMPA) in China, and the Pharmaceuticals and Medical Devices Agency (PMDA) in Japan, among others, are also expanding their adoption of RWE. In addition, health authorities around the world continue to invest in programs to advance the use of RWE for regulatory decisions, such as the FDA’s Sentinel and Biologics Effectiveness and Safety (BEST) programs,6 the National Evaluation System for health Technology (NEST),7 and the EMA’s Data Analysis and Real-World Interrogation Network (DARWIN EU).8 This collective global effort underscores the growing recognition and adoption of RWE as a valuable tool in shaping informed decision making within the healthcare regulatory landscape.
Having health authorities recognize the value of RWE, formalize its use through clear guidance, and sponsor initiatives to foster its use to inform regulatory decisions are important steps forward in realizing the full potential of RWD for advancing patient health. Pharmaceutical and medical device manufacturers can help further move the needle by applying relevant guidance effectively and consistently and, where applicable, by working through national RWE initiatives to generate the most robust regulatory-grade evidence possible.
Defining regulatory-grade RWE
The first step in facilitating robust RWE generation is establishing a clear, consistent definition of what is meant by regulatory-grade RWE. In other words, what requirements must be met to ensure the evidence generated from RWD is sufficiently robust for regulatory decision making? In 2016, as part of the 21st Century Cures Act, the FDA provided guidance on their expectations for RWE,9 noting that regulatory-grade RWE should be fit-for-purpose (meaning “reliable and relevant”) and generated with research that is transparent. Reliability refers to data accuracy, completeness, provenance, and traceability. Relevance refers to data that capture the necessary product exposure, outcome, and covariate information, and that cover sufficient numbers of patients that represent the target population of interest for a given study.10
Evolving regulatory applications of RWE
FDA11 and others12 have recently documented several examples in which RWE has been used in drug and medical device regulatory decisions that go beyond assessing product safety. For example, RWE has been used to support drug approval decisions by providing therapeutic context and by developing external control arms for clinical trials.13 In postapproval settings, significant progress has been made in using RWE to inform label expansions for medical devices. For example, the FDA recently approved Johnson & Johnson’s label expansion submission for a cardiac ablation catheter to treat persistent atrial fibrillation based on a comparative RWE study using exclusively electronic health record databases from healthcare systems—a first in the medical device industry.14 We have also succeeded in using local RWD to confirm that clinical trial results generalize to patients in countries that may not have been included in or were underrepresented in the development program. The resulting evidence was then used to support a transition from conditional to full approval in those countries and to expand indications to pediatric populations when used in combination with trial extrapolation.
"RWE has been used to support drug approval decisions by providing therapeutic context and by developing external control arms for clinical trials."
Similarly, health authorities are increasingly seeking data to confirm that the safety and efficacy observed in trials extends to all eligible populations, including underrepresented patient groups. A recent review of clinical trials used for FDA approvals of new molecules and biologics in 2020 found that trial populations comprised only 8% Black and 11% Hispanic individuals, while these individuals represent 13% and 16%, respectively, of the overall US population.15 By facilitating data relevance (ie, capturing information from a diverse spectrum of patients [age, sex, race/ethnicity, and disease profile]), RWE generated from data sources that include key patient characteristics and outcomes plays an important role in characterizing the safety and benefits of a particular treatment or device across different patient groups to improve health equity.
Considerations for regulatory-grade RWE generation for safety and effectiveness decision making
While the value of RWE throughout the product lifecycle is clear, generating regulatory-grade RWE that can be used successfully to inform regulatory and clinical decisions can be challenging. The real-world environments that give rise to RWD are both what make such data valuable and what creates potential challenges in RWD analysis and use. This is largely due to concerns about data quality, reliability, relevance, and the potential for biases to which observational studies are subject.
We outline several key considerations in the generation and acceptance of regulatory-grade RWE:
1. Engage health authorities early and often. The field of RWE is rapidly evolving and different stakeholders are at different points along their RWE adoption journey. This can result in divergent preferences for the way in which RWE is generated or processes by which RWE is considered, even within the same regulatory agency. By engaging regulators early and maintaining ongoing communication, stakeholders can ensure transparency in and alignment on the approach to RWE generation, while identifying and addressing challenges proactively. For example, the cardiac ablation catheter approval mentioned earlier was based on a test case study conducted through NEST, which is a public-private partnership supported by the FDA and in which members of the FDA are actively involved. This collaborative approach helped foster a deeper understanding of RWE, but also cultivated a foundation of trust and transparency to reinforce the value of RWE in regulatory decisions.
2. Design the RWE study like a randomized trial. While using reliable and relevant data is necessary for generation of regulatory-grade RWE, it alone is not always sufficient for producing robust evidence. Confounding, time-related biases, and inappropriate adjustment for post-baseline variables are common sources of bias in real-world studies of medical product outcomes. Designing an observational study to emulate a target trial that would answer the question of interest can increase the probability that the evidence generated is reliable.16 For example, clearly specifying eligibility criteria, treatment strategies, outcome(s), follow-up, causal contrast(s) of interest, analysis plan, and time zero (ie, baseline) can help reduce potential biases in observational studies.17
"The field of RWE is rapidly evolving and different stakeholders are at different points along their RWE adoption journey. This can result in divergent preferences for the way RWE is generated or considered, even within the same regulatory agency."
3. Prioritize rigorous, empirical diagnostics. Even when using fit-for-purpose data and employing a study design that emulates a randomized trial, some questions of regulatory importance cannot be reliably addressed using RWD. The reasons for which RWD may be insufficient for a particular question are sometimes apparent. The tenability of assumptions required for valid causal inference may be less obvious, but can be informed by empirical diagnostics, which are checks that can be performed on study databases and cohorts to assess whether an estimated treatment effect is likely to be valid. For example, one can assess whether sufficient balance in baseline or pretreatment patient characteristics exists between treatment groups to enable a valid comparison. Negative control outcomes can be used to assess the potential for residual confounding beyond observed balance in measured covariates.18 These and other diagnostics, including those that can be used to inform data fitness-for-purpose determinations, can be systematically deployed prior to estimating a treatment effect. This stepwise approach helps ensure that only evidence that passes diagnostics is generated, and results that are likely to be subject to substantial confounding or other biases—therefore inaccurate and misleading—are avoided.
4. Evaluate the robustness of the evidence. Certain prespecified analyses can be helpful for confirming that generated evidence is indeed robust, consistent, and generalizable. In their framework for RWE programs, the FDA indicated a focus on prespecifying sensitivity analyses for RWE studies for effectiveness.19 Conducting such analyses across multiple data sources provides an opportunity to assess for unanticipated or otherwise undetected biases or important differences in treatment effect that vary across patient populations.20 Quantitative bias analysis can also be used to assess the impact of potential residual biases, including bias due to confounding and outcome misclassification.21
"Accelerating innovation and unlocking the full value of RWE to advance patient care will require health authorities to further consider the evolving evidence needs of today’s healthcare landscape and their implications on safety and effectiveness decision-making standards."
Unlocking the full value of RWE for patients
Accelerating innovation and unlocking the full value of RWE to advance patient care will require researchers to build on now established examples to continue to advance the methodological rigor of RWE generation. It will also require health authorities to further consider the evolving evidence needs of today’s healthcare landscape and their implications on safety and effectiveness decision-making standards.
Moreover, it is important to remember that robust RWE is not only beneficial to health authorities, medical product innovators, and manufacturers, and that the considerations outlined here to support generation of regulatory-grade RWE are not only applicable for regulatory decision making. The value of robust evidence is critical across all types of healthcare decisions, including health technology assessments and payer decisions, clinical guidelines to inform treatment choices, and disease epidemiology to support health policy.
"We can create a world in which RWE produces a comprehensive understanding of disease and the effects of medical products used in routine clinical practice to enable more confident and informed benefit-risk decision making by all stakeholders."
Through a concerted effort across the healthcare ecosystem, we can create a world in which RWE produces a comprehensive understanding of disease and the effects of medical products used in routine clinical practice to enable more confident and informed benefit-risk decision making by all stakeholders.
References
1. US Food and Drug Administration. Considerations for the Use of Real-World Data and Real-World Evidence to Support Regulatory Decision-Making for Drug and Biological Products Guidance for Industry. August 2023. Accessed March 18, 2024. https://www.fda.gov/media/171667/download
2. US Food and Drug Administration. Real-World Data: Assessing Registries to Support Regulatory Decision-Making for Drug and Biological Products Guidance for Industry. December 2023. Accessed March 18, 2024. https://www.fda.gov/media/154449/download
3. US Food and Drug Administration. Data Standards for Drug and Biological Product Submissions Containing Real-World Data Guidance for Industry. December 2023. Accessed March 18, 2024. https://www.fda.gov/media/153341/download
4. US Food and Drug Administration. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices. August 2017. Accessed March 18, 2024. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-real-world-evidence-support-regulatory-decision-making-medical-devices
5. US Food and Drug Administration. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices: Draft Guidance for Industry and Food and Drug Administration Staff. December 19, 2023. Accessed March 18, 2024. https://www.fda.gov/media/174819/download
6. Biologics Effectiveness and Safety (BEST) Initiative. Accessed March 18, 2024. https://bestinitiative.org/
7. National Evaluation System for health Technology Coordinating Center (NESTcc). Test-Cases. Accessed March 18, 2024. https://nestcc.org/test-cases/
8. European Medicines Agency. Data Analysis and Real-World Interrogation Network (DARWIN EU). Accessed March 18, 2024.
9. US Food and Drug Administration. Real-World Evidence. February 2023. Accessed March 18, 2024. https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence
10. US Food and Drug Administration. Real-World Data: Assessing Electronic Health Records and Medical Claims Data to Support Regulatory Decision-Making for Drug and Biological Products Guidance for Industry. September 2021. Accessed March 18, 2024. https://www.fda.gov/media/152503/download
11. US Food and Drug Administration. Examples of Real-World Evidence (RWE) Used in Medical Device Regulatory Decisions. Accessed March 18, 2024. https://www.fda.gov/media/146258/download
12. Purpura CA, Garry EM, Honig N, Case A, Rassen JA. The role of real-world evidence in FDA-approved new drug and biologics license applications. Clin Pharmacol Ther. 2022;111(1):135-144. doi: 10.1002/cpt.2474.
13. US Food and Drug Administration. 2020 Drug Trials Snapshots Summary Report. February 2021. Accessed March 18, 2024. https://www.fda.gov/media/145718/download
14. Dhruva SS, Zhang S, Chen J, et al. Safety and effectiveness of a catheter with contact force and 6-hole irrigation for ablation of persistent atrial fibrillation in routine clinical practice. JAMA Netw Open. 2022;5(8):e2227134. doi: 10.1001/jamanetworkopen.2022.27134.
15. US Food and Drug Administration. 2020 Drug Trials Snapshots Summary Report. February 2021. Accessed March 18, 2024. https://www.fda.gov/media/145718/download
16. Bykov K, Patorno E, D’Andrea E, et al. Prevalence of avoidable and bias-inflicting methodological pitfalls in real-world studies of medication safety and effectiveness. Clin Pharmacol Ther. 2022;111:209-217. doi: 10.1002/cpt.2364.
17. Hansford HJ, Cashin AG, Jones MD, et al. Reporting of observational studies explicitly aiming to emulate randomized trials: a systematic review. JAMA Netw Open. 2023;6(9):e2336023. doi:10.1001/jamanetworkopen.2023.36023.
18. Khera R, Schuemie MJ, Lu Y, et al. Large-scale evidence generation and evaluation across a network of databases for type 2 diabetes mellitus (LEGEND-T2DM): a protocol for a series of multinational, real-world comparative cardiovascular effectiveness and safety studies. BMJ Open. 2022;12(6):e057977. doi: 10.1136/bmjopen-2021-057977
19. US Food and Drug Administration. Framework for FDA’s Real-World Evidence Program. December 2018. Accessed March 18, 2024. https://www.fda.gov/media/120060/download
20. Hripcsak G, Schuemie MJ, Madigan D, Ryan PB, Suchard MA. Drawing reproducible conclusions from observational clinical data with OHDSI. Yearb Med Inform. 2021;30(1):283–289. doi: 10.1055/s-0041-1726481
21. Lash TL, Fox MP, Fink AK. Applying Quantitative Bias Analysis to Epidemiologic Data. Springer New York, 14 April 2011. https://doi.org/10.1007/978-0-387-87959-8
