Updates From the Real-World Evidence Transparency Initiative

 Lucinda S. Orsini, DPM, MPH, ISPOR, Lawrenceville, NJ, USA

The ISPOR Real-World Evidence (RWE) Transparency Initiative continues to make progress towards the Society’s goals in highlighting the importance of transparent research methods in secondary data-use studies, particularly for hypothesis evaluating treatment effect or comparative-effectiveness research, including safety. This month Value in Health (the official journal of ISPOR) and Pharmacoepidemiology & Drug Safety (the official journal of the International Society for Pharmacoepidemiology) have copublished the recommendations from the initiative in an article titled, Improving Transparency to Build Trust in Real-World Secondary Data Studies for Hypothesis Testing—Why, What, and How: Recommendations and a Road Map From the Real-World Evidence Transparency Initiative.¹

"The pandemic caused by SARS-CoV-2 has put public attention on the urgent need for robust, quality, real-time clinical research that includes observational secondary data-use studies."

This publication follows the position paper based on the original working group meeting in February 2019, and has been refined through the presentation of the underlying concepts at a series of ISPOR meetings. These include our conference in May 2019, the 1-day ISPOR summit in October 2019, 3 sessions presented at the ISPOR conference in Copenhagen in November 2019, plus reconvening most of the original working group members for a meeting in February 2020. It is clear that the idea of transparent research methods resonates with our stakeholders and research experts and that the urgency to continue to develop mechanisms that promote trust in the evidence-generation process and enable decision makers to evaluate the quality of methods and resulting evidence from RWE studies continues to increase.^2-6

The pandemic caused by SARS-CoV-2 has put public attention on the urgent need for robust, quality, real-time clinical research that includes observational secondary data-use studies. Without previous experience in treating this virus, the healthcare system has had to resort to trial and error and shared case study experience from international sources, while also treating a tsunami of critically ill, highly infective patients. Researchers are looking to any real-time data sources to analyze trends and find patient characteristics that could predict high-risk cases. This approach has highlighted both good and bad examples of secondary data-use studies.

Never have we seen so many decision makers turning to preprints (submitted but not yet peer reviewed articles) for hints on safety signals and effectiveness of therapies (including ventilation, supportive oxygen, and steroid use), with many of these studies conducted as observational research or large simple trials conducted by hospitals treating patients in real time. While it is laudable to have these data available without proper understanding of the methods and underlying data, it is difficult to know which studies point to actionable evidence versus spurious or perhaps fraudulent results. Even in the peer review space, we have seen high-profile journals retract published observational research due to lack of transparency about the underlying data and methods,^7,8 but not before high-profile studies paused enrollment partially due to information that was relevant to their patient population.⁹

It is clear that health economic and outcomes researchers who know the power and value of retrospective research need to continue to lead the way, shining the light on the robust methods that exist for these studies and the need for transparent study processes so that we can continue to build credibility in our research. We also need to increase our outreach, particularly to clinical journals that are used to publishing randomized clinical trial evidence, to see how we can better equip reviewers with insight and tools to understand the quality of the research before them. Clearly, there is a role for ISPOR and our members to partner more closely to educate on the merits of requiring transparent processes similar to clinical trials and perhaps even provide access to experienced scientific reviewers who could help mitigate publication of observational research that is not appropriate.

To these ends, the transparency initiative has several ongoing work streams, including efforts to build a public study registration site that is focused on hypothesis-evaluating treatment effect studies of secondary data. This initial study registration site will allow continued education of RWE researchers on good study processes, including preplanning research questions, study structure, and protocols as a prerequisite to upload. This initial site will be additive to the options currently available or required for study registration, such as Clinicaltrials.gov and the European Post Approval Study (EU-PAS) registry. The initiative understands that many researchers with studies for regulatory intent will be required to register studies at these sites even if the variables are not optimized for secondary data-use studies. However, the RWE Transparency Initiative site will be available for all other researchers, providing a simplified tailored set of questions and version control, including embargo of study and study protocols to encourage “good behavior” regarding study transparency. This includes asking researchers to attest to the amount of data-handling that has occurred with the database the proposed study will utilize prior to registering the study.

The initiative has also continued to work with multiple stakeholders to inform and evaluate our work plans, for example, aligning the first version study register variables with the structured template and reporting tool for RWE studies (STaRT-RWE).¹⁰ Also, we will partner with the STaRT-RWE development team under the auspices of the International Society of Pharmacoepidemiology on a new joint task force that will develop a standardized protocol template specific to secondary data-use studies looking at causal inferences. This protocol working group will help inform the next generation of the Transparency Initiative’s registration site, which will continue to be optimized based on real-use cases and the protocol template, and can also feed into the regulatory sites like EU-PAS as they also look to optimize ease of use and relevance while fulfilling their government mandates.

"We in the ISPOR research community are very aware of what good outcomes research entails and are already holding ourselves accountable and should continue to do so publicly. By actively registering our studies, we are putting a stake in the ground and holding up health economics and outcomes research to the same bar as any other clinical research."

However, tools like study registration, reporting, and protocol templates are only useful if they are used. We know from discussions with our colleagues at ClinicalTrials.gov that it was not until the International Committee of Medical Journal Editors implemented a policy that registrations of any trial are required for results to be considered for publication that site usage started to increase at a rapid rate. Incentives matter. This is another area that the Transparency Initiative will pursue as we stand up the first version of the study registration site. We need to work with our end users—journal editors, payers, and assessors—to educate them on why asking questions about study registration of submitted research is in their best interest, and encourage their active participation in providing the “carrots” (and eventually “sticks”) for adherence to these recommendations.

However, we in the ISPOR research community are very aware of what good outcomes research entails and are already holding ourselves accountable and should continue to do so publicly. By actively registering our studies, we are putting a stake in the ground and holding up health economics and outcomes research (HEOR) to the same bar as any other clinical research. HEOR deserves a seat at the evidence-based medicine table. We would even say that evidence-based medicine cannot fully occur until it includes HEOR. But to be there, we need to continually foster the validity of our research and research methods. Now is the time to highlight and increase the external perception of what we do and how we do it. We welcome your input and ask you to watch for the announcement of the ISPOR-sponsored study registry. • 

References

1. Orsini LS, Berger M, Crown W, et al. Improving transparency to build trust in real-world secondary data studies for hypothesis testing—why, what, and how: recommendations and a road map from the real-world evidence transparency initiative. Value Health. 2020; 23(9):1128-1136.

2. Makady A, ten Ham R, de Boer A, et al. Policies for use of real-world data in health technology assessment (HTA): a comparative study of six HTA agencies. Value Health. 2017; 20(4):520-532. https://www.ispor.org/publications/journals/value-in-health/abstract/Volume-20--Issue-4/Policies-for-Use-of-Real-World-Data-in-Health-Technology-Assessment-(HTA)--A-Comparative-Study-of-Six-HTA-Agencies.

3. Malone D, Brown M, Hurwitz J, Peters L, Graff J. Real-world evidence: useful in the real world of US payer decision making? How? When? And what studies? Value Health. 2018; 21(3):326-333. http://dx.doi.org/10.1016/j/jval.2017.08.3013.

4. 114th Congress. H.R.34 - 21st Century Cures Act. https://www.congress.gov/bill/114th-congress/house-bill/34/text. Accessed September 18, 2020.

5. Completed PDUFA PDI Deliverables. https://www.fda.gov/ForIndustry/UserFees/PrescriptionDrugUserFee/ucm446608.htm. Accessed September 18, 2020.

6. Government of Canada. Optimizing the Use of Real-World Evidence to Inform Regulatory Decision Making. https://www.canada.ca/en/health-canada/services/drugs-health-products/drug-products/announcements/optimizing-real-world-evidence-regulatory-decisions.html. Accessed September 18, 2020.

7. Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Retraction: Cardiovascular disease, drug therapy, and mortality in Covid-19. NEJM. 2020; 382:2582, DOI: 10.1056/NEJMc2021225.

8. Mehra MR, Ruschitzka F, Patel AN. Retraction—Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet. 2020; 395:1820. https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)31180-6/fulltext.

9. WHO Pauses Hydroxychloroquine Clinical Trial. https://www.webmd.com/lung/news/20200527/who-pauses-hydroxychloroquine-clinical-trial. Accessed September 18, 2020.

10. Wang SV, Pinheiro S, Hua W, et al. Manuscript: A structured template and reporting tool for real-world evidence studies to facilitate unambiguous communication of methods and validity assessment (STaRT-RWE); 2020. https://doi.org/10.7910/DVN/6R1KCA, Harvard Dataverse, V1.