Registering Study Protocols: Helping Real-World Evidence Come of Age
Richard J. Willke, PhD, Chief Science Officer, ISPOR, Lawrenceville, NJ, USA; Shirley V. Wang, PhD, ScM, Associate Professor, Department of Medicine, Brigham and Women’s Hospital, Boston, MA, USA; and Harvard Medical School, Boston, MA, USA
Researchers often ask why they should consider preregistering their real-world data (RWD) study protocols. Prespecifying analytic plans and publicly registering protocols before analyzing data are part of ISPOR-ISPE’s good study conduct recommendations for studies that evaluate hypotheses about treatment effects through secondary analysis of observational data.1,2 Documenting the initial plans in a protocol, and subsequent amendments to the plan, can help assure readers that the study results were not selected or manipulated. Of course, the validity of study findings will still depend on the relevance and reliability of the data and methods. Nevertheless, having a clearly specified protocol registered will increase the reproducibility of RWD study findings as well as the ability of end users to evaluate the rigor of study methods.3 If real-world evidence (RWE) is to be seriously considered for healthcare decision making as a useful complement to randomized controlled trial data, it must become “adult” in the sense of satisfying the same requirements for registration, sharing of study protocols, and results.
ISPOR recently launched a registry for RWE study protocols, in collaboration with ISPE, Duke-Margolis Center for Health Policy, and the National Pharmaceutical Council. The registry is a key feature of our RWE Transparency Initiative, which published its deliberations and plans last year.4 As part of this initiative’s objective to grow a culture of transparency around RWE, we saw a need for an easy-to-use protocol registry for studies using RWD to evaluate treatment effects. We partnered with the Center for Open Science to develop the RWE Registry and use their online platform, called Open Science Forum. We initiated the registry in 2020, sought feedback from several early users, and made some modifications. We feel it is now ready for broader use and protocol registrations have begun.
Table. Real-World Evidence Registry elements.
A common question is how the RWE Registry compares to other registries. Researchers have a number of registry options, but the two best known for our field are probably clinicaltrials.gov and the European Union’s Post-Authorization Study registry (EU-PAS). Clinicaltrials.gov is intended for registering clinical trials but has been used for registration of noninterventional studies as well. However, it requires the user to answer many questions that aren’t normally relevant for the typical RWD study. EU-PAS is also a well-established study registry that is particularly important for postauthorization observational studies (usually safety oriented) needed for European regulatory purposes. Some studies are required to be registered on clinicaltrials.gov or EU-PAS. When that is the case, researchers should certainly use the required registry.
However, for the many researchers who don’t already have a mandate or workflow for study registration (eg, researchers in health technology assessment, health economics and outcomes research, payer organizations, and other data scientists), the RWE Registry is a simplified platform that is specifically designed for studies using data that have already been collected for other purposes, like claims or electronic health record data. The registration process has been streamlined to minimize administrative and irrelevant questions. There are 25 questions to answer regarding your research plan and your study team followed by the required upload of a more detailed protocol (see Table). More detailed instructions, with examples for each item, are available here. Instructional videos are available on ISPOR’s Real World Evidence Registry webpage. In addition, the registration is automatically linked to a “project” wiki space that allows registrants to store and share code, appendices, and other supporting materials for the project, all of which come with a citable DOI that is linked to the investigator’s ORCID ID.
The registration process asks that you attest to the stage of the study at the time of registration (item 24 in the Table). Possible responses range from “no data handling at this point” to “statistical comparisons completed”, so registration prior to analysis is not required, only preferable. Some data handling is common prior to finalizing a protocol to check on sample sizes, algorithms, etc. Hence the stage labeled “Conducted preliminary analyses, including descriptive statistics and observation of data distributions–but no analyses of association between treatment and outcome” would be considered a preregistration because the protocol was registered before inferential analyses were conducted.
Once one has completed the registration items and submitted it, an ISPOR staff “moderator” reviews it, confidentially, for completeness. If satisfactory, the moderator will approve it and the protocol is officially registered and posted. If not satisfactory for some reason, the moderator will send it back to the registrant with any questions or concerns.
In developing the RWE Registry, we understood that some research needs to remain confidential until publication. For that reason, the RWE Registry has an option to keep the date stamped study registration hidden from public view for some period of time (ie, an embargo feature) before making them public. While the registration is hidden from public view, links to date stamped study materials can be shared privately with peer reviewers, assessors, or other decision-making bodies.
There are other good reasons for using this Registry. In addition to enhancing the transparency of study processes that underpin study findings, many journals are now starting to look for and recognize manuscripts that are reporting on preregistered protocols. ISPOR’s own journal, Value in Health, has promised priority review for RWE manuscripts based on preregistered protocols. Also, as registration becomes more common, this Registry will provide a place for researchers to search for RWE studies based on their protocol characteristics. This aspect makes registration useful no matter what the stage of data analysis is.
The RWE Registry facilitates transparency and credibility by documenting the study protocol and its stage of registration relative to data analysis and by providing a forum to share study materials, code, and appendices over the life cycle of the study. Registering a study protocol improves study transparency, is part of good study conduct in general, and will help RWE achieve its needed role in healthcare decision making.
References
1. Cleary M. Unlocking the promise of real-world evidence. Value & Outcomes Spotlight. Sep/Oct 2020;6(5):13-16.
2. Berger ML, Sox H, Willke RJ, et al. Recommendations for good procedural practices for real-world data studies of treatment effectiveness and/or comparative effectiveness designed to inform healthcare decisions: Report of the joint ISPOR-ISPE Special Task Force on Real World Evidence in Heal. Value Health. 2017;20(8):1003-1008.
3. Wang S, Schneeweiss S, Berger M, et al. Reporting to improve reproducibility and facilitate validity assessment for healthcare database studies V1.0. Pharmacoepidemiol Drug Saf. 2017;26 (9):1018-1032.
4. Orsini L, Berger M, Crown W, et al. Improving transparency to build trust in real-world secondary data studies for hypothesis testing—why, what, and how: recommendations and a road map from the Real-World Evidence Transparency Initiative. Value Health. 23(9):1128-1136.
