The Many Tentacles of Estimands in HEOR—From PROs to RWD

Jessica Roydhouse PhD, Menzies Institute for Medical Research, University of Tasmania, Hobart, TAS, Australia; Roee Gutman PhD, Brown University School of Public Health, Providence, RI, USA; Pallavi Mishra-Kalyani PhD, Center for Drug Evaluation and Research, US Food and Drug Administration, Silver Spring, MD, USA; Stephanie Manson PhD, Novartis, East Hanover, NJ, USA

Estimands: What Are They and How Are They Useful?
What is the effect of an intervention on an outcome?
Determining the effect of an intervention involves examining the causal effect of that intervention. In a hypothetical world, to compare the effects of possible interventions for each patient, we would simultaneously assign patients to all interventions, observe their outcomes for all interventions, and identify differences between these outcomes. In the real world, we are only able to assign individuals to one intervention at a time and then see their outcomes for that intervention. Thus, we cannot estimate individual-level treatment effects using this mechanism. However, we can estimate sample-level or population-level treatment effects. Formally, an estimand is a “precise description of the treatment effect reflecting the clinical question posed by the trial objective. It summarizes at a population-level what the outcomes would be in the same patients under different treatments being compared.”¹ Therefore, the estimand provides a tool to address structural uncertainty in our data using preplanned analyses. In this article, we will use therapeutic studies as examples, but these issues are relevant to studies of any interventions.

Challenges in Assessing Treatment Effects
Because we cannot observe individuals under multiple interventions at the same point in time, we need to ensure that participants in all arms are similar on average to identify the effect of the intervention. Balancing participant characteristics across intervention arms is accomplished by randomization in randomized controlled trials (RCTs) and in real-world data (RWD) studies, by using procedures such as matching or weighting. In RCTs without complications, randomization supports the inference that the assigned treatment is causally related to the observed effects.

However, the clinical context of randomized or observational studies can present complications that lead to difficulties in measuring and interpreting observed effects. In studies with complications, there can be unplanned events that change the clinical course of patients. For example, patients may use a rescue medication or discontinue a medication because of lack of efficacy or because of adverse events. Such events are commonly referred to as intercurrent events. How and if these events are accounted for in the analysis can influence the understanding of the efficacy of interventions. This emphasizes the need to a priori construct a study design that addresses intercurrent events. Early engagement with statisticians, patients, and clinicians is important for identifying processes to address intercurrent events that may arise during the course of a study. The types of intercurrent events that may arise and the analytic strategies to address intercurrent events should be informed by the clinical context.

"The estimand provides a tool to address structural uncertainty in our data using preplanned analyses."

By carefully constructing study estimands, researchers can have a better understanding of the impact of interventions on the course of a disease while considering the influence of intercurrent events.

Estimands for PRO Data
Patient-centric trials are important and encouraged by regulatory agencies. Patient-reported outcomes (PROs) are one way to assess a patient’s experience during a trial. Although PRO data are valuable, hypotheses and objectives for PRO data are not always clearly stated. Analyses of PRO data are sometimes inconsistent in how they address intercurrent events that arise during the course of the trial.² Considering PRO data in the context of postrandomization, events can be useful in providing more encompassing information on the effect of interventions. For example, in oncology and other trials, patients may stop treatment and/or stop providing PRO data for many reasons, including death, disease progression, or intolerable drug side effects. There may be clinically relevant reasons to incorporate these events in a trial’s analytic strategy and overall design in different ways.

There may also be other events that can affect PRO endpoints that are important to consider. Consider a PRO endpoint of change from baseline in pain severity in a trial comparing 2 pain-relieving medications. It would be unethical to not allow the use of medications other than the trial medication, even if the use of those medications may improve patients’ pain and influence the endpoint. There are several possible estimands for estimating a treatment effect that consider this challenge. One would be to look at what would have happened in a hypothetical or imagined scenario where no additional medications were allowed. This estimand is usually modeled because it is hypothetical, and some patients are only observed while taking the additional medication. However, modeling can provide important information about the effect of treatment under different plausible scenarios. Another option would be to include PRO data only up to the point a nontrial medication is used while ensuring that assumptions based on randomization are not violated. A different estimand would examine the effect of treatment on pain severity using PRO data throughout the trial, regardless of whether additional medications were used. Alternatively, the use of additional medications could be considered a signal of pain severity and incorporated into a composite estimand. Lastly, an estimand may also describe the effect on a subset of patients (eg, patients who would not take additional medications regardless of their assigned treatment).

Figure 1. Considerations for Developing Estimands in HEOR 

Estimands for Real-World Data
Although trials are essential for comparing the effects of interventions, not all clinical questions can be answered using trial data. Moreover, trial eligibility criteria restrict some patients from participation. For example, the underrepresentation of older, sicker, and higher-risk patients in trials is a well-known issue.³ Nonetheless, understanding the effect of treatments for those patients is of interest. Observational studies and RWD offer the opportunity to understand treatment effects over a longer time period and in a broader population, but they require assumptions that cannot be verified from the data. Additionally, RWD can be relevant in trial contexts as external control data for single-arm trials.

Claims data are a critical and frequently used data source for RWD. Unlike a trial, RWD is already available before the study inception. To ensure the objectivity of such studies, it is important to define the estimands prior to examining and analyzing the outcomes.⁴ The definition of estimands in observational studies will be impacted by the structure of the databases. Not all relevant measurements or events may be included in claims data, or well-captured if they are included. This can make it difficult to define clinically relevant events and thus estimands. Consideration of the data structure and documentation of relevant events is critical for choosing estimands with RWD. For example, when using claims data, date of progression is often uncertain, and creating estimands for progression surrogates such as treatment duration can be challenging. In these situations, estimands might not be able to address this complexity, and additional sensitivity analyses would be required. Effect estimation that relies entirely on estimands should generally be limited to variables that can be supported by available data.

"Using estimands can help HEOR trialists and researchers measure and clearly communicate treatment effects."

Adjusting for rescue medication is one possible example of an estimand used in RWD. After defining which therapies might be considered as rescue medication, it is possible to create a composite endpoint that defines the progression of severe disease or death as the end of treatment medication or the introduction of a concurrent rescue therapy.

Planning Your Study
Systematic and careful consideration of the effects of intercurrent events on study endpoints is critical for choosing estimands. For registration trials, early communication with regulatory agencies is encouraged. This communication can ensure that the design and analysis plan are linked to the study objectives. Another important issue is sensitivity analysis. All statistical analyses rely on assumptions, and in all contexts (but especially in regulatory contexts) the sensitivity of study results to these assumptions is essential. Sensitivity analyses for estimands can assess the robustness of the treatment effect to different assumptions of the analytic methods. Trial and study planning should include sensitivity analyses, and these can be included in the discussion with regulatory agencies.

Critically, early input from patients and clinicians can also help choose the right set of estimands for a study. Talking to patients and understanding which treatment effects would be meaningful for them is important. Clinician input is essential for ensuring that the estimands are relevant in the clinical context. In collaboration with statisticians, patients, and researchers with relevant expertise, early planning and consideration can result in clinically relevant, meaningful estimands.

The collaboration needed for choosing the proper estimands is also important for interpreting estimand results. Carefully designed and planned estimands can make study results easier to interpret. Using estimands can help HEOR trialists and researchers measure and clearly communicate treatment effects. In turn, patients, regulators, and payers can use estimands to make more informed decisions.

References

1. Food and Drug Administration, US Department of Health and Human Services. E9(R1) Statistical Principles for Clinical Trials: Addendum: Estimands and Sensitivity Analysis

in Clinical Trials: Guidance for Industry. Silver Spring, MD. Accessed June 17, 2022. https://www.fda.gov/media/148473/download.

2. Bottomley A, Pe M, Sloan J, et al. Moving forward toward standardizing analysis of quality of life data in randomized cancer clinical trials. Clin Trials. 2018;15(6):624-630. doi:10.1177/1740774158795637

3. Jin S, Pazdur R, Sridhara R. Re-evaluating eligibility criteria for oncology clinical trials: analysis of investigational new drug applications in 2015. J Clin Oncol. 2017;35(33):3745-3752.

4. Rubin DB. For objective causal inference, design trumps analysis. Ann Appl Stat. 2008;2(3):808-840.