T. Joseph Mattingly II, PharmD, MBA, PhD; Julia F. Slejko, PhD, University of Maryland School of Pharmacy, Baltimore, MD, USA; Elisabeth M. Oehrlein, PhD, MS, National Health Council, Washington DC, USA; Eleanor M. Perfetto, PhD, MS, National Health Council, Washington DC, USA ; and University of Maryland School of Pharmacy, Baltimore, MD, USA

 

SUMMARY

Consistently and reliably incorporating patient-centered outcomes within value assessment cannot be onerous or overly burdensome for patients or economic researchers. Approaches to identify, synthesize, and disseminate patient-centered outcome data in a way that can facilitate the inclusion of these outcomes in more cost-effectiveness analyses and value assessments must ideally be practical and feasible, or they will be met with resistance, which could mitigate the impact a patient-centered approach can have on rigor, validity, and use of findings. Thus, the objective of this paper is to provide practical guidance on a process of identifying and presenting patient-centered outcomes in a way that makes it easier to include in value assessment, thereby nudging more economists to choose to include these in their models. The process includes forming a multistakeholder, patient-centered advisory board, engaging the board in the research agenda, conducting evidence synthesis and qualitative research to ensure viewpoints are not missed, and disseminating findings to multistakeholder audiences. Finally, a publicly available, centralized database of identified patient-centered value elements should be created to increase the likelihood of their uptake in value assessment.

 

Introduction

Value assessment is intended as a tool for evaluating healthcare treatments to gauge value and inform decisions.¹ Many existing value assessments fail to account for all “elements of value” that may be important to payers, society, and—most importantly—patients (eg, treatment tolerability, productivity, time, fear of contagion, spillovers, hope, social impact).^2,3 Value assessments typically incorporate cost-effectiveness analysis (CEA), focusing on costs and outcomes important to payers, missing important information to optimize resource allocation at a societal level.^4,5 The societal perspective is likely the closest we have to a patient-centered model, but that is only true if the outcomes selected are also what patients care about.^6,7 A recent review of 6907 CEAs found that many studies reporting a societal perspective were actually mislabeled.⁵ Further, when CEAs do incorporate variables for a “societal perspective” model, they frequently lack high-quality evidence to support the assumptions for these unique elements of value. These analytical difficulties have even led experts to recommend that the field embrace these challenges and adopt a standard for a “limited societal perspective” that attempts to incorporate time costs, opportunity costs, and community preferences.⁸

As value assessments are developed, the sources of evidence, methods, and assumptions of the underlying model may vary greatly between organizations or between assessments by the same organization.⁹ The Second Panel on Cost-Effectiveness in Health and Medicine further identified the process of evidence synthesis for CEAs as a key area for further work needed in the field, citing a lack of underlying theory guiding which studies to synthesize, assessing how studies may be biased, and assessing how findings generalize to a target population.¹⁰ A recent systematic review of cost-of-illness evidence for hepatitis C virus infection described significant challenges in quantifying the full disease burden due to significant heterogeneity in identification of different types of costs, high risk of bias for many common cost variables, and difficulty in capturing some components reported by patients (eg, fear of harming others, insurance issues, or stigma).¹¹ Attempting to determine the value for a new treatment without considering all of the evidence or without adjusting for bias in included evidence may produce biased results or findings that are difficult to validate.

These failings cannot be fully dismissed by simply describing them in a “limitations” section without raising the question: Would value assessment be better if we more intentionally capture commonly neglected elements of value? However, this question poses a challenge. Consistently and reliably incorporating patient-centered outcomes within value assessment cannot be onerous or overly burdensome for patients or economic researchers. Approaches to identify, synthesize, and disseminate patient-centered outcome data in a way that can facilitate the inclusion of these outcomes in more CEAs and value assessments must ideally be practical and feasible, or they will be met with resistance, which could mitigate the impact a patient-centered approach can have on rigor, validity, and use of findings. Thus, the objective of this paper is to provide practical guidance on a process of identifying and presenting patient-centered outcomes in a way that makes it easier to include in value assessment, thereby nudging more economists to choose to include these in their models.¹²

 

Forming and Continuously Engaging a Stakeholder Advisory Board

Before a value assessment process begins, we recommend starting with the formation of a multistakeholder, patient-centered advisory board that captures a variety of perspectives (eg, patients, caregivers, providers, payers, manufacturers, employers, and researchers). Stakeholder engagement, with an emphasis on patient engagement, is considered a foundational component of patient-centered outcomes research and, more specifically, meaningful patient engagement is recommended in the process of formal value assessment (health technology assessment bodies or value assessment frameworks) for a variety of reasons.^13-16 The Patient-Centered Outcomes Research Institute (PCORI) considers stakeholder engagement a critical component of its merit-review process and guides applicants with a detailed rubric describing several engagement principles (reciprocal relationships, colearning, partnership, trust, transparency, and honesty).¹⁷

 

"Would value assessment be better if we more intentionally capture commonly neglected elements of value?"

 

Once the board is formed, the roles and responsibilities of the board must be made clear. The board should have and understand its governance role and it should be obvious that its role is not to be a “rubber stamp” for what the researchers decide to do. We also recommend a formal plan to continuously engage the board throughout the entire project life cycle to enable multiple points for gathering input, collecting feedback, and demonstrating a true partnership between the research team and board.^7,17-19 Further, board members should be provided fair compensation for their time and the engagement strategy should be thoughtful of the time commitment expected of each member.^16,17 When multistakeholder advisory boards are meaningfully engaged, iterative improvements throughout the project planning, execution, and dissemination phases can help increase the patient-centeredness of the assessment.^20-22 Having an advisory board does not guarantee a high level of engagement, but rather provides a mechanism for engagement to occur. It is still up to the research team to develop meeting agendas with ample time for discussion, deliberation, voting, and “open microphone” time to let board members share their perspectives. Take real queues from the board and allow them to truly guide the research agenda.¹⁹

 

Identifying, Synthesizing, and Utilizing Patient-Centered Outcomes in Economic Evaluations

Identification and planning

Following advisory board formation, a research agenda must be determined. A formalized process of topic solicitation enables advisory board and research team members to deliberate and approve projects. Articulating this process in a governance document improves the transparency of decision making for all participants. Research agendas can be driven by financial support or other sustainability considerations. Treating the advisory board as a true decision-making body requires open and honest communication about all factors that could influence topic selection.

Once multiple topics have been identified, different experts may be needed to provide subject matter consultation or may be used to help with the work itself. The Institute for Clinical and Economic Review (ICER), a US-based organization that conducts healthcare-related value assessment, reports its process to engage a variety of stakeholders at various levels—from formal advisors to contracting external experts—to conduct a large proportion of their work.²³ If you plan to contract out major components of the value assessment work, a standard set of methods may improve consistency and reproducibility. Additionally, when contracting external organizations to lead a component of the value assessment, it may be important to reiterate the importance of the patient-centered advisory board throughout the process. Why bother engaging a multistakeholder advisory board in the first place if the patient perspectives just get drowned out by external contractors who may not be committed to patient centricity?

After consultation with topic experts to assess the current scope of the project and information needs, presenting this new information will help the advisory board and research team prioritize topics and document patient and other stakeholder comments or concerns. This step is critical to demonstrate to everyone that the research team is committed to incorporating this feedback and, if gaps exist, those can be identified early in the process. There should also be opportunity for additional stakeholders to contribute their perspectives on an appropriate framework for a value assessment.²⁴

 Synthesis and discovery

Synthesis and discovery should include not just systematically reviewing existing literature, but simultaneously beginning qualitative research to identify topics potentially missed in past research. Keep in mind that past literature is likely based on research that did not include patient engagement and can be biased toward inclusion of research questions and outcomes considered important to clinicians and researchers rather than patients. Thus, qualitative research with patients and caregivers will typically be needed. Evidence identified through published literature, interviews, and focus groups should be presented to the advisory board and potentially considered for poster or manuscript development.

With CEAs and value assessments relying on quantitative modeling techniques, gaps identified through literature review and qualitative research should inform new observational and/or experimental research. For example, if a disease typically imposes substantial burden on caregivers, but there are no relevant, existing cost-of-illness or health-utility elicitation studies with a focus on the caregiver, the research team may need to focus on filling these gaps before proceeding with the value assessment. Now, suggesting that health economists charged with building economic models are also responsible for conducting qualitative interviews, focus groups, or lead patient engagement activities might create additional problems without the proper training. The process of value assessment is, by necessity, a multidisciplinary science. It is likely unreasonable to expect any single organization to be responsible for filling all of these research gaps prior to proceeding. But this gap analysis should be presented to the advisory board for a final determination of whether the research team should proceed with the value assessment with the missing data given the impact of that variable.

It should also be noted that using rigorous methods during the synthesis and discovery stage improves the likelihood of publishing findings in a peer-reviewed journal and helps build the case for future funding if any gaps are identified where additional research is needed. Communicating these gaps to major federal agencies (eg, PCORI, Agency for Healthcare Research and Quality, National Institutes of Health), patient advocacy organizations, or other groups with a potential interest (eg, industry, insurers, foundations) could raise awareness in a way that could lead to future funding opportunities for research.

Dissemination

As the body of evidence for patient-centered outcomes grows, the stakeholder advisory board’s role is to help with reviewing and interpreting findings to assist to optimize communication and dissemination.^13,18 Engaging patients and other stakeholders in the dissemination phase enables thoughtful consideration of the most appropriate channels for communicating results. Publication of patient-centered outcome evidence synthesized through systematic review or discovered through research in traditional academic journals should still be a priority so that results are peer reviewed and considered to have followed good methods practices. However, the dissemination should not stop at academic journals or professional meetings. The research team should go further to ensure that patients, policy makers, and the public at large are able to understand the findings.¹⁸ This might include drafting patient guides or specifically working with patient advocacy organizations or patient navigators to get the information out in a way that fits the audience.¹⁸

 

Future Directions and Recommendations

Make data available and easily accessible

To further increase the likelihood that the identified value elements will be incorporated into value assessments, a publicly available, centralized database should be created and maintained to improve ready access. The CEA Registry, created and maintained by Tufts Center for the Evaluation of Value and Risk in Health, provides a similar service through a process of systematically reviewing published CEAs and synthesizing article information, cost-effectiveness ratios, and utility weights reported, allowing for users to quickly identify variables useful for economic analyses.²⁵ Currently, PCORI also offers a searchable database of funded projects on its website but key patient-centered outcomes aren’t extracted and synthesized in a user-friendly manner.²⁶ The PCORI infrastructure combined with our proposed process could make it much easier for value assessment developers to identify patient-centered outcomes that have been curated in a way that is consistent with PCORI’s own stated foundational elements required for patient-centered outcomes research.¹³

 Assess quality of data

Posting a set of patient-centered outcomes to a searchable database could improve the efficiency of value assessment development, but the job would not be complete without further context around the evidence used to support those outcomes. Any elements identified and quantitative variables posted should include some assessment of quality of the data and an assessment of the risk of bias when using the element in a value assessment. While no theory of evidence synthesis for inputs in a CEA currently exist, other standards for evidence rating and quality of evidence for recommendations in healthcare could serve as a template to guide how we consider existing patient-centered outcomes.^10,27 Providing some context around the quality of evidence available would help value assessment researchers consider the level of uncertainty for different components of their models and consider different sensitivity and scenario analyses to address.^28,29

Commit to transparency

While methods and model transparency have been recommended and advocated for in health economics for many years, simply having access to programming code and value assessment inputs is not enough.³⁰ ISPOR and the Society for Medical Decision Making Good Research Practices in Modeling Task Force described transparency in the economic modeling context as “the extent to which interested parties can review a model’s structure, equations, parameter values, and assumptions” with the intention “to provide sufficient information to enable the full spectrum of readers to understand a model’s accuracy.”³⁰ While publishing CEA protocols or making models publicly available in a repository could address transparency and accountability in the health economics research community, these steps may not improve the transparency to the patient community or general public.^7,10,31,32 Emerging practices such as providing nontechnical documentation, cataloging questions and concerns, documenting how concerns will be addressed, and providing updates can help improve the level of transparency for patient communities.¹⁶ We also recommend specifically highlighting patient-centered outcome considerations in results tables and presenting these to the advisory board for confirmation.

 

Conclusion

When significant evidence gaps exist, it is not enough to simply throw these flaws into the abyss of limitations and sensitivity analyses. Value assessment researchers should engage a broad community of stakeholders early and often, with an emphasis on the patient, and seek to fill these gaps. Health economists should not be expected to go down this path alone, but in collaboration with different disciplines with the skills needed.

References

1. Neumann PJ, Willke RJ, Garrison LP. A health economics approach to US value assessment frameworks—introduction: an ISPOR Special Task Force Report [1]. Value Health. 2018;21(2):119-123. doi:10.1016/j.jval.2017.12.012.

2. Lakdawalla DN, Doshi JA, Garrison LP, Phelps CE, Basu A, Danzon PM. Defining elements of value in health care—a health economics approach: an ISPOR Special Task Force Report [3]. Value Health. 2018;21(2):131-139. doi:10.1016/j.jval.2017.12.007.

3. dosReis S, Butler B, Caicedo J, et al. Stakeholder-engaged derivation of patient-informed value elements. Patient. 2020;(Online ahead of print July 17):1-11. doi:10.1007/s40271-020-00433-8.

4. Neumann PJ. Costing and perspective in published cost-effectiveness analysis. Med Care. 2009;47(7 suppl 1):S29-S32. doi:10.1097/mlr.0b013e31819bc09d.

5. Kim DD, Silver MC, Kunst N, Cohen JT, Ollendorf DA, Neumann PJ. Perspective and costing in cost-effectiveness analysis, 1974–2018. Pharmacoeconomics. 2020;(Online ahead of print July 22):1-11. doi:10.1007/s40273-020-00942-2.

6. Slejko JF, Mattingly II TJ, Mullins CD, Perfetto EM, dosReis S. Future of patients in healthcare evaluation: the patient-informed reference case. Value Health. 2019;22(5):545-548. doi:10.1016/j.jval.2019.02.003.

7. Perfetto EM, Oehrlein EM, Boutin M, Reid S, Gascho E. Value to whom? The patient voice in the value discussion. Value Health. 2017;20(2):286-291. doi:10.1016/j.jval.2016.11.014.

8. Garrison LP, Mansley EC, Abbott TA, Bresnahan BW, Hay JW, Smeeding J. Good research practices for measuring drug costs in cost-effectiveness analyses: a societal perspective: The ISPOR drug cost task force report - Part II. Value Health. 2010;13(1):8-13. doi:10.1111/j.1524-4733.2009.00660.x.

9. Desai B, Mattingly TJ, van den Broek RWM, et al. Peer review and transparency in evidence-source selection in value and health technology assessment. Value Health. 2020;23(6):689-696. doi:10.1016/j.jval.2020.01.014.

10. Neumann PJ, Kim DD, Trikalinos TA, et al. Future directions for cost-effectiveness analyses in health and medicine. Med Decis Mak. 2018;38(7):767-777. doi:10.1177/0272989X18798833.

11. Mattingly II TJ, Love BL, Khokhar B. Real-world cost-of-illness evidence in hepatitis C virus: a systematic review. Pharmacoeconomics. 2020;38:927-939. doi:10.1007/s40273-020-00933-3.

12. Thaler RH, Sunstein CR. Nudge: Improving Decisions About Health, Wealth, and Happiness. Penguin Group; 2009.

13. Frank L, Forsythe L, Ellis L, et al. Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute. Qual Life Res. 2015;24(5):1033-1041. doi:10.1007/s11136-014-0893-3.

14. Facey K, Boivin A, Gracia J, et al. Patients’ perspectives in health technology assessment: a route to robust evidence and fair deliberation. Int J Technol Assess Health Care. 2010;26(3):334-340. doi:10.1017/S0266462310000395.

15. National Health Council. The patient voice in value: the National Health Council patient-centered value model rubric. https://www.nationalhealthcouncil.org/wp-content/uploads/2019/12/Value-Rubric-March-2016-v2.pdf. Published 2016. Accessed September 14, 2020.

16. Perfetto EM, Harris J, Mullins CD, dosReis S. Emerging good practices for transforming value assessment: patients’ voices, patients’ values. Value Health. 2018;21:386-393. https://www.valueinhealthjournal.com/article/S1098-3015(18)30193-1/fulltext.

17. Patient-Centred Outcomes Research Institute. Engagement Rubric for Applicants. 2016. https://www.pcori.org/sites/default/files/Engagement-Rubric.pdf. Accessed September 5, 2020.

18. Mullins CD, Abdulhalim AM, Lavallee DC. Continuous patient engagement in comparative effectiveness research. JAMA. 2012;307(15):1587-1588.

19. Oehrlein EM, Love TR, Anyanwu C, Hanna ML, Kraska J, Perfetto EM. Multi-method patient-engagement approach: a case example from a PCORI-funded training project. Patient. 2019;12:277-280. doi:10.1007/s40271-018-0341-5.

20. Devine EB, Alfonso-Cristancho R, Devlin A, et al. A model for incorporating patient and stakeholder voices in a learning health care network: Washington State’s comparative effectiveness research translation network. J Clin Epidemiol. 2013;66(8 suppl ):S122-129. doi:10.1016/j.jclinepi.2013.04.007.

21. dosReis S, Castillo WC, Ross M, Fitz-Randolph M, Vaughn-Lee A, Butler B. Attribute development using continuous stakeholder engagement to prioritize treatment decisions: a framework for patient-centered research. Value Health. 2016;19(6):758-766. doi:10.1016/j.jval.2016.02.013.

22. Mattingly II TJ, Slejko JF, Onukwugha E, Perfetto EM, Kottilil S, Mullins CD. Value in hepatitis C virus treatment: a patient-centered cost-effectiveness analysis. Pharmacoeconomics. 2020;38(2):233-242. doi:10.1007/s40273-019-00864-8.

23. Institute for Clinical and Economic Review. How We Do Our Work. https://icer-review.org/methodology/icers-methods/. Published 2020. Accessed September 2, 2020.

24. National Health Council. A Dialogue on Patient-Centered Value Assessment: Overcoming Barriers to Amplify the Patient Voice. https://www.nationalhealthcouncil.org/dialogue-patient-centered-value-assessment-overcoming-barriers-amplify-patient-voice. Published 2018. Accessed September 14, 2020.

25. Tufts Center for the Evaluation of Value and Risk in Health. CEA Registry. https://cevr.tuftsmedicalcenter.org/databases/cea-registry. Published 2020. Accessed September 5, 2020.

26. Patient-Centered Outcomes Research Institute. Research & Results. https://www.pcori.org/research-results-home. Published 2020. Accessed September 5, 2020.

27. Guyatt GH, Oxman AD, Schünemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol. 2011;64(4):380-382. doi:10.1016/j.jclinepi.2010.09.011.

28. Jain R, Grabner M, Onukwugha E. Sensitivity analysis in cost-effectiveness studies. Pharmacoeconomics. 2011;29(4):297-314.

29. Briggs AH. Handling uncertainty in cost-effectiveness models. Pharmacoeconomics. 2000;17(5):479-500.

30. Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value Health. 2012;15(6):843-850. doi: 10.1016/j.jval.2012.04.012.

31. Cohen JT, Neumann PJ, Wong JB. A call for open-source cost-effectiveness analysis. Ann Intern Med. 2017;167(6):432-433. doi:10.7326/M17-1153.

32. Padula WV, McQueen RB, Pronovost PJ. Finding resolution for the responsible transparency of economic models in health and medicine. Med Care. 2017;55(11):915-917. doi:10.1097/MLR.0000000000000813.