Early Reflections on Stakeholder Engagement in Economic Model Development to Inform Value Assessment
Richard Z. Xie, PhD; Jennifer Bright, MPA; Erica deFur Malik, MTS; Richard H. Chapman, PhD; The Innovation and Value Initiative, Alexandria, VA, USA
In value assessments, we strive to ensure maximum benefits for resources used while identifying treatments appropriate and beneficial to individual patients. However, stakeholders may have very different perspectives of what constitutes value and how to measure it. In recent years, the health economics and outcomes research (HEOR) community has increasingly called for a more open and transparent process that engaged different stakeholders (eg, with patients) to account for such differences in methods and procedures to inform value assessments.1-3 However, the “how-to” of a continual stakeholder engagement process and its impacts on value assessments remain an understudied area.
Advancing Best Practices in Value Assessments
The nonprofit Innovation and Value Initiative (IVI) is dedicated to working with different stakeholders to advance the methods and practice of value assessments. A focal area for IVI is developing flexible and rigorous economic models in specific disease areas as part of its open source value platform (OSVP), to demonstrate novel methods and best practices in value assessments that can meet the decision needs of diverse stakeholders.
In July 2020, we launched an initiative to build its third OSVP model, focusing on major depressive disorder.4 As part of
this initiative, we are testing a novel approach to economic model development (Figure 1), wherein diverse stakeholders are engaged from the outset and throughout different phases of model development (ie, model design, construction, validation, and application). We hypothesize that such an approach will lead to a more relevant and useful economic model that can generate meaningful insights for different stakeholders.
Figure 1. A continual stakeholder engagement approach to economic model development.
Early Stages and Insights
As a first step, we established a 20-member multistakeholder advisory group including thought leaders and experts representing patient groups (n=5), industry (n=3), researchers (n=2), clinicians (n=5), payers (n=2), and employers (n=5). (Note that certain members represented more than one perspective.)
A year after launching this effort, we worked with the advisory group to finalize the model scope document and is currently working towards finalizing the model protocol. Our early stage findings show that continual stakeholder engagement yields important insights for the model design and the outcomes that should be included in economic models to be more widely considered by different decision makers (Figure 2).5
Figure 2. Examples of advisory group input informing model development.
This article summarizes the operational considerations and lessons learned in this ongoing process of stakeholder engagement for our OSVP model development. We share these insights to assist other researchers to implement a continual stakeholder engagement approach, and to increase the body of work that demonstrates the importance of engagement in development and application of new methods to inform value assessments (Figure 3).
Figure 3. Key considerations for continual stakeholder engagement.
Launching the Stakeholder Group
IVI’s premise at the outset was that the OSVP model development should be informed by stakeholder guidance. Therefore, our first task was to establish a multistakeholder advisory group representing diverse perspectives. Before reaching out to stakeholders, we took the important step to clearly define the roles and responsibilities of the advisory group and the expected level of engagement. In our initiative, the role of the advisory group was to provide perspectives and insights that inform and validate assumptions and technical choices. We also specified the approximate duration and frequency of engagement over the course of model development. These details were described in a brief overview document shared with the potential participants during outreach.
Identifying the individuals with the relevant expertise, interest, and availability to participate in such an effort is perhaps the most challenging aspect of launching this type of initiative. We reached out to IVI’s advisory boards, membership network, and research partners to seek suggestions on potential candidates. To ensure that we heard from a broader community of stakeholders, we made a deliberate effort to identify candidates that were not in our immediate connection circles. We then shared the brief overview document with these candidates and met via teleconference to discuss the project. For those who were not able to participate, we asked for recommendations and referrals.
Facilitating Continual Engagement
Prioritizing Questions for Discussions
Throughout engagement, we have found it important to prioritize the questions and tailor the process for feedback elicitation based on the expertise of the participants and type of communication channel (eg, group discussions). In the early stages, we held monthly full-group meetings (of 90-minute duration). In such meetings, we found that we were able to address 2-3 discussion topics per meeting at most. Broader and conceptual questions (eg, patient factors missing from the current value assessments in major depressive disorder) are more suited for full-group discussions, especially during the initial conceptualization phase (eg, key modeling objectives) of model development, while more specific technical questions (eg, statistical techniques to extrapolate long-term efficacy rates) require small-group discussions with experts with relevant experiences.
Utilize Multiple Communication Channels
We have used a variety of communication channels to elicit feedback, including email updates, surveys, small- and large-group discussions, and individual conversations. We have found that it is important to tailor the communication approach based on the questions, areas of expertise of the stakeholders, and individual preferences for providing input. In full-group discussions, we prepare prereads to provide advisory group members with uniform and sufficient background and prepare specific questions to elicit high-quality feedback. Since this is an ongoing learning process, we have worked to adapt based on feedback from stakeholders and to customize communication approaches. For example, we are currently exploring developing video clips to better illustrate key questions.
Respect Participants’ Time
Time, or the lack thereof, is a major challenge. We have attempted to avoid over-burdening the advisory group with meetings and lengthy document reviews, but also keeping them informed of our project. On average, we keep the monthly time commitment to be less than 2 hours. Communications channels can be tailored to facilitate input collection, and researchers should adhere to the delineated roles and responsibilities of the advisory group specified at the launch of the group. We have opted to acknowledge stakeholders’ time and contributions by providing an honorarium to all participants. If a project has a limited budget, it is especially important to compensate people with the condition of interest providing patient-specific information.
Work To Ensure Transparency
To build trust and foster an open environment for learning and discussion, research teams should take steps to ensure transparency throughout the process. In our effort, we have documented all discussions and feedback, and created a shared online folder where participants can easily access all materials. For the major depressive disorder advisory group, we communicated from the outset that: (1) the role of the advisory group is to advise rather than to vote on decisions, (2) the meetings should be viewed as brainstorming sessions, and (3) we might not be able to consider all recommendations in the initial version of the model. As a result, the advisory group felt comfortable in brainstorming as a group. Participants are free to use information from discussions in an anonymized format. Following feedback elicitation, we synthesized learnings and shared with the advisory group how their insights informed the model design. In prioritizing responses based on feedback, we also provided clear explanations to the advisory group of the rationale for our decisions (more on this below).
Lessons From Stakeholder Engagement
“Translations” Are Sometimes Needed
In aggregating feedback, researchers might need to translate both qualitative input into quantitative input for modeling purposes, and technical details for a less technical audience. On occasion, this required additional literature searches and follow-up discussions with specific stakeholders. For example, a key concept proposed by an advisory group member was “career disruption,” where an individual with Major depressive disorder might not achieve full career potential over a lifetime. To translate this in the modeling context, we engaged with several members in small-group discussions to understand ways to measure “career potential” and conducted literature searches to identify existing estimates of such influences.
Another important aspect of “translation” lies in the use of appropriate explanatory methods and terminologies for different stakeholder groups. For example, the term “patient” may be commonly used by the HEOR community but may be rejected by people with the condition being considered. This requires researchers to be aware of such differences and actively solicit input on appropriate terminologies, types of information, and modes of communication to ensure full participation across stakeholder groups.
Prioritize Feedback
In aggregating feedback, we continue to explore how to prioritize suggestions from different stakeholders and, at times, reconcile conflicting perspectives. The specific criteria we have developed, based on the modeling objectives and the mission of the organization,6 include prioritizing: (1) recommendations with the broadest stakeholder buy-in, (2) recommendations from traditionally under-represented stakeholders (eg, patients and employers in our case), (3) recommendations that are most feasible to implement in the short-term, and (4) recommendations that are most likely to have policy- or decision-making impacts.
Since the development of OSVP models is an ongoing process, feedback that we cannot incorporate into the initial version of the model is identified for future model updates. We continue to work with the advisory group to ensure that the base model includes structural placeholders for such updates and design future studies that will generate key data to address those updates.
Accounting for Missing Stakeholder Perspectives
Research teams might not be able to recruit representatives from all desired stakeholder groups. In our case, despite best efforts, we do not have representatives from public payers or government agencies (eg, Centers for Medicare and Medicaid Services). Research teams can consider conducting individual outreach to these stakeholder representatives throughout the different phases. For example, we reached out to specific individuals who represent or have had extensive experience working with such stakeholders (eg, state Medicaid agencies) during the model scope public comment period to solicit specific feedback and will continue to do so in finalizing the model protocol and developing use cases.
Continuing Stakeholder Engagement
Applying a continual stakeholder engagement approach to economic model development has been an important learning opportunity for IVI and confirms our view that all value assessors should engage with diverse stakeholders early and often. Although certainly a complex and intensive enterprise, the insights from the advisory group have already informed our model design, and have led to new partnerships and approaches to the model (Figure 2).5 Thus, we can affirm that stakeholder engagement has the potential to improve the value assessment methods and lead to economic models more useful and relevant to different stakeholders. Benefits accrue to the participants as well: through consistent engagement, the stakeholder participants also experienced direct exchange of experience and perspectives. While consensus for some key modeling considerations did not always result, such interaction increased learning about the viewpoints of each other and built trust in an open environment for closer collaboration in the future.
Closing Thoughts
Our next challenge is working with the advisory group to finalize the model protocol and validate the model prototype. Each interaction expands our thinking about the inputs, perspectives, and model functionality that are important to decision makers across the spectrum. And each is an opportunity to build our continuous learning, open-source model development approach. In keeping with our principles, IVI will continue to share our experiences and real-time learning with the HEOR community to contribute to improved methods and practice in value assessment. Follow along at thevalueinitiative.org.
References
1. Sorenson C, Lavezzari G, Daniel G, et al. Advancing value assessment in the United States: a multistakeholder perspective. Value Health. 2017;20(2):299-307. doi:10.1016/j.jval.2016.11.030.
2. National Health Council. The Patient Voice in Value: The National Health Council Patient-Centered Value Model Rubric. Published March 2016. Accessed January 31, 2021.
3. PCORI Engagement Rubric. Published February 4, 2014. Accessed October 8, 2021.
4. Innovation and Value Initiative. IVI-MDD Value Model–Innovation and Value Initiative. Accessed January 31, 2021.
5. Xie RZ, deFur Malik E, Linthicum MT, Bright JL. Putting stakeholder engagement at the center of health economic modeling for health technology assessment in the United States. PharmacoEconomics. 2021;39(6):631-638. doi:10.1007/S40273-021-01036-3.
6. Innovation and Value Initiative. Principles for Value Assessment in the US. Accessed October 11, 2021.
