HEOR Articles

Sarah Stothers Rosenberg, RN, MSN, MPH, US Food and Drug Administration, Center for Drug Evaluation and Research, Silver Spring, MD, USA; Brittany B. Carson, PhD, ApotheCom, New York, NY, USA; Amiee Kang, MPH, Bristol Myers Squibb, Lawrenceville, NJ, USA; Ting-Hsuan Lee, MHS, US Food and Drug Administration, Center for Biologics and Research, Silver Spring, MD, USA; Rajshree Pandey, PhD, MPH, Curta Inc, Seattle, WA, USA; Evelyn J. Rizzo, MSc, Lumen Value & Access, New York, NY, USA

Introduction
Patient-reported outcomes and measures (PROs/PROMs) are increasingly recognized by regulators, healthcare providers, and policy makers as important tools to evaluate efficacy and safety of interventions from a patient’s perspective.^1,2 As a result, there is greater recognition of the value of PROs within both clinical research and practice (Table 1).^1,3,4  With the growing interest in digital healthcare alternatives, like telemedicine and remote clinical trials, we are seeing increased use of ePROs/ePROMs. Also, patients can now increasingly access ePRO systems using their personal devices. This type of electronic patient reporting was very important during the recent COVID-19 pandemic as it allowed for remote access to important patient outcomes, such as remotely monitoring symptoms in oncology patients.¹¹

Table 1. Terms and definitions.





The COVID-19 pandemic highlighted how social determinants of health impact health equity.^7,12,13 It also placed a spotlight on the ripple effect of inequities on healthcare delivery, access, and health outcomes.¹⁴ For example, people with higher incomes are oftentimes more able to work from home, have reduced exposure to COVID-19, and have better internet connectivity allowing for easier access to virtual healthcare, which can translate into relatively better health outcomes versus those with greater social or economic disadvantage(s).¹³

To better address health equity, current healthcare practices and structures should be reevaluated. This includes taking a closer look into the use of ePROs/ePROMs and their impact on health equity and related health outcomes. This article highlights several key considerations for the healthcare community regarding health equity in ePRO/ePROM development as well as operability in clinical research and practice, which are described below and summarized in the Figure.

 
Figure. Key health equity-related considerations across ePROM development
and operability in clinical research and practice.

Health Equity in the Design of ePROMs
The utilization of ePROMs in clinical trials and practice has increased. This is driven by the electronic design and format that facilitates robust data collection, analysis, and interpretation with fewer errors in comparison to traditional PROMs.^15,16 While the growth of ePRO use brings many benefits for data collection (eg, expanding our reach to underserved populations), challenges in design of ePROMs persist. If not addressed in a timely manner, these may contribute to health inequity. 

 For example, some key barriers to implementation of ePROMs identified in the oncology literature include: lack of inclusion of cross-cultural items, limited access to technology, reduced language or literacy, and other psychosocial stressors that contribute to limited usability.¹⁷ People who are socioeconomically disadvantaged, with lower literacy levels, who are non-native English speakers, and/or have a disability may find it difficult to complete ePROMs if they are not designed, translated, or adjusted to account for these factors. Not only do these challenges present barriers to accurate and timely patient input, but they may also increase unnecessary burden on patients and caregivers.^9,15

 While there is growing interest in the bring-your-own device approach where individuals use their personal smartphone, tablet, laptop, or other device to complete a measure, users in lower socioeconomic groups may lack access to the latest devices equipped with the operating systems needed to support the platforms used to host the ePROM.^9,18,19 In addition, lower socioeconomic groups may not have sufficient and stable internet access and their dependence on public internet (eg, mobile hotspots) can prove to be problematic given data security requirements.^9,15 Potential data breaches when using unsecure internet connections can subject patients and caregivers to privacy risks, which could lead to missing data and generalizability issues that result in inappropriate conclusions.

"While the growth of ePRO use brings many benefits for data collection, challenges in design of ePROMs persist. If not addressed in a timely manner, these may contribute to health inequity."

As with conventional PROMs, newly developed ePROMs require content validity and rigorous psychometric evaluation before use (eg, qualitative evidence and statistical evaluation of instrument validity and reliability). However, in contrast to conventional PROMs, ePROMs also require usability testing to assess the electronic platform/interface.^5,8,9 Usability testing evaluates the electronic platform for accessibility, security, and privacy.⁸

 During the design phase, usability testing of an ePROM and user interface is key in assessing whether the tool is fit for purpose and acceptable to the end user, or the target population.¹⁶ The design must not only consider the domains relevant to the concepts that go into understanding and assessing a disease or condition that are relevant to patients, but they must also consider cultural relevance, age distribution, and other demographic characteristics that may influence outcomes.^15,20 A careful design that considers these social determinants of health factors, in addition to disease factors, is needed to facilitate the instrument’s capture of meaningful data that are applicable to all relevant subpopulations.⁸ For instance, level of education can serve as a proxy measure for assessing literacy level, which is one of the many important components when evaluating content validity and performance of a PROM. 

 

Health Equity and ePROs/ePROMs in Clinical Trials
In clinical trials, ePROMs can provide a comprehensive assessment of the impact of a new intervention on a patient’s health-related quality of life through the collection of relevant symptoms and functional impacts that represent the outcome(s) of importance to patients.¹⁹

 ePROs/ePROMs can readily capture demographic data that may be relevant to better understand disease heterogeneity, response to an intervention, or the experiences of patients with the disease.^8,19 ePRO/ePROM data can contribute to real-world data and can be critical in evaluating a therapeutic intervention’s value as a part of the totality of evidence in understanding risks or a clinically meaningful benefit.^2,5,9 This is facilitated by the electronic design and format that enables robust and accurate data collection in comparison to traditional PROs (eg, paper-based instruments).^15,21  

However, a larger concern with ePROM implementation in clinical trials stems from the lack of racial and ethnic representation and consideration for social determinants of health.² Some key barriers to implementation of ePROMs identified in the oncology literature include lack of inclusion of cross-cultural items, access to technology, language, or literacy, and other psychosocial stressors that contribute to limited usability.¹⁷

 Furthermore, although the researchers have the ability to collect more granular data, the data are often analyzed and reported in aggregate to increase the power of the sample.¹⁵ This is likely due to missing demographic or related social determinants of health data,¹⁰ or “sparse data,” that lead to aggregate reporting of results. As a result, the important differences from ePROs that are relevant to various racial and ethnic, or socioeconomic groups may be masked.^9,22 Lack of careful implementation as well as interpretation of ePROs/ePROMs at the clinical trial level could lead to large generalizations that may have ripple effects elsewhere in the healthcare system. 

 Many of the ePROs/ePROMs challenges can be addressed in the design phase of a clinical trial (see Table 2 for resources). Clearly outlined protocols and a statistical analysis plan can mitigate some of the concerns related to ePROs/ePROMs and the role they can play in providing a more holistic picture of patients’ health. 

Table 2. Resources and best practices for addressing health equity in ePROs/ePROMs.

Health Equity and ePROs/ePROMs in Clinical Practice
Despite expansion of ePROs/ePROMs development, their widespread adoption in clinical practice is not fully realized due to technological challenges, workflow inefficiencies, and human factors.³ In clinical practice, ePROs/ePROMs may have direct impacts on the quality of care, access to treatment, and better identification of patients’ unmet needs;²³ in addition to health outcomes and early prediction of disease regression.²⁴ Anecdotal evidence suggests that ePROs/ePROMs can increase completion rates of important assessment measures.¹⁶ They also can serve as an opportunity to detect inequities related to health that may have traditionally gone unidentified.²²

Alternatively, ePROMs may pose a risk by helping to perpetuate health inequities when instrument evaluation does not include a representative sample population. As with the design of ePROMs, low computer literacy, lack of language proficiency, or limited access to technology may pose challenges due to an individual’s capacity to use or comprehensively interpret and answer these measures, which in turn can impact the later use of ePROs in healthcare management.^2,15 From the perspective of healthcare providers in oncology, barriers to using ePROMs frequently include limited access to technology and software, time constraints, lack of adherence and compliance in regular ePRO collection from patients, and difficulty in ePRO interpretation.^17,25

 

"ePRO/ePROM data can be used more effectively to improve health equity in healthcare decision making if the results are delivered in a meaningful way to patients/caregivers and their clinical team."

To facilitate accurate operability of ePRO/ePROMs, healthcare personnel will need to receive culturally competent training on how to use ePROs effectively in clinical practice. Moreover, when designing and implementing ePROMs, the burden on the healthcare provider or facility must be considered, especially in resource-constrained settings.²⁶ To further support sustainable implementation of ePROs, it is important to install leadership and governance structures within healthcare systems that are adaptable and that support the objectives of diverse healthcare settings.³

Finally, ePRO/ePROM data can be used more effectively to improve health equity in healthcare decision making if the results are delivered in a meaningful way to patients/caregivers and their clinical team.¹⁹ Research shows that for data to be relevant and actionable, different audiences may prefer alternative ways of communicating and interpretating results.²⁷ Providing ePRO/ePROM information in user-friendly and preferred formats could increase provider uptake and empower patients to be informed and active participants in their own care.^1,2 Furthermore, dissemination of any subgroup results from ePRO/ePROM analyses should be published along with other key clinical outcomes and/or findings to promote their value for future use.

 

Discussion and Concluding Remarks
At the core of any healthcare delivery system is the patient. With appropriate design and implementation, ePROMs may provide an opportunity to capture relevant sociodemographic information and important outcomes relevant to patients by broadening our reach to include the underserved populations that may otherwise be missed with traditional PRO data collection.

Despite these advantages, there remains limited evidence regarding the role of ePROs in highlighting health equity issues since much of the health equity data are often underrepresented in medical and health systems research.²⁸ Therefore, it is unclear whether the evidence we have holds true for specific subpopulations. Lack of representation can contribute to an electronic measure that has implicit biases built in that could potentially further perpetuate health inequity.

Opportunities remain in research where appropriate approaches for ePROs/ePROMs design and implementation can be explored. These opportunities include usability testing and the investigation of methods that more comprehensively capture experiences, priorities, and factors associated with social determinants of health. These measures could be a useful tool in identifying and addressing social determinants of health to help reduce disparities in healthcare delivery, access, and health outcomes. Furthermore, ePROs/ePROMs can help us reduce key missing data that may supplement our understanding of a disease or condition from a patient’s perspective throughout their healthcare journey. Data from a reliable and valid ePROM for a clinical trial may be used to inform and support clinical practice, payer decisions, drug approvals, and policy decisions.

To fully harness the potential of ePROs/ePROMs, future research is needed to better understand health equity-related data needs as well as appropriate instrument design and implementation strategies for these tools in both clinical research and practice settings.

 

References

1. Lavallee DC, Chenok KE, Love RM, et al. Incorporating patient-reported outcomes into health care to engage patients and enhance care. Health Aff (Millwood). 2016;35(4):575-582. doi:10.1377/hlthaff.2015.1362

2. Maruszczyk K, Aiyegbusi OL, Torlinska B, Collis P, Keeley T, Calvert MJ. Systematic review of guidance for the collection and use of patient-reported outcomes in real-world evidence generation to support regulation,
reimbursement and health policy. J Patient-Rep Outcomes. 2022;6(1):57. doi:10.1186/s41687-022-00466-7

3. Austin E, LeRouge C, Hartzler AL, Chung AE, Segal C, Lavallee DC. Opportunities and challenges to advance the use of electronic patient-reported outcomes in clinical care: a report from AMIA workshop proceedings. JAMIA Open. 2019;2(4):407-410. doi:10.1093/jamiaopen/ooz042

4. Damberg CL, Sorbero ME, Lovejoy SL, Martsolf GR, Raaen L, Mandel D. Measuring success in health care value-based purchasing programs: findings from an environmental scan, literature review, and expert panel discussions. Rand Health Q. 2014;4(3):9.

5. Food and Drug Administration, US Department of Health and Human Services. Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims.2009.

6. Centers for Medicare & Medicaid Services. Supplemental Material to the CMS MMS Blueprint: Patient-Reported Outcome Measures. Published online September 2021. Accessed May 4, 2022. https://www.cms.gov/files/document/blueprint-patient-reported-outcome-measures.pdf

7. Centers for Disease Control and Prevention. Social Determinants of Health: Know What Affects Health. Published September 30, 2021. Accessed January 20, 2023. https://www.cdc.gov/socialdeterminants/index.htm

8. Zbrozek A, Hebert J, Gogates G, et al. Validation of electronic systems to collect patient-reported outcome (PRO) Data—recommendations for clinical trial teams: report of the ISPOR ePRO Systems Validation Good Research Practices Task Force. Value Health. 2013;16(4):480-489. doi:10.1016/j.jval.2013.04.002

9. Coons SJ, Eremenco S, Lundy JJ, O’Donohoe P, O’Gorman H, Malizia W. Capturing patient-reported outcome (PRO) data electronically: the past, present, and promise of ePRO measurement in clinical trials. Patient. 2015;8(4):301-309. doi:10.1007/s40271-014-0090-z

10. Working Group for Monitoring Action on the Social Determinants of Health. Towards a global monitoring system for implementing the Rio Political Declaration on Social Determinants of Health: developing a core set of indicators for government action on the social determinants of health to improve health equity. Int J Equity Health. 2018;17(1):136. doi:10.1186/s12939-018-0836-7

11. Patt D, Wilfong L, Hudson KE, et al. Implementation of electronic patient-reported outcomes for symptom monitoring in a large multisite community oncology practice: dancing the Texas two-step through a pandemic. JCO Clin Cancer Inform. 2021;(5):615-621. doi:10.1200/CCI.21.00063

12. Commission on Social Determinants of Health. Closing the Gap in a Generation: Health Equity Through Action on the Social Determinants of Health. Final Report of the Commission on Social Determinants of Health. World Health Organization; 2008:1-256. Accessed January 20, 2023. https://apps.who.int/iris/bitstream/handle/10665/43943/9789241563703_eng.pdf;jsessionid=365271ACE2052888542881700EEDCA8B?sequence=1

13. Burström B, Tao W. Social determinants of health and inequalities in COVID-19. Eur J Public Health. 2020;30(4):617-618. doi:10.1093/eurpub/ckaa095

14. Palmer RC, Ismond D, Rodriquez EJ, Kaufman JS. Social determinants of health: future directions for health disparities research. Am J Public Health. 2019;109(S1):S70-S71. doi:10.2105/AJPH.2019.304964

15. Mercieca-Bebber R, King MT, Calvert MJ, Stockler MR, Friedlander M. The importance of patient-reported outcomes in clinical trials and strategies for future optimization. Patient Relat Outcome Meas. 2018;9:353-367. doi:10.2147/PROM.S156279

16. Aiyegbusi OL. Key methodological considerations for usability testing of electronic patient-reported outcome systems. Qual Life Res. 2020;29(2):325-333. doi:10.1007/s11136-019-02329-z

17. Sandhu S, King Z, Wong M, et al. Implementation of electronic patient-reported outcomes in routine cancer care at an academic center: identifying opportunities and challenges. JCO Oncol Pract. 2020;16(11):e1255-e1263. doi:10.1200/OP.20.00357

18. Block RG, Puro J, Cottrell E, et al. Recommendations for improving national clinical datasets for health equity research. J Am Med Inform Assoc. 2020;27(11):1802-1807. doi:10.1093/jamia/ocaa144

19. Meirte J, Hellemans N, Anthonissen M, et al. Benefits and disadvantages of electronic patient-reported outcome measures: systematic review. JMIR Perioper Med. 2020;3(1):e15588. doi:10.2196/15588

20. Benjamin K, Vernon MK, Patrick DL, Perfetto E, Nestler-Parr S, Burke L. Patient-reported outcome and observer-reported outcome assessment in rare disease clinical trials: an ISPOR COA Emerging Good Practices Task Force Report. Value Health. 2017;20(7):838-855. doi:10.1016/j.jval.2017.05.015

21. Deshpande P, Sudeepthi Bl, Rajan S, Abdul Nazir C. Patient-reported outcomes: a new era in clinical research. Perspect Clin Res. 2011;2(4):137. doi:10.4103/2229-3485.86879

22. Calvert MJ, Cruz Rivera S, Retzer A, et al. Patient reported outcome assessment must be inclusive and equitable. Nat Med. 2022;28(6):1120-1124. doi:10.1038/s41591-022-01781-8

23. Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21(8):1305-1314. doi:10.1007/s11136-011-0054-x

24. Basch E, Charlot M, Dueck AC. Population-level evidence of survival benefits of patient-reported outcome symptom monitoring software systems in routine cancer care. Cancer Med. 2020;9(21):7797-7799. doi:10.1002/cam4.3480

25. Cheung YT, Chan A, Charalambous A, et al. The use of patient-reported outcomes in routine cancer care: preliminary insights from a multinational scoping survey of oncology practitioners. Support Care Cancer. 2022;30(2):1427-1439. doi:10.1007/s00520-021-06545-7

26. Andermann A. Taking action on the social determinants of health in clinical practice: a framework for health professionals. Can Med Assoc J. 2016;188(17-18):E474-E483. doi:10.1503/cmaj.160177

27. Lacklear T, Weinfurt KP, Abernethy A, Flynn KE, Riley W, Johnson LL. Patient-reported outcomes. Published online January 2014. https://dcricollab.dcri.duke.edu/sites/NIHKR/KR/PRO%20Resource%20Chapter.pdf

28. Hernández-Cancio S, Sanchez D, Fishman E, Albritton E. The Role of Patient-Centered Outcomes Research in Improving Evidence and Advancing Health Equity. Families USA; 2018:24. https://familiesusa.org/wp-content/uploads/2019/10/HEV_Patient-Centered-Outcomes_Report.pdf

 

Disclaimer:
Sarach Stothers Rosenberg is an employee at the US. Food and Drug Administration (FDA) and was a fellow during the development of this article. This article reflects the views of the author and should not be construed to represent FDA’s views or policies. Ting-Hsuan Lee was a fellow at the FDA during the development of this article, and as such her contributions are an informal communication and represent her best judgement. These comments do not bind or obligate FDA.

 Acknowledgement:
The authors thank Julia F. Slejko, Kathleen Wyrwich, Newell E. McElwee, Ashley Martin, Tan Nguyen, Wai Man Maria Chan, Rinat Ariely, Saara Nasruddin, Caroline Jacobsen, Huda Eldosougi, Ramiro Gilardino, Rebecca Raciborski, Karolina Baloghova, Manan Shah, and Soham Shukla for their contributions to and review of this article.