Navigating the Digital Mental Health Landscape: One Mind PsyberGuide

Stephen M. Schueller, University of California Irvine, Irvine, CA, USA; Executive Director, One Mind PsyberGuide, Napa, CA, USA

Digital health is a rapidly growing area. As it stands, over 350,000 health apps exist and over 90,000 of those apps were added in 2020 alone.¹ Mental health is an important and growing segment of this market. The COVID-19 pandemic has resulted in both an increased need for scalable mental health solutions and an increased interest in digital solutions. Venture capital investments in mental health start-ups rose 72.6% from Q1 2020 to Q1 2021, totaling $2.4 billion in 2020 and accounting for 19% of all digital health funding.² Despite there being over 10,000 mental health apps, little guidance is available to help consumers. In the United States, very few products have pursued and received clearance from the US Food and Drug Administration (FDA). In fact, the FDA relaxed regulations in light of the COVID-19 pandemic, which allows for products to market themselves to consumers without FDA approval or clearance.³ As such, we need frameworks and resources to support the evaluation of digital mental health solutions and to provide this information to consumers in useful ways.

A consumer might be okay with their data being shared with a third party as long as they find value in the product, whereas a practitioner might be unwilling to recommend a product that shares any information with outside parties.

One Mind PsyberGuide is a nonprofit project that aims to help people to use technology to live mentally healthier lives. Started in 2013, One Mind PsyberGuide conducts evaluations of digital mental health products, especially mental health apps. To date, One Mind PsyberGuide has reviewed over 460 digital mental health products. The major outputs of One Mind PsyberGuide are product reviews comprising evaluations of 3 metrics: (1) credibility, (2) user experience, and (3) transparency around data security and privacy practices, and also provides professional reviews that give additional background and describe the use case for the product. (Figure).

Figure. Evaluation metrics used to look at mental health apps.

One Mind PsyberGuide uses these 3 metrics rather than an overall total score such as a percentage or a 5-star rating, because we believe there is no “magic number” for digital mental health products. Different things will work for different people and similarly, people might weight or consider factors differently. For example, a tech-savvy 20-something might be willing to tolerate a product with a more difficult learning curve when it comes to user experience as long as it is effective. A consumer might be okay with their data being shared with a third party as long as they find value in the product, whereas a practitioner might be unwilling to recommend a product that shares any information with outside parties. Providing these metrics separately allows consumers to make individual decisions based on their needs and resources. As such, each evaluation metric is intended to be able to answer a question a consumer might have when considering using a specific digital mental health product:

• Credibility: How likely is it that this app can do what it says it can?

• User experience: How likely is it that I will actually use this app?

• Transparency: What happens to the data I enter into this app?    

We describe each of these metrics below in more detail and provide some lessons learned from our 8 years of reviewing products in the digital mental health space.

One Mind PsyberGuide Evaluation Metrics

Credibility: How likely is it that this app can do what it says it can?
Digital mental health products make a range of claims such as treating depression, boosting well-being, increasing focus, or decreasing stress. The vast majority of these claims are unsupported. Only about 3%-5% of digital mental health products are evidence-based.⁴ Even those that are evidence-based have rarely been rigorously evaluated through methods such as randomized controlled trials. Instead, most rely on indirect research evidence, that is, they digitize a previously validated evidence-based intervention or practice.

 

The highest scores are provided to products with at least 2 rigorous research evaluations such as a randomized controlled trial with findings that have been published with peer review and involved funding or collaboration from nonprofit sources or outside investigators.

The One Mind PsyberGuide credibility metric is intended to quantify the support for an app including the direct and indirect research evidence, as well as other aspects of the development team and processes. The credibility metric has 4 components: (1) research, (2) development, (3) intervention specificity, and (4) consumer ratings. The research subscore evaluates the amount of research evidence for a product as well as the degree of independence and review of data collected in that research. The highest scores are provided to products with at least 2 rigorous research evaluations such as a randomized controlled trial with findings that have been published with peer review, and involved funding or collaboration from nonprofit sources or outside investigators. The development subscore evaluates inclusion of developmental processes incorporating stakeholder engagement and early feedback such as design, feasibility, and acceptability data, the inclusion of clinical input on development, credibility of the development team gained from past evidence-based products, and ongoing maintenance of the product. The intervention specificity subscale looks at the clarity of the proposed goal of the intervention and whether it is clear, measurable, and specific. Lastly, the consumer ratings subscale considers the number and average value of app store reviews.

User Experience: How likely is it that I will actually use this app?
A major challenge facing digital mental health products is lack of engagement and long-term sustainment. Most apps that are downloaded are never opened and digital mental health is no exception.⁵ The user experience and usability of an app impacts which apps people start to use and differentiates those that would be used long-term and those that would be quickly discarded. One Mind PsyberGuide uses the Mobile App Rating Scale created by Stonyanov and colleagues⁶ to evaluate an app’s user experience and usability.

The Mobile App Rating Scale evaluates 4 components of user experience: (1) engagement, (2) functionality, (3) aesthetics, and (4) information quality. Each component is evaluated using 3 to 8 questions each rated on a 5-point scale with well-defined anchors. These scores are averaged, producing scores that range from 1.0 to 5.0. In general, a 3.0 on the Mobile App Rating Scale is considered average, and a 4.0 is considered good. Of all the products currently reviewed on One Mind PsyberGuide about 41% have received Mobile App Rating Scales greater than 4.0. Although the products reviewed might be selected in ways that mean this is not fully representative of the market at large.

Transparency: What happens to the data I enter into this app?
People want to know that their private data are secure and protected when they are interacting with mobile apps. This is especially important when dealing with sensitive information related to mental health. One Mind PsyberGuide conducts an evaluation of data security and privacy policies using our transparency metric. We refer to it as a transparency metric, rather than data security and privacy itself, because we review the policies provided by developers, rather than a technical audit of the app. This is useful to consumers as privacy policies are rarely consumer-friendly. They are long, contain technical jargon, and are written at high literacy levels. Furthermore, consumers might not appreciate all the information they should consider when reviewing a policy and guidance conforming to standards are useful. 

Our transparency scale was developed by reviewing privacy policies of various apps and aligning these with clinical, ethical, and industry standards. We initially tested our transparency scale on a set of 116 mobile apps for depression.⁷ We found that only 49% of those apps even had privacy policies and that only 4% received an acceptable rating on our scale. Similar reviews have demonstrated the poor data security and privacy practices of many mental health apps.⁸

A major challenge facing digital mental health products is lack of engagement and long-term sustainment.

Lessons Learned From Practice
One Mind PsyberGuide has been operational for over 8 years, and in those 8 years we’ve learned several lessons relevant to the evaluation of digital mental health products and dissemination of information to consumers. People are interested in using digital tools to support their mental health and want high-quality solutions that they can trust. As such, the information provided by One Mind PsyberGuide fills an important gap in the current digital health ecosystem. The most popular content on our website, in terms of how people find our site and which pages they visit, are our professional reviews. These professional reviews are narrative reviews that describe the background, use case, and pros/cons of individual products. We have also found that our credibility metric and user experience metric are not correlated, demonstrating that many of the evidence-based products—those developed by academic teams with expertise in clinical evaluation—do not have a good user experience and that many of the most engaging products—those developed by industry team with expertise in product design—often lack clinical evidence. Increasingly, industry teams are bringing on clinical and research expertise, but their contributions to the clinical evidence base in this space is still limited. Nevertheless, there is a desire for credible information to help guide decision making in this space.

Conclusion
Digital mental health is a large and rapidly growing area. At the same time this area has advanced with little regulation or guidance, resulting in a “Wild West” for consumers and purchasers. One Mind PsyberGuide is one resource to help navigate this area, providing evaluations of credibility, user experience, and transparency around data security and
privacy.

References:

1. IQVIA Institute for Human Data Sciences. Institute Report. Digital Health Trends 2021: Innovation, Evidence, Regulation, and Adoption. https://www.iqvia.com/insights/the-iqvia-institute/reports/digital-health-trends-2021. Published July 22, 2021. Accessed July 22,2021.

2. Wang E, Zweig M. Rock Health. A Defining Moment for Digital Behavioral Health: Four Market Trends. https://rockhealth.com/reports/a-defining-moment-for-digital-behavioral-health-four-market-trends/. Published 2021. Accessed September 1, 2021.

3. US Food and Drug Administration. Guidance for Industry and Food and Drug Administration Staff. Enforcement Policy for Digital Health Devices for Treating Psychiatric Disorders During the Coronavirus Disease 2019 (COVID-19) Public Health Emergency. Published April 2020. Accessed September 1, 2021.

4. Larsen ME, Huckvale K, Nicholas J, et al. Using science to sell apps: evaluation of mental health app store quality claims. NPJ Digit Med. 2019:2(1);1-6.

5. Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res. 2019:21(9);e14567.

6. Stoyanov, SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015:3(1);e27.

7. O’Loughlin K, Neary M, Adkins EC, Schueller SM. Reviewing the data security and privacy policies of mobile apps for depression. Internet Interv. 2018:15;110-115.

8. Nguyen S, Fitzgerald B, Richter B, Brookman J. Peace of mind…evaluating the privacy practices of mental health apps. Consumer Reports. https://advocacy.consumerreports.org/wp-content/uploads/2020/12/CR_mentalhealth_full-report_VF.pdf. Updated January 8, 2021. Accessed September 8, 2021.