OBJECTIVES: The RAND/UCLA modified Delphi panel method is a formal group process that systematically and quantitatively combines expert opinion and evidence to arrive at consensus, which traditionally includes an in-person meeting. Experts (physicians, advocates) meet in-person to discuss results of a first-round survey before repeating the survey. The COVID-19 pandemic made in-person meetings impossible. In this study, we examine the impact on achieving consensus when moving from in-person to virtual panel meetings.

METHODS: We conducted 5 virtual panels over 13 months and compared them to 4 pre-pandemic in-person panels. We report the number of panelists, meeting duration, items rated, and percent disagreement in first- and second-round surveys.

RESULTS: Both the in-person and virtual panels included a mean of 11 panelists. Panelists joined virtual meetings for 6-7 hours across 2-4 hour sessions. In-person meetings lasted 6-9 hours plus up to 10 hours of travel. Panelists rated a mean of 488 and 453 items in the virtual and in-person panels, respectively. Disagreement was higher in first-round surveys (range 13-67% virtual, 34-67% in-person) than in second-round surveys (range 1-32% virtual, 10-43% in-person). Mean decreases in disagreement were 19% (virtual) and 27% (in-person).

CONCLUSION: We maintained certain aspects of the panel method (e.g., review of existing evidence, number of panelists, number of survey items) and found similar decreases in disagreement between first- and second-round surveys. We engaged a diverse group of experts, including those with busy clinic schedules who may not have traveled to an in-person meeting. While we completed panel discussions in less time virtually, we were unable to recreate the social interactions that built rapport among panelists during in-person meetings. Transitioning from in-person to virtual meetings was not without challenges, but there were also unexpected advantages. This virtual Delphi panel method can be an effective and efficient alternative for researchers and clinicians.