The US Comparative Effectiveness Research Movement: Further Insights from the Frontlines of the Debate

The comparative effectiveness research (CER) movement burst on the scene in the US with the passage of the American Recovery & Reinvestment Act of 2009, which allocated $1.1 billion in funding for this new type of research.

This legislation initiated a great deal of activity on the part of researchers, policymakers, health insurers, life sciences companies, and other stakeholders interested in providing definition to CER and influencing the trajectory of the movement.

Three years ago, in the March/April 2009 issue of ISPOR CONNECTIONS, we published an interview conducted with three thought leaders who are playing an influential role in the CER movement in the United States: Peter J. Neumann, ScD of Tufts Medical Center in Boston, Massachusetts; Sean D. Sullivan, PhD of the University of Washington School of Pharmacy in Seattle, Washington; and Michael Drummond, DPhil, of the University of York, in the UK. We elicited their insights on the debate and asked them to speculate on how things might unfold as incoming President Barack Obama took office.

A lot has transpired since then and we thought it timely to reengage with our experts to get an update on the CER movement. The following is an additional interview that took place in the past three months.

Thompson: When we last discussed these issues in 2008-9, the concept of comparative effectiveness research had only just been introduced in the American Recovery & Reinvestment Act (ARRA) legislation, Mr. Obama had just been inaugurated as President of the United States, the Democratic Party controlled both houses of Congress

in the US, and .... need I go on? Certainly, few could have predicted the political turmoil we now face and the gridlock we currently see in Washington, DC. It’s inevitable that we’ll touch on these issues over the course of our conversations, but for now let’s stick to the basics. What do we now know about the comparative effectiveness movement that we didn’t know then?

Neumann: Well, of course, since we last discussed these issues, we’ve had the passage of major health reform legislation in the US that included a new Patient-Centered Outcomes Research Institute (PCORI) to conduct comparative effectiveness research. So, despite some strong opposition to the concept, we legislated CER into existence in a permanent way (unless Congress decides to change that in the future).

The Patient Protection and Affordable Care Act (ACA) also reveals something about how we will conduct CER in the US. One important aspect is the idea that PCORI, as evident by its name, will focus on “patient-centered research.” The name itself is telling. No one will oppose “patient-centeredness.” Calling the institute PCORI, as opposed to the CER Institute, probably helped shore up support for the legislation–or at least diluted some of the potential opposition. It addressed fears that some critics had about CER inappropriately imposing population-wide averages on individuals and concerns about the government getting between doctors and patients. The ACA also prohibits PCORI from developing or using cost per QALY thresholds, which responds to a concern that CER will lead to cost containment and “rationing.”

We now have PCORI established with a Board of Governors and a Methods Committee, and we have the first inklings about how they will conduct their work. The Institute seems to be focusing on developing and testing methods, and not necessarily primary research, at least at the outset. They have announced broad subject areas, such as informing research priorities; bringing

together different stakeholders; translating research into practice; identifying gaps in relation to disadvantaged populations; and identifying predictors of patient outcomes. What we see is PCORI taking its first tentative and very cautious baby steps. PCORI has begun hiring staff and initiating a process for soliciting grants, and we’ll know more as time goes on.

Drummond: From the outside, it is good to see PCORI being established. I think close observers of the situation in the US recognize that the compromises Peter outlines were necessary in order to get any initiative in CER off the ground. However, I wonder whether PCORI, with the restrictions that have been imposed, will be able to undertake studies that help us tackle some of the problems facing the US health care system. I guess the hope is that, if more evidence on the comparative effectiveness of health care interventions becomes available, private health plans may be able to use it in determining their benefit package. Sean, do you think this is a possibility?

Sullivan: PCORI will be financially supported by a mix of public and private funds. Initially, a modest amount of funding to initiate PCORI activities has come as an appropriation from the federal government. As Peter indicated, PCORI has used these funds to establish the Board of Governors, appoint and convene the CER Methods Committee and hire a small program staff. But the base of funding for the main PCORI CER activities expands in the coming years and is not primarily from federal sources. For private health plans, this includes a $2 tax per covered member per year, starting in 2014. Because the private plans will be paying for PCORI activities, perhaps this creates an incentive for them to care about and maybe even use the CER information.

Framers of the ACA understood that information from CER studies would be valuable to a wide variety of stakeholders, including private health plans. However, as Mike points out, there are no explicit requirements in the law that make use of CER evidence compulsory by either public or private health care funders. In fact, some argue that language in the ACA actually restrict use of certain types of CER information by public payers. How this will be overseen and regulated is a mystery. What we do know is that some of the larger private health plans are not waiting for PCORI to initiate activities. Wellpoint, for example, has signaled its interest in CER by disseminating guidance on how health technology manufacturers should prepare and submit comparative evidence to support coverage and reimbursement decisions and continued listing. The Academy of Managed Care Pharmacy (AMCP) has just initiated a revision (4th) to the Format for Formulary Submission guidelines that will very likely include explicit language on CER.

Thompson: This is quite interesting. Certainly the introduction of PCORI constitutes a major step forward for CER in the US, but I’m sure many of our international readers will question how effective it can be with cost-effectiveness considerations mandated out of the picture. Mike, perhaps it would be useful for you to compare and contrast what you know of PCORI to comparable institutions in other countries, such as NICE in the UK, PBAC in Australia, and ANVISA in Brazil? On a high level, how does PCORI compare? Is there another country that is more similar to the US than these others?

that this is not a function PCORI will perform. Rather, it will undertake or commission studies that we hope will be useful to various parties within the health care system. There are agencies in other countries that have a similar role, such as the Swedish Council for Health Technology Assessment (SBU) or the German Institute for Medical Documentation and Information (DIMDI). Some of these agencies have been in existence much longer than the likes of NICE and have a track record of publishing a wide range of reports—some that include consideration of costs, some that don’t. In general, these agencies are well respected in their own countries and internationally. In addition, their impartiality and independence is valued. However, because the studies they conduct are not always connected with a given decision or decision-maker, questions often arise concerning whether their reports have much impact on health care delivery. This suggests that, in the US, attention will need to be paid to how the results of CER studies are communicated in a way that meets the needs of the various decision makers.

Thompson: Sean and Peter, what do we know at present regarding the direction that PCORI will take with research funding? Are there specific types of studies they will favor? Specific disease areas? Target populations?

Sullivan: PCORI are evolving their priorities with respect to methods and research. Recently, PCORI issued its initial set of research funding requests, focusing on several methodology topics. The nature of the PCORI research priorities signal an interest in improving methods related to stakeholder engagement, value of information to inform research prioritization and patient-centered outcomes. PCORI received well in excess of 1,300 Letters of Intent. This response indicates substantial interest from the CER community in PCORI research priorities. Of note, PCORI is forming its own peer review panel to evaluate applications. We should know more by mid-2012 as to what research will be funded by PCORI.

Neumann: PCORI’s priorities will become clearer over time. Out of the gate, its leaders seem to be moving very cautiously, which is undoubtedly a smart thing to do from a political standpoint. A mis-step this early in their tenure–say, for example, a study that came down on one side of a sensitive debate such as appropriate policy for mammography screening–could be disastrous for the Institute in the polarized political environment in which we live (made worse in an election year). The rhetoric out of PCORI is largely about methods, the need to engage stakeholders, and the need to understand patient preferences. These are important topics, but notably, not terribly controversial ones.

Thompson: You mention stakeholders, so let’s turn our attention to that topic. What is the anticipated impact (current and anticipated) of CER on various stakeholders in the health care system, including government agencies (eg, AHRQ, NIH, CMS), third-party payers (eg, HMOs), and life sciences companies (eg, pharmaceutical, biotech, and medical device manufacturers). What are you seeing there?

Drummond: Since CER is still in the early stages of development, I think that there is everything to play for in terms of stakeholder involvement. In addition to those groups you identify, I would also include professional societies and patient organizations as important stakeholders in these activities. Work that Peter, Sean and I have done with colleagues, examining existing organizations in the CER/HTA space, indicates that the extent of the involvement of stakeholders is quite variable (Neumann et al. Int J Tech Assess Healthcare 2010;26:71-78). Most organizations ask key stakeholders to comment on draft reports and, in the case of manufacturers, make submissions of clinical and economic data. However, some organizations go further and invite the participation of key stakeholders in the scoping of projects and also include them as members of expert committees. My sense is that, in the case of CER, adequate stakeholder input is likely to be critical to the success of the enterprise, given the wide range of actors in the US health care system. The early signs are that PCORI intends to embrace the concept of extensive stakeholder involvement. But perhaps you folks from the US have a different take on this.

Sullivan: I think Mike is correct. As I mentioned earlier, the initial PCORI requests for applications (RFAs) included several that focus on developing appropriate methods to engage and utilize stakeholders to identify evidence gaps, prioritize research, and implement CER findings. As Mike points out, the US has many stakeholders, each with its own view on how health care should be financed, organized, delivered and reimbursed. What role each of the actors should play in CER is of tremendous interest to PCORI, but effectively incorporating the many viewpoints will be challenging.

Neumann: I agree that adequate stakeholder input is likely to be critical to the success of the enterprise. From a political standpoint, it is hard to be against “stakeholder involvement” and it makes sense for PCORI to emphasize it. The downside is that too much stakeholder involvement can slow the research— and ultimately the decision-making process. Furthermore, there can be a delicate balance between appropriate stakeholder engagement and the undue influence of “interest groups.” I worry that in our rush to embrace stakeholders we haven’t given enough consideration to these potential drawbacks.

Thompson: So I’d like to conclude the discussion by shifting to issues that I think are crucial to our Society. I must admit I’ve been skeptical of CER ever since I read the key passages from the ARRA legislation and began our initial dialog on this issue—in fact, I came away thinking that things could be summed up by the simple equation, CER = CEA – C, meaning that comparative effectiveness is what you get when you remove cost considerations from cost-effectiveness analysis. To be honest, I was disappointed and felt that CER was just a new term for evidence-based medicine—the proverbial “old wine in new bottles” phenomenon—and I worried that we were taking a step backwards in our efforts to bring value considerations into health care decision making. Do you consider this a fair assessment? Before you answer, let me continue. In October 2011, I attended a summit meeting on CER that featured a number of impressive speakers involved in the comparative-effectiveness movement, including the executive director of PCORI. Costs were like the unacknowledged elephant in the room, seldom mentioned and never asserted to be critical to the decision-making process. The health economist in me wondered: What has happened to our science? Why have we stood by while congress has legislated cost-effectiveness analysis out of the picture? Sure, political compromise is necessary, but where do we draw the line? Talk me off the ledge here, guys!

Drummond: Dave, if your concern is for the efficiency of the US health care system, I’m tempted to say ‘jump now’! I think all those outside the US appreciate the political sensitivities of including consideration of cost, but we’re puzzled as to why you are going to all this trouble, with the investments in research that will follow, if there is no intention to tackle the main problem. In most discussions of the US health care system, cost seems to be a big part of the problem, but it is never part of the solution.

I think the original intent was that more authoritative data on the comparative effectiveness of alternative therapies would enable us to eliminate waste and to prioritize funding therapies for which there was good evidence of effectiveness. However, my sense is that the emphasis changed. It now seems to be accepted that CER will have a strong patient focus and certainly PCORI seems to be taking its name very seriously.

In addition, it is being pointed out by a number of commentators that, although RCTs are a good way of assessing the average clinical effect of an intervention, heterogeneity in the patient population could mean that the most effective, or cost-effective, therapy overall may not be the best one for some patients. Although this has been known for some time, the question is what do we do with this information. Many of the European HTA agencies, such as NICE, recognize patient heterogeneity in so far as they may make recommendations for defined sub-groups of the patient population, based on baseline risk for example. However, in the US, it seems that the implication is that we should provide the best possible care for each individual patient. Moreover, we should take account of heterogeneity not only in factors such as baseline risk, but also heterogeneity in patient preferences.

Looking at the current CER agenda, especially from PCORI, an alien from outer space would be forgiven for thinking that the main problems facing the US health care system are that it offers poor quality of care and does not consider patients’ views enough, not that the system was facing escalating costs and possibly wasting resources. Perhaps all these problems exist to some extent, but it seems crazy to ignore the cost issues completely.

However, if I were trying to talk you down off the ledge, I’d be saying that we are still in the early days of CER and many parties will be conducting studies. Perhaps in the longer term we will see more studies addressing resource utilization, if not cost per se, and more use of studies by payers making value-based coverage decisions. I always take comfort in the quote from Winston Churchill, who said that ‘You can always rely on the Americans to do the right thing, but only after all other possibilities have been exhausted’.

Neumann: Dave, get off the ledge! You and all of us at ISPOR can take comfort in the fact that even the US Congress can’t defy gravity and can’t outlaw resource constraints and tradeoffs in our health decisions (even though our legislative language from the ACA and PCORI’s mandate may suggest that we can). Of course we need a more efficient health care system. Decision makers will continue to strive to obtain better value for their spending. Our research can help in those efforts. However, the words we use, the institutions we set up, and the rules under which we operate are important. We will not create a “NICE” in the US. We won’t use cost-per-QALY ratios in an explicit way. Medicare will continue to assert that it does not use cost-effectiveness analysis in its coverage decisions. The discussions of CER will sidestep simple words like “costs.” But information about the clinical and economic consequences of health interventions will continue to inform coverage and reimbursement decisions, though likely in an indirect manner in the US. I think our future as a Society is actually quite bright. However, in the US, we’ll develop our own rules and even language for engaging CER and CEA. You can study the impact of alternative strategies on patient adherence or readmissions or ICU days, for example, but you can’t do a cost-per-QALY analysis.

Thompson: My final question brings ISPOR into the spotlight. I’ve observed that our Society’s US-based meetings have paid a great deal of attention to CER in the past few years, witih a multitude of plenaries, issue panels, workshops, and other submissions devoted to this topic. But aside from a few familiar faces, the most prominent players at our ISPOR meetings—and, indeed, ISPOR itself— had virtually no representation at the CER Summit mentioned above. So what do you think? As a Society, are we doing enough to advocate for the science of cost-effectiveness analysis and its place in health care decision making? Should ISPOR be more active in the CER movement in hopes of influencing its trajectory or is that game already over? Give us all you got here, gentlemen. What role should ISPOR be playing in the CER movement?

Neumann: ISPOR members are certainly speaking and attending key meetings, conducting studies, responding to PCORI requests for proposals, and serving in public and private leadership roles. The question of what more the Society itself can do is an important one. I think we need to advocate for good methods for conducting analyses and good practices for communicating them. ISPOR has done a reasonably good job at developing and disseminating reports on good research practices for the field—indeed, I can’t think of another society that has done as much. In addition, ISPOR should provide a neutral forum for debates on key topics. Here, too, the Society has done pretty well, I’d argue. We can always do more. ISPOR should try to “be at the table” when opportunities arise and, when possible, help shape the debate. That will be a harder task, but could involve responding to open calls for public comments, and more aggressively promoting our good research practice documents.

Drummond: I agree with all the points Peter makes above. The only point I would add is that ISPOR is in a good position to orchestrate international debates on CER, HTA, and related topics. Not only does the Society have a broad international membership, it also has membership from a broad range of public and private constituencies. The latter is likely to give ISPOR a broad range of perspectives on CER. This is important because, unlike HTA, CER is not going to be the main preserve of government.

Sullivan: It’s hard to argue with Peter and Mike, but for the sake of ending this discussion with a bit of controversy, I’ll take a different view. Both Peter and Mike make good points about the activities of ISPOR members. But, as you put the question to us, “Is ISPOR doing enough?” I wonder if there may be more that can be done. Peter suggests that ISPOR has done more than other organizations. This may be true, but I would like to suggest additional activities that ISPOR might consider given its considerable infrastructure, Special Interest Groups, committees and vehicles for dissemination. For example, the ISPOR Board of Directors could charge its Health Policy Council to weigh in on the research and methods priorities and implications for CER in the US and Relative Effectiveness (RE) in Europe. The RE debate in Europe is taking a decidedly different form than the CER discussions in the US. How will the life sciences industry be able to respond to what may be highly divergent post-licensing requirements that emerge from these debates? The editors of Value in Health could commission and publish commentaries and papers onCER methods and implications for various stakeholders. See, for example the October 2010 issue of Health Affairs and the constant policy dialogue on CER in the New England Journal of Medicine. ISPOR has, in the past, held focused symposia on topics of interest to the Society—recall the special meeting about the QALY a few years back. Should ISPOR convene a meeting on RE and CER or co-sponsor a meeting on methods with another society? More challenging for ISPOR, however, will be to frame and make public comments on matters of policy without an established mechanism to engage and seek consensus from its membership. Given the importance of RE and CER to all ISPOR members, it seems that ISPOR could and should do more.

Thompson: Excellent ideas. This has been an informative update. Many thanks to all three of you.

  Issues Index | 2012 Issues Index