OBJECTIVES: NMAs and other anchored ITCs leverage randomization via comparison of relative effects compared to placebo. In most cases, trials included in ITCs are powered to detect large differences compared to placebo. Differences between any two active therapy are expected to be smaller, and thus this combined with the increased variance of treatment differences estimated via an anchor may mean that ITCs are drastically under-powered. We use a recent Cochrane NMA in Psoriasis to explore the power of ITCs compared to the same comparisons in subsequent head to head trials.

METHODS: We use extracted pairwise comparisons and confidence intervals from a recent Cochran review in Psoriasis and compare the estimate of the standard error of a given pairwise comparison to the hypothesized effect size used for planning subsequent head to head studies. We use this data to conduct a design analysis to assess the power of comparison from the NMA, and describe how much statistically significant results can be expected to have exaggerated magnitude (type-M error) or increased probability of being significant in the wrong direction (type-S error).

RESULTS: ITCs from NMAs are extremely under-powered compared to the hypothesized effect size in subsequent head to head randomized controlled trials. The result is that effect estimates from statistically significant comparisons in NMAs excluding the head to head trial can be inflated by as much as five times, and results that are statistically significant have a greater than 5% chance of being in the wrong direction.

CONCLUSIONS: ITCs are likely extremely under powered which increases the risk that filtering results (by emphasizing those that are significant) could lead to important public health consequences via upwardly biased effect estimates that have higher than nominal probability of being in the wrong direction.