OBJECTIVES: Different statistical approaches are available for conducting an indirect treatment comparison (ITC) when studies differ in effect modifier distributions. Matched Adjusted Indirect Comparison (MAIC) uses propensity score weighting to match populations and is the most prevalent approach. However, there are circumstances when its use is limited, such as when overlap is poor between covariate distributions in included studies. In such cases, Simulated Treatment Comparison (STC) and Multi-Level Network Meta-Regression (ML-NMR) may be preferred, as they use covariate adjustment to balance effect modifiers between studies. Our objectives were to replicate results obtained by each covariate adjustment method and to explore their relative ease of implementation.

METHODS: We performed STC (using G-computation) and ML-NMR (using multinma R package) on a simple ITC of two psoriasis studies to compare Ixekuzumab versus Secukinumab, attempting to replicate approaches as closely as possible.

RESULTS: We found numerous analytic choices involved in performing the analyses, which prevented us from reproducing an identical result using both methods. Estimated ORs from STC and ML-NMR for Ixekizumab versus Secukinumab were 1.31 (0.54-3.23) and 1.26 (0.54-2.83) respectively (comparator trial population). STC involves several steps performed at the discretion of the analyst that can impact estimates, and uncertainty is greater. For ML-NMR, multinma facilitates consistency when using the default options.

CONCLUSIONS: Whilst both STC and ML-NMR are valuable tools for population adjustment in ITC numerous analyst degrees-of-freedom can impact results. Reproducibility and transparency is potentially a limitation since analysts rarely report steps in detail and precise guidance is limited. A dedicated R package for ML-NMR provides consistency, though users need to be fully aware of what the default options imply for the model. More studies are needed to investigate implications of these choices and whether they introduce bias in practice, as well as to clarify details that must be reported to make analyses reproducible.