A Comparative Study of Alternative Software to Conduct Hazard Ratio-Based Network Meta-Analysis

OBJECTIVES: With the growing need for comparative evidence to inform health technology assessment in the absence of head-to-head trials, it is important to investigate efficient solutions. This research seeks to compare alternative software used to conduct hazard ratio-based network meta-analysis (NMA), by replicating a published analysis in three different software (WinBUGS, Stan, JAGS).

METHODS: A published Bayesian NMA of survival endpoints combining both count and hazard ratio statistics on the hazard ratio scale was replicated to inform the comparison of hazard ratio results and 95% credible intervals (CrIs). The network of evidence for this analysis included three trials and three treatment regimens. Efficiency was measured quantitatively through total sampling run-time and qualitatively through software ease-of-use.

RESULTS: Difference in median hazard ratio ranged from 0.004 to 0.006, showing consistent findings across all software. The CrIs were also in alignment, showing very minimal differences. The median hazard ratio estimates from Stan and JAGS were generally lower than WinBUGS, this may be attributable to the different sampling algorithms employed in these software. The overall interpretation of the results did not change across. WinBUGS and JAGS were superior to Stan with regards to computation time with a mean run time of 8.04 seconds. Stan offered the best user experience for programming and debugging the model, particularly due to its superior integration within the R Studio integrated development environment. Sensitivity analysis tested fixed effects, random effects and a different size networks.

CONCLUSIONS: Using alternative software resulted in comparable results and interpretations in this comparative study. Both Stan and JAGS provided significant improvements in terms of software ease-of-use. WinBUGS and JAGS provided faster model run time for this case-study. The efficient running of analyses is crucial for meeting the growing need for rapid comparative evidence, thereby aiding the validation, reproducibility, and communication of results.