OBJECTIVES: To determine key themes and issues identified by Evidence Review Groups (ERG) and the NICE committee during the NICE Highly Specialised Technology (HST) Appraisal process. Also, to explore the relationship between issues that limit the ability of the NICE committee to approve a product, as well as looking at ways that companies mitigate uncertainty in their appraisals.

METHODS: All products that followed the NICE HST process (to December 2021) were identified and analysed. In addition, an analysis of committee papers and subsequent publications was carried out, along with a targeted literature review of associated publications.

RESULTS: Of the 16 products that have followed the NICE HST process, the most common major criticism (87.5% of products) from the Evidence Review Group was that resource utilisation estimates were inaccurate, or that the methodology used was not sufficiently robust. Other criticisms included utility modelling not being robust enough (68.75%), utility estimation by clinicians (56.25%), clinician estimates of efficacy (43.75%), model approaches not being sufficient for decision making (31.25%), and trial endpoint robustness (25%). This led to 93.75% of cases that resulted in a positive recommendation having managed access agreements and confidential discounts applied to them. Major criticisms of submissions tend to centre around the lack of a robust methodology for derivation of estimates (resource utilisation and utility values) from clinicians.

CONCLUSIONS: Given that products that qualify for a NICE HST process tend to be in a rare disease area, there is typically a paucity of data. This usually leads to manufacturers turning to clinicians to seek estimates – it is crucial here to have a recognised, robust methodological process to elicit and validate estimates. Further review of NICE publications suggests that Modified Delphi and vignette studies may be most appropriate if carried out in a robust and meaningful way; validation across multiple stakeholders adds extra validity.