OBJECTIVES: To develop evidence generation strategic insights and lessons for ultra orphan drugs based on mistakes identified by NICE during Highly Specialized Technology (HST) assessments.

METHODS: A systematic review of all NICE HST HTAs was conducted. Case studies were designed to develop insights and lessons from NICE’s committee review of company submission. Assessments with mention of evidence lacking face validity were selected and reviewed in detail. The product, indication, category of evidence and specific concerns by NICE were reviewed to develop insights and lessons learned for future submissions.

RESULTS: In 5 HST assessments NICE criticized the company submission for lacking face validity. For Atidarsagene autotemcel, NICE said collection of EQ-5D scores from the general public lacked face validity because challenging health states were rated as better than less challenging health states. For Burosumab, the committee said ICERs lacked face validity due to the choice of prior distributions for transition probabilities. For Migalastat, NICE questioned and reduced the infusion disutility, which was set to be similar as a disease complication. For Patisiran, the mortality hazard ratio was not accepted because it was not in line with previous natural history study. For Voretigene neparvove, the utility values, including a negative score, from a survey of ophthalmologists was questioned because it likely focused more on vision than patient quality of life.

CONCLUSIONS: Case studies of mistakes in previous submissions provides valuable lessons for companies to design evidence generation strategies for rare disease products.