Published: June 2024

Last updated: June 2024

Development of a Tool to Assist in the Identification of Study Designs for the Purposes of HTA

INTRODUCTION: As the most internally rigorous design, randomized controlled trials (RCTs) are the gold standard for assessing the efficacy and safety profile of interventions. Increasingly, health technology assessment (HTA) considers evidence from non-randomized studies. Guidance recommends synthesizing different study designs separately due to their different inherent biases/limitations. But when authors or reviewers misclassify studies, this could affect which studies are included and therefore have an impact on review results.

METHODS: We are conducting a methods project to (i) identify a clear study design classification system, (ii) explore whether its use produces consistent study design categorizations among reviewers, and (iii) iteratively improve the classification system. We performed a pragmatic web-based search for study design categorization tools and used the resulting schemas to develop a clear algorithm for use by reviewers of all levels of experience, specifically in reviews of treatment interventions. Next, we tested tool consistency and user experience by web-based survey in a small internal sample of reviewers, each independently using the system to categorize 18 published studies.

RESULTS: A median of seven reviewers (range four to eight) categorized each study. Rater agreement using the chart varied widely, with 100 percent agreement on the designs of three studies (17%), and at least 75 percent of reviewers agreeing on one design for nine studies (50%). The most common agreement was reached on RCTs and non-randomized controlled trials. The most common sources of disagreement were between different types of cohort studies and between case series and controlled cohort studies, largely due to inconsistent reporting. We also identified several improvements: the wording of prompt questions, the ordering of designs, and the addition of new elements.

CONCLUSIONS: The classification system as initially designed led to too much variation in study design categorization to be useful. Consequently, we present a revised version that we now aim to evaluate in a larger sample of reviewers. Further research will also investigate whether using the tool would change the results of systematic reviews, using a small sample of published reviews.

You may also be interested in

Peer-reviewed publication

The Economic and Clinical Impact of an AI-Assisted Tool for Colonoscopy to Support Diagnosis of Colorectal Cancer in Symptomatic Patients in England

YHEC authors: Luc Curtis-Gretton, Angel Varghese, Karina Watts, Hayden Holmes
Publication date: November 2026
Publishers: Springer Nature Link
Journal: BMC Digital Health

Abstract

BACKGROUND: Colorectal cancer (CRC) remains a leading cause of cancer-related mortality in the UK. Early detection and removal of adenomatous polyps are essential to improving outcomes. Artificial intelligence-assisted colonoscopy tools,...

Peer-reviewed publication

Cost-Effectiveness of Transcatheter Aortic Valve Implantation for Asymptomatic Severe Aortic Stenosis in the Kingdom of Saudi Arabia

YHEC authors: Madison Montgomery, Heather Davies, Lucy Hillcoat
Publication date: October 2026
Journal: Frontiers in Cardiovascular Medicine

Abstract

INTRODUCTION: Aortic stenosis (AS) is defined as the narrowing of the aortic valve. The benefits of implementing early aortic valve replacement (AVR), while the AS is asymptomatic, have been demonstrated...

Peer-reviewed publication

What is the Role of Multidisciplinary Methods to Support Environmental Sustainability in Health Technology Assessment?

YHEC authors: Melissa Pegg, Emma Bishop, Karina Watts
Publication date: October 2026
Publishers: HTAi
Journal: International Journal of Technology Assessment in Healthcare

Abstract

Healthcare contributes up to 8.5 percent of global carbon emissions, amongst other environmental impacts including intensive water consumption, plastics in our environment and pharmaceutical associated pollution. Healthcare supply chains are...