Methodological Research

NICE is increasingly turning to the use of cost comparison analysis to cope with increased demand for HTA without an increase in budgets. The fundamental requirement for such an analysis to be valid is demonstration of similar effectiveness and safety between the intervention and relevant comparator(s).

In many cases a head-to-head study is not available and demonstration of similar effectiveness must be performed via NMA. Unfortunately, there is no guidance provided by NICE (or any other HTA agency) on what should be defined as “similar.” In the context of head-to-head non-inferiority studies demonstration of non-inferiority has generally been accepted as sufficient.  In the context of an NMA the acceptable margins are less clear.

A number of publications are available which consider what the definition of non-inferiority should be within the context of NMA, however, the next step of consideration of how this methods might best be applied within a HTA context has not been undertaken. This research, funded via the ABPI, aims to identify and synthesise available literature discussing potential methods to demonstrate “similarity” or non-inferiority within an NMA context and applied examples of these methods and use this to produce recommendations on methods suitable for a HTA context and describe the pros and cons of the various alternatives.

People involved: GJ Melendez Torres, Dawn Lee

Surrogate outcomes have been defined as biomarkers or laboratory measures used as substitutes for final patient-relevant outcomes in clinical trials in order to assess the efficacy of health technologies. The advantages of surrogate over final outcomes are that they may occur faster or may be easier to assess, thereby shortening the duration, size, and cost of clinical trials. But relying on surrogate outcomes evidence alone has been shown to lead to unreliable conclusions about the value of new treatments.
As part of a PhD and postdoctoral research work, we have explored key issues related to the use of surrogate outcomes in HTA and contributed to improve technology appraisal that uses surrogate outcomes evidence. Oriana Ciani and Rod Taylor have been involved in the update of the Methods Guidance for Technology Appraisal at the National Institute of Health and Care Excellence (NICE) released in 2013 (https://www.nice.org.uk/process/pmg9/chapter/foreword), as well as to the EUnetHTA guideline for surrogate endpoints in Relative Effectiveness Assessment (REA) of pharmaceuticals. They are also collaborating with colleagues at the University of Leicester on the joint use of multiple surrogate endpoints to improve their predictive value.
Future research development will involve application of trial-level and individual patients data validation strategies to putative surrogate outcomes in cardiovascular diseases.

Publications:
Ciani O, Buyse M, Drummond M, Rasi G, Saad ED, Taylor RS. Time to review the role of surrogate endpoints in health policy: state of the art and the way forward. Value in Health 2016 (in press)

Ciani O, Buyse M, Drummond M, Rasi G, Saad ED, Taylor RS. Use of surrogate endpoints in healthcare policy: proposal for consistent adoption of a validation framework. Nature Rev Drug Disc 2016 Jun 3. doi: 10.1038/nrd.2016.81.

Ciani O, Taylor RS. Validation of surrogate end points for overall survival in advanced colorectal cancer: A harmonized approach is needed. J Clin Epidemiol. 2016 Feb;70:277-8.

Ciani O, Buyse M, Garside R, Peter J, Saad E, Stein K, Taylor RS. Validation of surrogateoutcomes for overall survival in advanced colorectal cancer: a systematic review and metaanalysis of randomised controlled trials. J Clin Epidemiol. 2015 Jul;68(7):833-842.

Taylor RS, Piepoli MF, Smart N, Coats AJ, Ellis S, Dalal H, O'Connor CM, Warren FC, Whellan D, Ciani O. Exercise training for chronic heart failure (ExTraMATCH II): Protocol for an individual participant data meta-analysis. Int J Cardiol. 2014 Apr 26. pii: S0167-5273(14)00885-7.

Ciani O, Davis S, Tappenden P, Garside R, Stein K, Buyse M, Saad E, Taylor RS. Validation of surrogate end points in advanced solid tumours: systematic review of statistical methods, results and implication for policy-makers. Int J Tech Assessment Health Care 2014 Jul;30(3):312-24.

Ciani O, Hoyle M, Pavey T, Cooper C, Garside R, Rudin C, Taylor RS. Complete cytogenetic response and major molecular response as surrogate outcomes for overall survival in first-line treatment of chronic myelogenous leukemia: a case-study for technology appraisal on the basis of surrogate outcomes evidence. Value in Health. 2013 Sep-Oct;16(6):1081-90. doi: 10.1016/j.jval.2013.07.004.

Ciani O, Buyse M, Garside R, Pavey T, Stein K, Sterne JA, Taylor RS. Comparison of treatment effect sizes associated with surrogate and final patient relevant outcomes in randomised controlled trials: meta-epidemiological study. BMJ. 2013 Jan 29;346:f457. doi: 10.1136/bmj.f457.

Ciani O, Taylor RS. Surrogate, friend or foe? The need for case studies of the use of surrogate outcomes in cost-effectiveness analyses. Health Economics. 2013 Feb;22(2):251-2. doi: 10.1002/hec.2826. Epub 2012 May 24.

Taylor RS, Elston J. The use of surrogate outcomes in model-based cost-effectiveness analyses: a survey of UK health technology assessment reports. Health Technology Assessment 2009;13:8

People involved: Oriana Ciani, Sarah Walker, Rod Taylor

Projects:
Poor quality reporting of studies and related problems of reporting biases threaten the usefulness of health research to inform policy-making. Mark Pearson and Jaime Peters have explored the existence and likely impact of reporting bias in a systematic review of public health interventions, finding that almost a quarter of included studies were affected by some form of reporting bias with the potential to bias the findings of the systematic review.

With Thomas Fuller and Rob Anderson, Mark and Jaime are also involved in a CLAHRC-funded project to assess how the TREND (Transparent Reporting of Evaluations with Non-Randomised Designs) reporting guideline has been used and whether its use is associated with improved reporting completeness. Questionnaires and interviews with authors and journal editors have also been undertaken to explore the contextual and psychological factors that affect the use of reporting guidelines. The team expect this work to make a significant contribution to the understanding of the how the TREND reporting guideline is used by health researchers, and, more generally, how strategies to improve reporting quality can be enhanced.

Jaime has also been involved in a number of studies investigating methods for the detection of funnel plot asymmetry in meta-analyses.

Publications:
Fuller, T.E., Peters, J.L. Pearson, M., Anderson, R. Impact of the Transparent Reporting of Evaluations with Nonrandomized Designs reporting guideline: ten years on. American Journal of Public Health. 2014; 104(11):e110-e117. Abstract

Fuller, T.E., Pearson, M., Peters, J.L. Transparent reporting, the foundation for full disclosure. A letter to Peters, Abraham & Crutzen (2012) and Hagger, Conner & O’Connor (2013). The European Health Psychologist. 2013; 15(3):67-68. Letter

Fuller, T., Pearson, M., Peters, J., Anderson, R. Evaluating the impact and use of Transparent Reporting of Evaluations with Non-randomised Designs (TREND) reporting guidelines. BMJ Open. 2012; 2(6):e002073. Abstract

Pearson, M., Peters, J. Outcome reporting bias in evaluations of public health interventions: evidence of impact and the potential role of a study register. Journal of Epidemiology and Community Health. 2012; 66(4):286-9. Abstract

(Also see commentary: Hawe, P. The truth, but not the whole truth? Call for an amnesty on unreported results of public health interventions Journal of Epidemiology & Community Health.2012; 66:285)

Sterne, J.A.C., Sutton, A.J., Ioannidis, J.P.A., Terrin, N., Jones, D.R., Lau, J., Carpenter, J., Rucker, G., Harbord, R.M., Schmid, C.H., Tetzlaff, J., Deeks, J.J., Peters, J., Macaskill, P., Schwarzer, G., Duval, S., Altman, D.G., Moher, D., Higgins, J.P.T. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. British Medical Journal. 2011;343. Abstract

People involved: Jaime Peters, Mark Pearson, Rob Anderson, Thomas Fuller

Projects:
Bibliographic databases are a day-to-day tool of the researcher: they offer the researcher easy and organised access to knowledge, but how much is actually known about the databases on offer and the role they can play in secondary research and analysis?

Led by Chris Cooper, the IS team (Morwenna Rogers, Alison Bethel, Simon Briscoe and Jenny Lowe) have investigated the role UK bibliographic databases play in secondary research.  This study has been conducted in two parts: First, a mapping review has been undertaken to locate studies which analyse the content, utility and functionality of core UK health and social care databases. Secondly, this mapping review has been supplemented by 14 face-to-face interviews with senior information professionals to explore how these bibliographic databases contribute to systematic reviews from a user’s perspective.

The findings of this study suggest that UK-focused health and social care bibliographic databases have a distinct role to play in systematic reviews. As reviews become increasingly complex, with varying and specific information needs, an evidence base on how bibliographic databases work and what they can uniquely offer researchers is timely. From locating specific UK studies, which would not have been picked up by the international databases of MEDLINE and EMBASE, to finding specific, hard-to-locate pieces of grey literature for theory-led or qualitative reviews, the team’s research demonstrates the role which these databases will come to play in systematic reviews.

The team are presently involved in writing up further studies relating to this topic. In particular, a series of head-to-head studies comparing UK-focused databases against those which are international in their remit. This work has been funded by the Health Libraries Group and the University Health and Medical Librarians Group, through the ‘Research In the Workplace Award’.

Publications:
Cooper, C., O'Mara-Eves, A., Rogers, M., Bethel, A., Lowe, J., Crathorne, L., Gomersall, A. The best of the UK? A report on the value and future of UK databases in the health and social care fields: a systematic map protocol. BMJ Open. 2012; 2(3). Abstract

Cooper, C., Rogers, M., Briscoe, S., Bethel, A., Lowe, J. A Mapping Review of the Literature on UK-focused Health and Social Care Databases Health Libraries Journal [forthcoming]

Briscoe, S., Cooper, C., The British Nursing Index and CINAHL: A comparison of journal title coverage and the implications for information professionals Health Libraries Journal [forthcoming]

Bethel, A., Rogers, M. A checklist to assess database-hosting platforms for designing and running searches for systematic reviews. Health Information and Libraries Journal. 2014; 31(1): 43-53. Abstract

People involved: Chris Cooper, Morwenna Rogers, Alison Bethel, Simon Briscoe, Jenny Lowe

Projects:
Realist approaches to evaluation and synthesis can provide a logic of inquiry for researching policy and practice, and provide contextualised understanding of the mechanisms that underlie complex processes. We aim to develop realist research methods ‘at the sharp end of practice’ by doing realist evaluation and synthesis in projects of high-relevance to stakeholders and decision-makers. We do this particularly in relation to health service design, the development and implementation of complex interventions in health care, and knowledge mobilisation in health and social care.

We promote understanding of realist approaches and facilitate debate and methodological development through our ‘Realist Hive’ blog and discussion group.

Ongoing projects include:
The effectiveness and cost effectiveness of shared care for long term conditions: A realist review. Protocol

Developing a collaborative care intervention for prisoners with common mental health problems, near to and after release (Engager II): A realist review. Protocol

Understanding how appraisal of doctors produces its effects: a realist review. Protocol

Publications:
Pearson, M., Hunt, H., Cooper, C., Sheppard, S., Pawson, R. & Anderson, R. (2014) Providing effective and preferred care closer to home: A realist review of intermediate care. Health and Social Care in the Community (accepted for publication)

Chilton, R., Pearson, M. & Anderson, R. (2014) Health promotion in schools: a scoping review of systematic reviews. Health Education (accepted for publication)

Pearson, M., Chilton, R., Buckley Woods, H., Wyatt, K., Ford, T., Abraham, C. & Anderson, R. (2013) Implementing health promotion programmes in schools: a realist systematic review of research and experience in the UK. Final report. Exeter: PenTAG, NIHR School for Public Health Research.

Pearson, M., Hunt, H., Cooper, C., Shepperd, S., Pawson, R., & Anderson, R. (2013) Intermediate care: a realist review and conceptual framework. Final report. NIHR Service Delivery and Organisation Programme. Report

People involved: Mark Pearson, Rebecca Hardwick, Rob Anderson, Harriet Hunt, Chris Cooper

Projects:
Improved curve fits to summary survival data: Mean costs and quality-adjusted-life-years are central to the cost-effectiveness of health technologies. They are often calculated from time to event curves such as for overall survival and progression-free survival. Ideally, estimates should be obtained from fitting an appropriate parametric model to individual patient data. However, such data are usually not available to independent researchers. Instead, it is common to fit curves to summary Kaplan-Meier graphs, either by regression or by least squares. A more accurate method of fitting survival curves to summary survival data is investigated and an easy-to-use Excel spreadsheet to implement the method is available here https://www.mq.edu.au/research/research-centres-groups-and-facilities/prosperous-economies/centres/centre-for-the-health-economy/our-people/team-bios/prof-hoyle.

Multiple patient cohorts: Most health technology economic evaluations simulate only the prevalent cohort or the next incident cohort of patients.  We investigate how to estimate and aggregate the incremental cost effectiveness ratios (ICERs) for both currently eligible (prevalent) and future (incident) patient cohorts within the same model-based analysis.  Assuming multiple incident cohorts and decreasing real price of drugs, all drugs are predicted to be better value than assuming just a single incident cohort.

Assumptions for future drug prices in HTA: Cost-effectiveness analyses worldwide assume that the price of any single drug increases with inflation.  The objective of this analysis is to challenge the widespread assumption that the price of any single drug increases with inflation in the UK, and to calculate the impact on the incremental cost-effectiveness ratio (ICER) of using a more realistic estimate for the future price of individual drugs.  In this analysis the mean annual decrease in the real price of individual drugs was calculated as 3.8%.

Publications:
Hoyle, M., Henley, W. Improved curve fits to summary survival data: application to economic evaluation of health technologies. BMC Medical Research Methodology. 2011; 11:139. Open Access

Hoyle, M., Anderson, R. Whose drugs and benefits? Why economic evaluations should simulate both prevalent and all future incident patient cohorts. Medical Decision Making. 2010; 30(4):426-437. Abstract

Hoyle, M. Accounting for the drug life cycle and future drug prices in cost-effectiveness analysis. Pharmacoeconomics. 2011; 29(1):1-15. Abstract

Hoyle, M. Future drug prices and cost-effectiveness analyses. Pharmacoeconomics. 2008; 26(7), 589-602. Abstract

People involved: Martin Hoyle, Rob Anderson

Health Technology Assessment (HTA) is increasingly used worldwide by policymakers to inform decisions about health technology coverage and reimbursement. However, the current HTA framework may not fully address the challenges presented by non-drug technologies, in particular the differences of medical devices relative to drugs (i.e. rapid changes overtime, clinical outcomes dependency on the training and experience of the operator, and dynamic pricing associated with devices in contrast to drugs).

Methods for Health Technology Assessment of Medical Devices (MedtecHTA) is a European Union commissioned project which focused on improving the existing methodological framework of HTA of medical devices and developing a tool that provides structured, evidence-based input into health policies.

The project started in January 2013 and was completed in December 2015. An international team of researchers, representative of six universities and one scientific association, completed seven related MedtecHTA work packages. The stream led by ESMI on cross-country comparison of structures, processes, methods for the HTA of medical devices has been awarded the Egon Jonsson Award during the HTAi 2016 conference in Tokyo.

Over the course of three years, recommendations to policy-makers, clinical professionals and analysts have been issued on improved methods for comparative effectiveness, economic evaluation and multidimensional evaluation of medical devices, including organizational impact and uncertainty surrounding coverage decisions and further investments in research.

Publications:
Ciani O, Wilcher B, van Giessen A, Taylor RS. Linking the regulatory and reimbursement processes for medical devices. Health Economics 2016 (in press)

Schnell-Inderst P, Iglesias CP, Arvandi M, Ciani O, Matteucci Gothe R, Peters J, Blom AW, Taylor RS, Siebert U. Synthesis of RCT and observational data: a bias adjusted evidence synthesis of total hip replacement. Health Economics 2016 (in press)

Ciani O, Wilcher B, Blankart RC, Hatz M, Prevolnik Rupel V, Slabe Erker R, Varabyova Y, Taylor RS. Health Technology Assessment of Medical Devices: A survey of non-European Union Agencies. Int J Technol Assess Health Care. 2015 Jan;31(3):154-65.

People involved: Oriana Ciani, Jaime Peters, Rod Taylor

Pushing the boundaries of Cost and Outcome analysis of Medical Technologies (COMED) is a European Commission Horizon 2020 funded project.

The overarching objective is to push the boundaries of existing methods for cost and outcome evaluation of medical devices, both within the Health Technology Assessment (HTA) and Health System Performance (HSP) frameworks and to develop tools to foster the use of economic evaluation in policymaking. COMED is a highly ambitious project whose predominant purpose is to tangibly improve decision-making process in European healthcare systems, by providing a rigorous evidence base on some of the most pressing issues on the health policy agenda. COMED is a three-year long, intensive research endeavour that will develop novel concepts and approaches across the costs and outcomes evaluation of medical technologies, including use and evaluation of surrogate outcomes.

COMED involves a consortium of research groups that include:
ESMI/PenTAG, University of Exeter Medical School, Exeter;
CERGAS Bocconi University, Milan (coordinator);
HCHE, University of Hamburg, Hamburg;
University of Bern, Bern;
Erasmus University, Rotterdam;
Syreon Research Institute, Budapest.

People involved: Oriana Ciani, Bogdan Grigore and Rod Taylor