Meta-analysis of clinical prediction models

BMJ Open

ObjectivesTo empirically assess the relation between study characteristics and prognostic model performance in external validation studies of multivariable prognostic models.DesignMeta-epidemiological study.Data sources and study selectionOn 16 October 2018, we searched electronic databases for systematic reviews of prognostic models. Reviews from non-overlapping clinical fields were selected if they reported common performance measures (either the concordance (c)-statistic or the ratio of observed over expected number of events (OE ratio)) from 10 or more validations of the same prognostic model.Data extraction and analysesStudy design features, population characteristics, methods of predictor and outcome assessment, and the aforementioned performance measures were extracted from the included external validation studies. Random effects meta-regression was used to quantify the association between the study characteristics and model performance.ResultsWe included 10 systematic review...

BMC Medical Research Methodology, 2014

Background Risk prediction models estimate the risk of developing future outcomes for individuals based on one or more underlying characteristics (predictors). We review how researchers develop and validate risk prediction models within an individual participant data (IPD) meta-analysis, in order to assess the feasibility and conduct of the approach. Methods A qualitative review of the aims, methodology, and reporting in 15 articles that developed a risk prediction model using IPD from multiple studies. Results The IPD approach offers many opportunities but methodological challenges exist, including: unavailability of requested IPD, missing patient data and predictors, and between-study heterogeneity in methods of measurement, outcome definitions and predictor effects. Most articles develop their model using IPD from all available studies and perform only an internal validation (on the same set of data). Ten of the 15 articles did not allow for any study differences in baseline risk...

PLOS ONE, 2016

Many new clinical prediction rules are derived and validated. But the design and reporting quality of clinical prediction research has been less than optimal. We aimed to assess whether design characteristics of validation studies were associated with the overestimation of clinical prediction rules' performance. We also aimed to evaluate whether validation studies clearly reported important methodological characteristics.

OBJECTIVES: Our aim was to improve meta-analysis methods for summarizing a prediction model's performance when individual participant data are available from multiple studies for external validation. STUDY DESIGN AND SETTING: We suggest multivariate meta-analysis for jointly synthesizing calibration and discrimination performance, while accounting for their correlation. The approach estimates a prediction model's average performance, the heterogeneity in performance across populations, and the probability of "good" performance in new populations. This allows different implementation strategies (e.g., recalibration) to be compared. Application is made to a diagnostic model for deep vein thrombosis (DVT) and a prognostic model for breast cancer mortality. RESULTS: In both examples, multivariate meta-analysis reveals that calibration performance is excellent on average but highly heterogeneous across populations unless the model's intercept (baseline hazard) is recalibrated. For the cancer model, the probability of "good" performance (defined by C statistic ≥0.7 and calibration slope between 0.9 and 1.1) in a new population was 0.67 with recalibration but 0.22 without recalibration. For the DVT model, even with recalibration, there was only a 0.03 probability of "good" performance. CONCLUSION: Multivariate meta-analysis can be used to externally validate a prediction model's calibration and discrimination performance across multiple populations and to evaluate different implementation strategies.

Research Synthesis Methods, 2019

A systematic review and meta-analysis is an important step in evidence synthesis. The current paradigm for meta-analyses requires a presentation of the means under a random-effects model; however, a mean with a confidence interval provides an incomplete summary of the underlying heterogeneity in meta-analysis. Prediction intervals show the range of true effects in future studies and have been advocated to be regularly presented. Most commonly, prediction intervals are estimated assuming that the underlying heterogeneity follows a normal distribution, which is not necessarily appropriate. In this article, we provide a simple method with a ready-to-use spreadsheet file to estimate prediction intervals and predictive distributions non-parametrically. Simulation studies show that this new method can provide approximately unbiased estimates compared with the conventional method. We also illustrate the advantage and real-world significance of this approach with a meta-analysis evaluating the protective effect of vaccination against tuberculosis. The nonparametric predictive distribution provides more information about the shape of the underlying distribution than does the conventional method.

BMJ (Clinical research ed.), 2017

Validation of prediction models is highly recommended and increasingly common in the literature. A systematic review of validation studies is therefore helpful, with meta-analysis needed to summarise the predictive performance of the model being validated across different settings and populations. The aims of this article are (1) to provide guidance for systematically reviewing and meta-analysing the existing evidence on a specific prediction model, (2) to discuss 'good practice' when quantitatively summarizing the predictive performance of the model across studies and (3) to provide recommendations for interpreting meta-analysis estimates of model performance. We present key steps of the meta-analysis and illustrate each step in an exemplar review where we summarize the discrimination and calibration performance of the EuroSCORE for predicting operative mortality in patients undergoing coronary artery bypass grafting. Summary points • Systematically reviewing the validation studies of a prediction model may help to identify whether its predictions are sufficiently accurate across different settings and populations. • Efforts should be made to restore missing information from validation studies and to harmonize the extracted performance statistics. • Heterogeneity should be expected when summarizing estimates of a model's predictive performance. • Meta-analysis should primarily be used to investigate variation across validation study results.

2008

This paper proposes a new method of improved meta analysis to combine relative risk for both homogeneous and heterogeneous set of studies. The standard meta analyses don't give any conclusive result when the effects of heterogenous studies are combined. The proposed improved meta analysis uses the predicted relative risk, and chi-square test to check the heterogeneity of the effects. Confidence intervals for the relative risks obtained via improved method concentrate more towards the value of the pooled estimate than that of the standard meta analysis. Exclusion of identified studies with outliers from the analysis brings the results of the remaining studies closer to the pooled estimate. An illustration shows that the new method improves the results and provide conclusive estimate of the relative risk.

Statistical Methods in Medical Research

It is widely recommended that any developed—diagnostic or prognostic—prediction model is externally validated in terms of its predictive performance measured by calibration and discrimination. When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment. We discuss how to undertake meta-analysis of the performance of prediction models with either a binary or a time-to-event outcome. We address how to deal with incomplete availability of study-specific results (performance estimates and their precision), and how to produce summary estimates of the c-statistic, the observed:expected ratio and the calibration slope. Furthermore, we discuss the implementation of frequentist and Bayesian meta-analysis methods, and propose novel empirically-based prior distributions to impr...

Statistics in Medicine

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta-analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6-month mortality based on individual patient data using meta-analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.