Predicting emergency coronary artery bypass graft following PCI: application of a computational model to refer patients to hospitals with and without onsite surgical backup

Introduction

Despite a decline in the rate of emergency coronary artery bypass graft (ECABG) following percutaneous coronary intervention (PCI) in the current-stent era (0.3%–0.6%), the mortality and morbidity associated with ECABG remains constant and high.1–4 Historically, PCI has been performed at sites that have access to onsite cardiac surgery. In recent years, however, the number of primary PCI cases being performed at sites without cardiothoracic surgical backup has grown globally for both patients presenting with acute myocardial infarction (MI) and those undergoing elective procedures.5 The American College of Cardiology (ACC)/American Heart Association (AHA)/Society for Cardiovascular Angiography and Interventions (SCAI guidelines designate primary PCI a class IIa indication (is reasonable), and elective PCI a class IIb indication (may be considered) when performed at facilities without onsite surgical backup with the caveat that appropriate planning for programme development has been accomplished and rigorous clinical and angiographic criteria are used for proper patient selection.6 There has been a steady increase in PCI without surgical backup and it was estimated that over 16% of the facilities participating in the National Cardiovascular Data Registry were performing PCI without onsite backup7 by the end of 2005.

Large meta-analyses and randomised controlled trials have found no differences in adverse outcomes between carefully selected patients undergoing PCI at facilities with onsite cardiac surgery and those undergoing PCI at facilities without surgical backup.8–10 The outcome of emergent surgery is extremely uncommon, however, and these studies have been underpowered with respect to this end point to definitively establish the superiority of one strategy versus the other. What is clear is that the low prevalence of ECABG means that surgical backup may not be needed for the vast majority of patients, and that there will be further increases in the number of hospitals providing PCI without onsite surgery.

In this setting, risk scores or decision-support algorithms that can accurately differentiate between patients at high or low risk of ECABG following PCI hold significant value as screening tools to selectively refer cases to facilities with or without surgical backup. Such tools fit particularly well within a hub-and-spoke model of PCI centres by guiding decisions related to patient transfer.11 The availability of tools that can accurately quantify patient risk also provide for more accurate assessments of quality and outcomes across institutions (both with and without onsite surgical backup) by creating the opportunity to perform risk-adjusted observed-to-expected analyses. However, predicting the risk of ECABG following PCI has traditionally been an extremely hard problem, with the low prevalence and multifactorial nature of events resulting in no satisfactory predictors for this outcome.12

The purpose of this study is to explore a computational approach that addresses this goal by drawing on recent advances in statistical machine learning to integrate information from a diverse set of clinical variables that are individually weak predictors of ECABG following PCI. The specific hypothesis underlying this study is that the process of stratifying patients for ECABG can be significantly improved through algorithms that directly optimise cohort-level metrics relevant for clinical use. Specifically, across a fairly broad range of medical applications, the metrics that determine practical utility (measures such as precision, recall and area under the receiver operating characteristic curve (AUROC)) are defined in terms of how the risk scores or predictions are distributed across a cohort. There is no notion of metrics such as an AUROC, for example, at the level of a single patient and they are defined instead in terms of how the risk scores or predictions change across a set of patients. In contrast, the outstanding majority of existing approaches to stratify patients (including the popular methodology of logistic regression) focus in their underlying optimisation problems on individual patients separately without direct regard for how the risk scores or predictions arising from these approaches collectively translate to cohort-level performance for meaningful performance metrics. This study diverges from this trend, and investigates how the process of predicting ECABG following PCI can be improved through a computational model directly optimised for cohort-level performance using novel machine learning methodology; as an objective basis that can be used for selectively referring patients to hospitals with and without onsite surgical backup.

Methods

Results

Stepwise elimination and variable weighting

Table 6 presents the risk variables with the highest weight in the SVM_AUROC models for ECABG. The top 30 variables and the absolute values of their coefficients in the models are shown. The discrimination of the SVM_AUROC models diminished as the number of variables used to develop the models was reduced. Figure 1 illustrates this observed effect. In general, model performance was stable when 10 or more variables were used for risk stratification.

Table 6

View inline•Open as popup

Risk variables with highest absolute coefficients in the SVM_AUROC model to predict ECABG following PCI (top 30 variables shown)

Figure 1

Request permissions

Relation between the number of variables with highest weights included in the model (x-axis) and the AUROC (y-axis) for predicting ECABG following PCI (maximum AUROC obtained for model with 11 variables). Table 6 provides more information on the top variables with the highest absolute coefficients in the SVM_AUROC model. AUROC, area under the receiver operating characteristic curve; ECABG, emergency coronary artery bypass graft; PCI, percutaneous coronary intervention; SVM, support vector machine.

Discussion

With a growing number of PCI procedures being performed at hospitals without onsite surgical backup, the ability to identify patients at risk of ECABG can help streamline case distribution across providers and allow for patients to be matched to resources that are ideally suited for their individual acuity. Unfortunately, exploring this opportunity or realising its full potential is constrained by the absence of clinical tools that can reliably stratify patients for ECABG after PCI. Prediction of procedural risk is an intricate exercise and is especially difficult for events that are multifactorial and relatively uncommon. Conventional modelling approaches to predict ECABG are severely challenged by these issues. As confirmed by the data from this study, approaches such as logistic regression and even methodologies in the conventional machine learning literature (like standard SVM classification) fail to achieve levels of discrimination that correspond to genuine clinical utility (an AUROC of 0.8 or higher20).

In this setting, the main finding of our study is that it is possible to predict the need for ECABG following PCI with a potentially useful degree of discrimination through the aid of advanced computational methods. These methods performed significantly better in our experiments than both logistic regression and standard SVM techniques. We demonstrated, in particular, the ability of a computational model directly optimised for multivariate non-linear performance at the cohort level using an AUROC loss function to identify subpopulations undergoing PCI at significantly increased or decreased risk of ECABG. This approach achieves higher levels of discrimination, net reclassification and calibration than existing methodologies. One of the primary reasons for this improvement is that unlike the statistical problem formulations of more traditional methods, which focus only on reducing the error assessed at the level of individual training examples, the computational techniques investigated in our study focus on simultaneously considering the classification of multiple training examples. This allows for a reduction of training error on individual examples to be enhanced with the joint maximisation of multivariate cohort-level performance metrics. As a result, the resulting classifier or predictive model is directly constructed to optimise cohort-level metrics that are relevant for practical use and significantly outperforms more conventional techniques.

Multiple models have previously been described to predict in-hospital mortality after PCI and serve to establish benchmarks for quality improvement and public reporting. Prediction of the need for ECABG has implications beyond risk adjustment and informed consent as a growing number of institutions start PCI services without surgical backup. Our methods can help identify patients who should not be treated at sites that do not have cardiac surgery on site and instead should be selectively referred to PCI centres with onsite surgical backup. Furthermore, even at centres that have cardiac surgery on site, it is rare to have surgical team and suite available around the clock and accurate models can help optimise resource utilisation by facilitating optimal planning for procedures that potentially have a higher need for surgical backup. The mortality associated with ECABG after PCI has remained unchanged over the last decade and it is possible that computational models such as the studied here can reduce burden by facilitating better case selection, guiding performance of the extremely high-risk procedures in hybrid catheterisation laboratories with surgery on standby, or the use of alternative treatment such as planned CABG or medical therapy. Finally, regulatory agencies that oversee hospital accreditation can use these models to define reproducible risk thresholds that should trigger transfer of patients from centres without onsite surgery to centres with surgical backup. This is likely to become an important issue as PCI is expanded to a larger number of sites in response to the change in national guidelines and the emerging data.6 ,8 ,10

Our findings are important for many reasons. First, by harnessing sophisticated computational methods, we were able to develop a robust predictive approach for a PCI complication that is relatively uncommon, has dramatic implications for patients, is expensive and has been difficult to reliably predict using traditional methodology. The generality of these methods and their demonstration on an end point that is representative of many PCI complications suggests that the techniques investigated here may have broader merit for stratifying patients undergoing PCI. Second, these methods are fully automated both in terms of training and prediction, and build on data elements that are routinely collected by national PCI registries. The models arising from this work can therefore be easily embedded into the electronic medical records used by the catheterisation laboratories and it should be possible to have the risk calculation automatically generated as soon as diagnostic angiography is completed. This should permit seamless calculation of procedural risk using data collection tools that are routinely used in catheterisation laboratories around the country. Third, the risk for these outcomes is dependent on a multitude of factors that are by themselves only weak predictors of outcome and cannot be incorporated into a simplified risk scores. Yet in aggregate these predictors are powerful determinants of clinical outcomes. We believe that our study provides an important example for how to leverage such data for a variety of clinical use cases.

We conclude with a discussion of some limitations. Given the extremely low prevalence of ECABG even with a highly discriminative model, there are a large number of false positives. For example, among patients in the top 2.5% of the predicted risk scores, the rate of ECABG is only around 2%. Addressing this false-positive rate for events that occur with such low prevalence is difficult (unless models with an AUROC >0.99 can be developed). From the perspective of the use case described here, however, we do not believe that the false-positive rate is prohibitive. For example, the high-risk subgroup above is at an almost 13-fold to 14-fold increased risk and represents over a quarter of all ECABG cases. Moreover, at a rate of 1 in every 50 patients requiring ECABG, this group may be considered appropriate for selectively being referred to hospitals with onsite cardiac surgery. We note, in particular, that the objective of this study is to demonstrate the feasibility of a computational approach such as the one described here as a screening tool. In this setting, despite the potential for improving precision and recall across different decision thresholds, our results offer positive validation through their ability to partition the population of patients undergoing PCI into groups at significantly higher or lower risk. Similarly, at the other end of the spectrum, we also observe that while the computational model constructed in this study was able to robustly stratify patients with a significantly elevated risk of ECABG following PCI, no group could be found that had a zero risk of complications. For example, even in the patients placed within the lowest deciles of risk, there were at least one or two cases of ECABG across several thousand patients. This suggests that even though PCI risk can be minimised by careful case selection, it cannot be totally negated. This factor needs to be recognised by both patients and providers and should be included in the consent process. We also note that despite accounting for information in a broad range of variables, the computational model investigated here does not account for operator experience or judgement, factors that may significantly influence procedural complications. Availability of these data may improve performance further. Related to this, we note that while models such as the one presented here have value in selectively screening patients for PCI to reduce outcomes such as ECABG, many factors other than patient and procedural characteristics influence the occurrence of a rare outcome like ECABG and processes such as multidisciplinary meetings, morbidity and mortality reviews, and medical error reporting are still essential to reducing event rates. Finally, the incidence of ECABG is susceptible to technological advances and may change with the emergence of newer technology. With increasing use of percutaneous support devices, the incidence of these complications may drop further in the future necessitating model recalibration. However, no survival benefit has been demonstrated with the first generation of these devices, and both the development and validation cohorts reflect patients treated with contemporary PCI. The findings of our study are therefore likely extant for the near future; and additionally given the fully automated process of developing the computational model any process of recalibrating it can be implemented without being burdensome. We anticipate, in particular, a use case for our model where it is retrained on an ongoing basis (eg, annually from registry data) to ensure sustained performance across prolonged periods of time.

Conclusions

Computational modelling directly optimised for cohort-level performance can achieve potentially high levels of discrimination for ECABG following PCI. This approach may have value in screening PCI cases and selectively referring patients to hospitals with and without onsite surgical backup.