Predicting complication risk in spine surgery: a ...many spine surgery procedures.6 A previously described Risk Assessment Tool (RAT) was developed for patients undergoing spine surgery

CLINICAL ARTICLEJ Neurosurg Spine 27:81–91, 2017

ABBREVIATIONS ACS NSQIP = American College of Surgeons National Surgery Quality Improvement Program; AUC = area under the ROC curve; BMI = body mass index; BMP = bone morphogenetic protein; CCI = Charlson Comorbidity Index; RAT = Risk Assessment Tool; ROC = receiver operating characteristic. SUBMITTED August 15, 2016. ACCEPTED December 15, 2016.INCLUDE WHEN CITING Published online April 21, 2017; DOI: 10.3171/2016.12.SPINE16969.

Predicting complication risk in spine surgery: a prospective analysis of a novel risk assessment toolAnand Veeravagu, MD,1 Amy Li, BA,1 Christian Swinney, BA,1 Lu Tian, ScD,2 Adrienne Moraff, MD,1 Tej D. Azad, BA,1 Ivan Cheng, MD,3 Todd Alamin, MD,3 Serena S. Hu, MD,3 Robert L. Anderson, MS,4 Lawrence Shuer, MD,1 Atman Desai, MD,1 Jon Park, MD,1 Richard A. Olshen, PhD,2 and John K. Ratliff, MD1

Departments of 1Neurosurgery, 2Biomedical Data Science, and 3Orthopedic Surgery, Stanford University School of Medicine; and 4LogicThread Inc., Stanford, California

OBJECTIVE The ability to assess the risk of adverse events based on known patient factors and comorbidities would provide more effective preoperative risk stratification. Present risk assessment in spine surgery is limited. An adverse event prediction tool was developed to predict the risk of complications after spine surgery and tested on a prospective patient cohort.METHODS The spinal Risk Assessment Tool (RAT), a novel instrument for the assessment of risk for patients undergo-ing spine surgery that was developed based on an administrative claims database, was prospectively applied to 246 patients undergoing 257 spinal procedures over a 3-month period. Prospectively collected data were used to compare the RAT to the Charlson Comorbidity Index (CCI) and the American College of Surgeons National Surgery Quality Im-provement Program (ACS NSQIP) Surgical Risk Calculator. Study end point was occurrence and type of complication after spine surgery.RESULTS The authors identified 69 patients (73 procedures) who experienced a complication over the prospective study period. Cardiac complications were most common (10.2%). Receiver operating characteristic (ROC) curves were calculated to compare complication outcomes using the different assessment tools. Area under the curve (AUC) analysis showed comparable predictive accuracy between the RAT and the ACS NSQIP calculator (0.670 [95% CI 0.60–0.74] in RAT, 0.669 [95% CI 0.60–0.74] in NSQIP). The CCI was not accurate in predicting complication occurrence (0.55 [95% CI 0.48–0.62]). The RAT produced mean probabilities of 34.6% for patients who had a complication and 24% for patients who did not (p = 0.0003). The generated predicted values were stratified into low, medium, and high rates. For the RAT, the predicted complication rate was 10.1% in the low-risk group (observed rate 12.8%), 21.9% in the medium-risk group (observed 31.8%), and 49.7% in the high-risk group (observed 41.2%). The ACS NSQIP calculator consistently produced complication predictions that underestimated complication occurrence: 3.4% in the low-risk group (observed 12.6%), 5.9% in the medium-risk group (observed 34.5%), and 12.5% in the high-risk group (observed 38.8%). The RAT was more accurate than the ACS NSQIP calculator (p = 0.0018).CONCLUSIONS While the RAT and ACS NSQIP calculator were both able to identify patients more likely to experience complications following spine surgery, both have substantial room for improvement. Risk stratification is feasible in spine surgery procedures; currently used measures have low accuracy.https://thejns.org/doi/abs/10.3171/2016.12.SPINE16969

KEY WORDS spine surgery; complications; risk assessment

©AANS, 2017 J Neurosurg Spine Volume 27 • July 2017 81

Unauthenticated | Downloaded 09/10/20 05:00 AM UTC

A. Veeravagu et al.

J Neurosurg Spine Volume 27 • July 201782

There has been a significant rise in the number of spine surgeries performed in the United States over the last 2 decades.5,21,41 Perioperative adverse events

and complications negatively impact clinical and patient-reported outcomes as well as increasing costs.15,43 Under-standing and reducing the incidence of complications is an important clinical goal.

Adverse event occurrence is a common theme in qual-ity-improvement efforts. Many national quality-improve-ment activities champion efforts to decrease complication occurrence in surgical procedures. Adverse event occur-rence may be interpreted as an indicator of lower quality of care.

The incidence of adverse events provides a foundation for numerous quality-reporting metrics. Accurate risk ad-justment is necessary to compare performance between different clinicians. Several studies investigated risk fac-tors affecting the occurrence of postoperative complica-tions in spine surgery patients. Patient factors influence complication occurrence, with sicker patients having a greater risk of perioperative complications.23,27,28,36,45 Poor modeling of complication occurrence and poor risk ad-justment impairs comparative-effectiveness research and makes reliable assessment of expected occurrence of com-plications impossible.

Attempts have been made at risk modeling. Two com-monly used tools for the assessment of complication risk in surgical patients are the Charlson Comorbidity Index (CCI) and the American College of Surgeons National Surgery Quality Improvement Program (ACS NSQIP) Surgical Risk Calculator. The CCI categorizes each pa-tient’s morbidities and assigns an appropriate weight ac-cording to the associated risk of mortality. A sum of all the weights allows for a single cumulative CCI score for each patient.8 A number of studies have found the CCI score to be a competent predictor of survival and outcomes fol-lowing various treatments such as radical prostatectomy, elective noncardiac surgery, dialysis, and peritoneal dialy-sis.7,16,18,19,24 The CCI was not specifically designed to as-sess patients undergoing spine surgery, and the usefulness of the CCI score to accurately predict complication occur-rence after spine surgery remains uncertain.2,22,44

The ACS NSQIP Surgical Risk Calculator was de-veloped from a multi-institutional database of surgical procedures across all surgical specialties and draws on a model utilizing 21 preoperative factors to predict postop-erative outcomes in individual patients.3 The ACS NSQIP Surgical Risk Calculator has been shown to perform as well as procedure-specific risk calculators for colon sur-gery patients.3 No study has evaluated the accuracy of this universal prediction tool in spine surgery patients. The ACS NSQIP calculator is a foundational element noted in a National Quality Forum (NQF)–sanctioned Physician Quality Reporting System element (Measure 358: Patient-Centered Surgical Risk Assessment and Communication); this measure is endorsed by NQF as applicable to patients undergoing a variety of surgical procedures, including many spine surgery procedures.6

A previously described Risk Assessment Tool (RAT) was developed for patients undergoing spine surgery. The tool was developed using statistical modeling of longitu-

dinal prospective data from an administrative claims da-tabase.34 This model predicts adverse events after spine surgery. The RAT incorporates procedure, patient, and di-agnosis elements to develop a prediction rule for individ-ual patients. Better modeling of complication occurrence will improve comparative-effectiveness research, provide superior patient counseling, and foster shared decision making.

MethodsDeveloping the Spinal Risk Assessment Tool

The senior author’s research group had previously de-veloped an algorithm from a statistical model using a lon-gitudinal claims database (MarketScan) of 279,145 spine surgery patients in the United States from 2006 to 2010.34 The algorithm was trained to predict adverse events from the first 179,145 records, with the remaining 100,000 re-cords used for validation. We constructed a prediction rule from the model using logistic regression with main effects and logistic regression with additive main effects, in ad-dition to 2 and 3 factor interactions. Receiver operating characteristic (ROC) curve analysis showed an area under the ROC curve (AUC) of 0.7 for the final model in predict-ing adverse events after spine surgery.

Patient Selection and Study DesignWe initially tested the RAT on a retrospective cohort

of 200 adults who underwent spine surgery performed by a single surgeon (J.K.R.) at Stanford University from 2011 to 2013. This initial study was a simple longitudinal as-sessment of a single practice. No inclusion or exclusion-ary criteria were applied. Complication occurrence was recorded through manual chart abstraction.

After a retrospective review revealed promising results of the RAT, a prospective assessment was performed. To avoid the bias intrinsic to retrospective reviews and to in-crease sample diversity, we conducted this 3-month pro-spective evaluation from February to April 2015, including all adult patients who underwent spine surgery procedures in either the departments of neurosurgery or orthopedic surgery. A trained, independent auditor prospectively fol-lowed a total of 246 patients undergoing 257 procedures and captured postoperative complication occurrences. One patient was lost to follow-up and was excluded only from analyses comparing observed outcomes and model predictions. There were no exclusionary criteria applied to the prospectively tracked patients. All patients were fol-lowed at centralized outpatient facilities, improving cap-ture of postoperative adverse event occurrence.

For all measures, we used a broad definition of postop-erative adverse events.25,35 We captured all adverse events occurring in the postoperative period, regardless of a pre-sumed direct or indirect relationship to the surgery being performed. The study period was limited to the 30 days after a given procedure. The auditor followed patients pro-spectively during their hospitalization and recorded post-operative adverse events occurring after discharge based upon chart review and direct contact with treating clinical teams. While the analysis focused on 4 common types of perioperative complications (pulmonary, cardiac, wound,


Predicting complication occurrence

J Neurosurg Spine Volume 27 • July 2017 83

and thrombotic), a wide variety of complications were captured in our assessment (urinary tract infection, post-operative radiculopathy, dysphagia, delirium, and others).

We applied the algorithm in the RAT to the prospective cohort to test for accuracy in predicting complication risks following spine surgery. The prospective calculation of RAT did not affect preoperative surgical optimization; all patients followed similar standard preoperative protocols. Where necessary, we conducted a retrospective review of the prospectively collected data to calculate the CCI score and the ACS NSQIP surgical risk for each patient.

Assessments for the RATWe applied the RAT to both the retrospective and pro-

spective cohorts. Preoperative and intraoperative char-acteristics (9 factors total) for each patient were entered into the RAT’s interface. Required preoperative variables include age, sex, preoperative diagnosis or indication for surgery (degenerative, trauma, tumor, or infection/miscel-laneous), and comorbidities. The medical comorbidities available to be selected include pulmonary dysfunction, neurological dysfunction, hypercholesterolemia, smoking (active and former), hypertension, cardiac dysfunction, diabetes mellitus, systemic malignancy, gastroesophageal dysfunction, substance abuse, and psychiatric disorder. Intraoperative characteristics include the surgical ap-proach (anterior cervical, anterior thoracolumbar, poste-

rior cervical, or posterior thoracolumbar), the use of bone morphogenetic protein (BMP), surgical fusion, number of operative spinal levels, and the placement of instru-mentation. Based on preoperative factors and surgery performed, the RAT generates a series of risk scores for adverse event occurrence after the planned spine surgery. The risk scores include a general case probability for car-diovascular, thrombotic, pulmonary, wound and overall complications.

Assessment of the CCI and the ACS NSQIP Surgical Risk Calculator

The CCI and the ACS NSQIP Surgical Risk Calcula-tor were applied to the prospective cohort only. Table 1 compares the measured factors between the RAT, CCI, and ACS NSQIP calculator. The CCI generates a summed score based on the number and type of patient morbidities. A higher CCI score indicates a higher risk for potential postoperative complications.8 The ACS NSQIP Surgical Risk Calculator estimates the risk of complications af-ter surgery and is available as an interactive online tool (http://riskcalculator.facs.org/). The calculator was ap-plied to each patient after entering the procedure name or CPT code as well as 21 preoperative factors in its online interface.3 The ACS NSQIP surgical risk output for each patient includes the percent risk of any or a serious com-plication, pneumonia, cardiac complication, surgical site

TABLE 1. Elements assessed by the RAT, CCI, and ACS NSQIP Surgical Risk Calculator

RAT CCI (weighted score) ACS NSQIP Surgical Risk Calculator

AgeSexComorbidities Pulmonary dysfunction Neurological dysfunction Hypercholesterolemia Smoker Hypertension Cardiac dysfunction DM Systemic malignancy Gastroesophageal dysfunction Substance abuse Psychiatric disorderPreop diagnosis Degenerative disease Tumor Trauma Infection/misc.Location of surgery Anterior cervical Posterior cervical Anterior thoracolumbar Posterior thoracolumbar>1 level surgeryUse of BMPFusion statusInstrumentation status

AgeComorbidities MI (1) CHF (1) Peripheral vascular disease (1) Cerebrovascular disease (1) Dementia (1) COPD (1) Connective tissue disease (1) Peptic ulcer disease (1) DM—uncomplicated (1) DM—end-organ damage (2) Moderate to severe CKD (2) Hemiplegia (2) Leukemia (2) Malignant lymphoma (2) Solid tumor—not metastasized (2) Solid tumor—metastatic (6) Liver disease—mild (1) Liver disease—moderate to severe (3) AIDS (6)

AgeBMISexFunctional statusASA classEmergency case statusComorbidities Steroid use for chronic condition Ascites w/in 30 days prior to surgery Systemic sepsis w/in 48 hrs prior to surgery Ventilator dependence Disseminated cancer DM Hypertension requiring medication CHF w/in 30 days prior to surgery Dyspnea Current smoker w/in 1 yr History of severe COPD Dialysis w/in 2 wks prior to surgery Acute renal failureProcedure CPT code

ASA = American Society of Anesthesiologists; CHF = congestive heart failure; CKD = chronic kidney disease; COPD = chronic obstructive pulmonary disease; DM = diabetes mellitus; MI = myocardial infarction; misc. = miscellaneous.


A. Veeravagu et al.


infection, urinary tract infection, venous thromboembo-lism, renal failure, return to operating room, death, and discharge to postacute care. A serious complication was defined by the calculator as cardiac arrest, myocardial in-farction, pneumonia, progressive renal insufficiency, acute renal failure, pulmonary embolism, deep vein thrombo-sis, return to the operating room, deep incisional surgical site infection, organ space surgical site infection, systemic sepsis, unplanned intubation, urinary tract infection, or wound disruption. The calculator further defined the cate-gory of “any complication” as including all of the “serious complications” previously listed in addition to superficial incisional surgical site infection, ventilator usage for more than 48 hours, and stroke.

The ACS NSQIP calculator records complications similar to those captured in our assessment, but there are specific subcategories of complications in the ACS NSQIP calculator not measured in the RAT. Likewise, some rare, severe complications of spine surgery procedures, such as postoperative neurological deficit, are not captured in the ACS NSQIP assessment (Table 2). Our focus is on the overall complication risk measured in the RAT and ACS NSQIP calculator to provide an adequate comparison be-tween the 2 models.

Statistical AnalysisWe compared the performance of 3 prediction models

(RAT, CCI, and ACS NSQIP Surgical Risk Calculator) for predicting complication risk. First, an ROC curve was es-timated for each model based on observed occurrence of complications, and 95% confidence intervals for the AUCs were constructed. Comparisons of the AUCs for the RAT, CCI, and ACS NSQIP Surgical Risk Calculator were per-formed using DeLong’s test.11 Next, we estimated the aver-age distance between the predicted risk and the observed

binary outcome indicating the occurrence of complication to measure prediction accuracy. The average prediction accuracy was compared among 3 predictions with paired t-tests. Lastly, we used a nonparametric spline estimator to estimate complication risk as a function of each of the risk predictions. If a risk prediction is accurate for the true risk, we would expect the nonparametrically estimated risk function to be similar to the identity function. If the estimated risk function is consistently above (or below) the identify function, then the corresponding prediction model systematically underestimates (or overestimates) the actual risk. We performed all data analysis using R (http://www.R-project.org).

ResultsPatient Population and Definition of a Complication

Patient demographics and comparisons of both the ret-rospective cohort (n = 200) and prospective cohort (n = 246) are summarized in Table 3. We used a previously validated, broad definition of perioperative complications, including medical complications and adverse events not directly related to surgery.25,35 We used a 30-day window for adverse event occurrence.

Retrospective AssessmentIn the retrospective cohort (n = 200), the mean age of

the patients (± SD) was 50.8 ± 14.0 years and 57.5% were male. The mean body mass index (BMI) was 28.6 ± 6.5. Most patients (92.5%) presented with at least 1 comorbidity, the most common being hypertension (39.0%), psychiatric disorder (35%), and smoking (34.0%). The primary diag-nosis or indication for a spinal procedure was degenerative disease in 68.0% of patients, trauma in 11.0% of patients, tumor in 9.5% of patients, and infection/miscellaneous in 11.5% of patients. Five of the 200 patients in the retrospec-tive cohort had 2 procedures as part of a staged operation on different dates for a total of 205 procedures. Approxi-mately half (50.2%) of all procedures were performed in the posterior thoracolumbar spine. Other approaches were performed on the anterior cervical (22.4%), anterior thora-columbar (2.9%), and posterior cervical spine (24.4%). Of the total group of procedures, 60.0% were performed on more than 1 spinal level, 3.4% used BMP, 67.3% included fusion, and 68.3% involved instrumentation placement. Seventy-two procedures (35.1%) led to at least 1 complica-tion postoperatively. Of the 4 common complication types assessed (pulmonary, cardiac, thrombosis, and wound-re-lated), the most common was cardiac (9.8%), followed by wound issues (4.9%), pulmonary (4.4%), and thrombosis (1.0%).

Prospective AssessmentIn the prospective cohort (n = 246), the mean age of

the patients was 61.6 ± 15.3 years and 57.3% were male. The mean BMI was 27.4 ± 5.5. Five patients did not have their BMI values recorded. A majority of patients (74.4%) presented with at least 1 comorbidity, the most common being hypertension (45.9%), hypercholesterolemia/hyper-lipidemia (28.9%), and cardiac dysfunction (25.2%). The primary diagnosis or indication for spine surgery was

TABLE 2. Complication types in outcomes measured by the RAT and the ACS NSQIP Surgical Risk Calculator

RAT ACS NSQIP Surgical Risk Calculator

Cardiac dysrhythmiaCHFDeliriumDVTMINeurologicalPneumoniaPulmonaryPulmonary embolusRenal failureUTIWound hematomaWound infectionOther wound complication

Cardiac complication (cardiac arrest or MI)DeathDischarge to nursing or rehab facilityPneumoniaPulmonary embolismReadmissionRenal failure (progressive renal insuf-

ficiency or acute renal failure)Return to ORSSI (deep, superficial, or organ space)StrokeSystemic sepsisUnplanned intubationUTIVenous thromboembolismVentilator > 48 hrsWound disruption

DVT = deep venous thrombosis; OR = operating room; rehab = rehabilitation; SSI = surgical site infection; UTI = urinary tract infection.




spinal degenerative disease (87.9%), followed by trauma (6.5%), tumor (4.5%), and infection/miscellaneous (2.0%). Eleven of the 246 patients in the prospective cohort had 2 procedures on separate days as part of a staged operation, yielding a total of 257 procedures. The procedures largely involved the posterior thoracolumbar spine (63.8%), with 13.2% involving the anterior cervical spine, 9.3% involv-ing the anterior thoracolumbar spine, and 13.6% involving the posterior cervical spine. Of the total group of proce-dures, 48.6% were performed on greater than 1 spinal lev-el, 0.4% involved the use of BMP, 55.6% involved fusion, and 59.1% involved placement of instrumentation.

Of the 246 patients assessed prospectively, 1 patient was lost to follow-up and could not be assessed for post-operative outcomes following surgery. Of the remaining 245 patients with a total of 256 procedures, 69 patients (73 procedures, 28.5%) had at least 1 postoperative compli-cation. We assessed 4 complications specifically (pulmo-nary, cardiac, thrombosis, and wound-related) and found the following rates of complications in the cohort: 10.2% cardiac, 9.8% wound-related, 4.3% pulmonary, and 0.8% thrombosis. Other complications (13.3%) were not classi-fied into the 4 most common groups.

Predicting Adverse Events With RATThe RAT generates a series of risk scores for adverse

events after spine surgery based on 9 preoperative fac-tors: age, sex, diagnosis, comorbidity burden, surgical ap-proach, BMP use, fusion status, number of spinal levels operated on, and instrumentation use. The output of the tool provides the probability of complication occurrence.

We initially tested the RAT on the retrospective cohort to determine its ability to distinguish between patients who would and would not experience a postoperative complication. The mean predicted probability of experi-encing a complication was significantly higher in patients with observed postoperative complications (n = 72, mean 0.449) compared with the no-complication group (n = 133, mean 0.371, p = 0.0436, t-test).

We then applied the RAT to a prospective cohort. Table 4 provides a summary of the predicted probabilities of oc-currence of overall complications as well as specific com-plications. The mean probability of complication occur-rence calculated by the application was 0.273 ± 0.196 in the entire cohort of 257 procedures in 246 patients (range 0.049–0.964). The remaining prospective cohort of 256 procedures (excluding the single procedure undergone by the patient lost to follow-up) was stratified into 3 equal groups based on the calculated probability for complica-tion occurrence in RAT: a low probability score (n = 86, range 0.049–0.157), a medium probability score (n = 85, range 0.158–0.289), and a high probability score (n = 86, range 0.291–0.964). Out of those with low probability scores, 12.8% were observed to have postoperative com-plications. Out of those with medium probability scores, 31.8% ultimately had postoperative complications. The group with high probability scores had the highest per-centage of observed postoperative complications at 41.2% (Fig. 1). The predicted complication rates from the RAT were similar to the observed complication rates in each low, medium, and high score group: 10.1% in the low score

TABLE 3. Patient characteristics in retrospective and prospective cohorts

CharacteristicRetrospective

CohortProspective

Cohortp

Value*

No. of patients 200 246Mean age (yrs) 50.8 ± 14.0 61.6 ± 15.3 <0.001Mean BMI 28.6 ± 6.5 27.4 ± 5.5 0.04Male sex (%) 57.5 57.3 0.75Comorbidity burden (%) 92.5 74.4 0.74 Pulmonary dysfunction 31.0 11.4 0.98 Neurological dysfunction 31.0 4.9 0.99 Hypercholesterolemia 32.5 28.9 0.83 Smoking (active and former) 34.0 6.1 0.19 Hypertension 39.0 45.9 0.72 Cardiac dysfunction 20.5 25.2 0.41 DM 17.0 15.4 0.46 Systemic malignancy 11.5 11.4 0.32 Gastroesophageal dysfunc-

tion26.0 15.9 0.21

Substance abuse 10.5 2.0 0.73 Psychiatric disorder 35.0 20.7 0.75 Other 39.0 11.2 0.22Preop diagnosis (%) 0.55 Degenerative disease 68.0 87.9 Tumor 9.5 4.5 Trauma 11.0 6.5 Infection/misc. 11.5 2.0No. of procedures 205 257Location of surgery (%) 0.09 Anterior cervical 22.4 13.2 Posterior cervical 24.4 13.6 Anterior thoracolumbar 2.9 9.3 Posterior thoracolumbar 50.2 63.8Procedure factors (%) 78.0 68.9 0.20 >1 level surgery 60.0 48.6 0.80 Use of BMP 3.4 0.4 0.01 Fusion status 67.3 55.6 0.64 Instrumentation status 68.3 59.1 0.97Complications (%) 35.1 28.5 0.14 Pulmonary 4.4 4.3 0.72 Cardiac 9.8 10.2 0.79 Thrombosis 1.0 0.8 0.001 Wound 4.9 9.8 0.70 Other 26.8 13.3 0.57

Mean values are given with standard deviations. Percentages for location of surgery, procedure factors, and complications were calculated on the basis of the number of procedures in the given cohort. For all other variable categories, the percentages were calculated on the basis of the number of patients. One patient in the prospective cohort (who underwent 1 operation) was lost to follow-up; for this reason, rates of complications for the prospective cohort were calculated on the basis of 256 procedures rather than 257.* Based on 2-sample t-test for continuous variables and Pearson’s chi-square test for categorical variables.


A. Veeravagu et al.


group, 21.9% in the medium score group, and 49.7% in the high score group.

Complication rates were analyzed by patient BMI on a per-procedure basis in the prospective cohort (256 pro-cedures), using standard BMI categories: less than 18.5 (n = 5, observed complications in 20.0%), 18.5–24.9 (n = 90, observed complications in 25.6%), 25.0–29.9 (n = 81, ob-served complications in 25.9%), and 30.0 or greater (n = 75, observed complications in 34.7%). Univariate regres-sion showed no correlation between BMI and observed complication occurrence (p = 0.143).

Prospective patients who developed complications had, on average, a significantly higher preoperative predicted probability for experiencing an adverse event after spine surgery. The mean predicted probability of a complication was 0.346 for patients who experienced an actual compli-cation (n = 73) compared with 0.242 for patients who had no postoperative complications (n = 183, p = 0.0003).

The RAT further calculated the probabilities of spe-cific complication occurrence in the pulmonary, cardiac, thrombosis, and wound categories for each patient. When stratifying patients in the prospective cohort between those who experienced pulmonary complications post-operatively and those who did not, the mean calculated probability for pulmonary complications with the RAT was significantly different between the groups. Those who actually experienced pulmonary complications (n = 11) had a higher predicted probability of pulmonary com-plications (mean 0.432) compared with the no-pulmonary complication group (n = 245, mean 0.249, p = 0.0004, t-test). The RAT did not distinguish between patients who had a cardiac complication or wound complication and those who did not (n = 26, mean predicted probability 0.462 with a complication and n = 230, mean predicted

probability 0.411 for patients without a complication, p = 0.153, for cardiac complication; n = 25, mean predicted probability 0.246 for patients with a complication and n = 231, mean predicted probability 0.223 for patients without a complication, p = 0.095, for wound complications). Only 2 patients had a thrombosis complication in the entire co-hort; this sample size was insufficient to perform a statisti-cal analysis.

Charlson Comorbidity IndexWe calculated the CCI score for the prospective cohort

only. The overall mean CCI score was 0.853 (range 0–9). A low score (0–3) was calculated for 245 procedures, a medium score (4–5) for 6 procedures, and a high score (6+) for 6 procedures. The low numbers of patients in the CCI medium and high score groups may reflect the inabil-ity of the CCI score to reflect comorbidities that predict complication occurrence in spine surgery procedures.

One patient from the low score group was lost to fol-low-up and was subsequently excluded. Of those in the remaining low score group, 28.7% were observed to have postoperative complications. In the medium score group, 33.3% resulted in actual postoperative complications. In the highest score group, postoperative complications were observed after 16.7% of total procedures (Fig. 1). The CCI score did not distinguish between those who did and did not experience a complication. The mean CCI score for procedures after which patients experienced at least 1 complication was 1.0 compared with 0.8 for procedures after which patients had no complications (p = 0.332, t-test).

ACS NSQIP Surgical Risk CalculatorIn the prospective cohort, the ACS NSQIP Surgical

Risk Calculator yielded individual percent risk scores for developing an adverse event after surgery in the follow-ing categories: serious complication (mean 5.4% ± 3.4%, range 1.3%–22.0%), any complication (mean 7.2% ± 4.7%, range 1.3%–26.9%), pneumonia (mean 0.6% ± 0.7%, range 0.1%–4.0%), cardiac complication (mean 0.3% ± 0.3%, range 0%–2.1%), surgical site infection (mean 1.5% ± 1.1%, range 0.3%–8.1%), urinary tract infection (mean 1.2% ± 1.0%, range 0.2%–6.5%), venous thromboembo-lism (mean 0.8% ± 0.7%, range 0.2%–4.4%), renal failure (0.2% ± 0.2%, range 0%–1.5%), return to operating room (mean 3.1% ± 1.9%, range 0.8%–13.7%), death (mean 0.4% ± 0.8%, range 0%–7.0%), and discharge to postacute care (mean 11.7% ± 14.4%, range 0.9%–77.4%) (Table 4).

The prospective cohort was stratified into 3 groups based on the estimated percent risk of experiencing any complications after surgery, as determined by the ACS NSQIP Surgical Risk Calculator preoperatively. The re-maining cohort (n = 256 procedures) was classified as low (n = 87, range 1.3%–4.7%), medium (n = 84, range 4.8%–7.6%), and high (n = 85, range 7.7%–26.9%) risk score groups. Of those in the low “any complications” risk score group, 12.6% were observed to develop complications. In the medium risk score group, 34.5% were observed to have postoperative complications. Of those categorized in the highest risk score group, 38.8% ultimately developed at least 1 postoperative complication (Fig. 1). The predict-

TABLE 4. Comparison of complication risks generated by the RAT and ACS NSQIP Surgical Risk Calculator for the prospective cohort

Observed

RAT, Mean

(range)

ACS NSQIP Surgical Risk

Calculator, Mean (range)

Any complication risk (%)

28.5 27.3 (4.9–96.4) 7.2 (1.3–26.9)

Complication type risk (%)

Pulmonary 4.3 25.8 (10.1–73.3) 0.6 (0.1–4.0) Cardiac 10.2 41.8 (17.2–78.4) 0.3 (0–2.1) Thrombosis 0.8 11.2 (3.4–28.3) 0.8 (0.2–4.4) Wound/surgical

site9.8 22.6 (10.3–46.5) 1.5 (0.3–8.1)

UTI — 1.2 (0.2–6.5) Renal failure — 0.2 (0–1.5) Return to OR — 3.1 (0.8–13.7) Death — 0.4 (0–7.0) Discharge to post-

acute care— 11.7 (0.9–77.4)




ed complication rates from the ACS NSQIP Surgical Risk Calculator consistently underestimated the observed com-plication rates in each low, medium, and high score group. The calculated predicted complication rate was 3.4% in the low score group, 5.9% in the medium score group, and 12.5% in the high score group.

The ACS NSQIP Surgical Risk Calculator further gen-erated chance of outcome statistics in the form of “above average,” “average,” and “below average” risks of any complication occurrence for each patient. Of the proce-dures leading to an “above average” risk of complication occurrence (n = 229), 27.5% (n = 63) were observed to have at least 1 postoperative complication. Those classified with an “average” risk (n = 21) had an observed complica-tion rate of 42.9% (n = 9). Those classified with a “below average” risk (n = 6) had an observed complication rate of 16.7% (n = 1).

The ACS NSQIP Surgical Risk Calculator significantly distinguished between those who developed complications and those who did not. The mean calculated risk of a seri-ous complication for procedures associated with a postop-erative complication was 6.22% compared with 5.05% in those without any postoperative complications (p = 0.012, t-test). The mean calculated risk of having any complica-tions at all for those who actually experienced a postoper-ative complication (8.55%) was also significantly higher than that in the no-complication group (6.72%, p = 0.004, t-test).

Comparing Prediction ModelsIn ROC analyses, RAT and the ACS NSQIP Surgical

Risk Calculator similarly predicted adverse event occur-rence after spine surgery with moderate efficacy. In the prospective cohort, the AUC for predicting complication occurrence was 0.670 (95% CI 0.599–0.741) in RAT and 0.669 (95% CI 0.601–0.737) in the ACS NSQIP Surgical Risk Calculator. The CCI was a poor predictor of compli-cation occurrence in the spine patient population, with an AUC of 0.555 (95% CI 0.485–0.625). A summary of the ROC analyses is shown in Fig. 2.

The AUCs in the RAT and ACS NSQIP Surgical Risk Calculator were statistically indistinguishable from each other (p = 0.975). However, the AUC in the CCI score was significantly lower (worse) than that in the RAT and the ACS NSQIP Surgical Risk Calculator (p = 0.003, p = 0.003). Based on the ROC curves, the predictive accura-cies of RAT and ACS NSQIP scores are comparably fair.

While ROC curve analysis suggests a relatively high probability of concordance of predicted risk and actual risk of complication in both the RAT and ACS NSQIP models, it does not examine absolute value accuracy. We

FIG. 1. Observed complication rate in groups stratified by low, medium, and high estimated risk scores generated by the RAT, CCI, and ACS NSQIP Surgical Risk Calculator.

FIG. 2. ROC curves for the RAT, CCI, and ACS NSQIP Surgical Risk Calculator in predicting complication occurrence after spine surgery. The gray diagonal line represents prediction by pure chance. The more above and to the left of the diagonal line, the better the predictive ac-curacy of the model. The AUC values are 0.670 for RAT, 0.555 for the CCI, and 0.669 for the ACS NSQIP Surgical Risk Calculator. Figure is available in color online only.


A. Veeravagu et al.


measured the absolute value distance between the predict-ed probability risk and the observed binary complication occurrence in each model. The average prediction error scores for RAT and ACS NSQIP were 0.309 and 0.359, re-spectively. The difference between the 2 scores was -0.051 (95% CI -0.082 to -0.019, p = 0.0018). Therefore, the aver-age prediction error of the scores generated by RAT was significantly lower than that generated by the ACS NSQIP Surgical Risk Calculator.

We further examined the relationship between the out-come of actual complication occurrence and predicted scores. A nonparametrically estimated risk as a function of the risk prediction (RAT or ACS NSQIP model) was graphically reported in Fig. 3. We found that the predict-ed risk scores from RAT were substantially closer to the observed complication risks, while the ACS NSQIP risk scores systematically underestimated the true complica-tion risk for most patients. For the entire prospective popu-lation, the true complication rate was 28.5%. The average predicted complication rate from our model in RAT was 27.2% compared with the 7.2% rate predicted by the ACS NSQIP Surgical Risk Calculator.

DiscussionRisk Adjustment

Assessing the rate of adverse event occurrence is a theme of many quality-improvement efforts; decreasing the risk of complication occurrence in surgical procedures is a worthy goal. Adverse event occurrence may be inter-preted as an indicator of lower quality of care; some public reporting efforts use complication occurrence as a proxy for the quality of care provided by a physician (https:// projects. propublica.org/surgeons/).

Population-based assessments that do not incorporate specific patient and procedural factors may not adequately

risk-adjust and may not appreciate the risk of complica-tions in high-risk patients. Measures that fail to capture ac-curate relative risk profiles and fail to offer risk adjustment may not offer proper assessment of clinician performance. These measures may falsely impugn clinicians who care for high-risk patients.

Quality improvement efforts should incorporate risk adjustment to insure that patient and procedure factors are incorporated into assessing the performance of clinicians.

Risk Assessment ToolOur group developed the Risk Assessment Tool (RAT)

from an administrative claims database of over 270,000 spine patients to predict adverse events in patients under-going spine surgery. This study evaluated the accuracy of the RAT in predicting adverse events after individual spine surgeries. We applied the algorithm prospectively to a patient cohort whose adverse event rate was 28.5% and found that the probability score generated by the RAT significantly distinguished between those that did and did not experience at least 1 postoperative complication. The model was effective at predicting general complication oc-currence with an AUC of 0.670.

A streamlined approach by the RAT is feasible with-out losing predictive power because of our emphasis on specific preoperative and intraoperative risk factors appli-cable to spine surgery patients. Patient variables such as age, sex, and the number and type of medical comorbidi-ties have been found to influence the risk of complications that include deep wound infection, cardiopulmonary com-plication, thromboembolic disease, pulmonary embolism, cerebrovascular accident, and neurological deficit.12,23, 28,

29,36, 37,39 The types of medical comorbidities that the RAT incorporates are diverse, ranging from cardiac dysfunc-tion to psychiatric disorders. Psychiatric disorders are well established in the spine population, considering the high

FIG. 3. Estimated association of predicted risk scores from the RAT (left) and the ACS NSQIP Surgical Risk Calculator (right) with the observed risk of complications after spine surgery. Curves are based on nonparametric spline estimations for the complication risk as a function of the risk prediction. Dotted lines correspond to 95% confidence intervals. The red lines represent the ideal 1:1 correlation with which to compare the data. The rugs at the bottom of each graph show the distribution of the respective predicted risk scores in the prospective cohort. Figure is available in color online only.




rate of depression in patients with back pain.1,31 Procedural variables like the surgical approach, area of spine operated on, fusion status, instrumentation use, and BMP use can impact the development of postoperative complications.4,9,

10,13, 14,17, 30,32, 38,42 These types of preoperative and intraopera-tive risk factors are included in the RAT’s predictive mod-eling algorithm and are assessed in its interface.

Comparison of the RAT and ACS NSQIP CalculatorThe accuracy of the RAT’s prediction model is similar

to that of the current universal adverse events predictor of-fered by the ACS NSQIP Surgical Risk Calculator in ROC analysis. The ACS NSQIP calculator, however, underesti-mated complication occurrence in our prospective patient population. The ACS NSQIP Surgical Risk Calculator was not designed specifically for spine surgery patients, although a subset of these patients were incorporated in its development.3,6 To our knowledge, no study has yet exam-ined the accuracy of the universal ACS NSQIP Surgical Risk Calculator when applied only to spine surgery pa-tients. Our study offers evidence that this risk assessment tool can be applied with fair predictive accuracy to spine surgery populations. The ACS NSQIP Surgical Risk Cal-culator, similar to the RAT, significantly differentiated be-tween patients with and without adverse events after spine surgery. Its predictive accuracy was similar to the RAT with an AUC of 0.669. Despite their similar probabilities of concordance, the average prediction error of the scores generated by the RAT was significantly lower than for the ACS NSQIP Surgical Risk Calculator, suggesting a higher absolute value accuracy of the RAT for predicting adverse events in patients undergoing spine surgery.

The ACS NSQIP Surgical Risk Calculator, while prov-en to be an effective predictive tool from ROC analysis, consistently underestimates true risk in the spine surgery population. The underestimation of complication risk in the ACS NSQIP instrument was evident from another analysis performed outside of the nonparametric spline estimations. After stratifying the percent risk scores into 3 groups (low, medium, and high scores), the predicted complication rates for each group were compared with the observed complication rates. We found that the ACS NSQIP Surgical Risk Calculator consistently underesti-mated the risk of complications in our prospective cohort in each low, medium, and high score group. The predicted complication rates from the RAT were similar to the ob-served complication rates in the low, medium, and high probability score groups.

Limitations of the ACS NSQIP CalculatorThe limitations of the ACS NSQIP Surgical Risk Cal-

culator for spine surgery patients include consistently un-derestimating the risk of complications and generating chance-of-outcome statistics that may not be applicable. Such limitations may result from the universal calculator’s inability to capture neurological complications or other complications specific to spine surgery. While these com-plications are exceedingly rare, they are relevant to patient education and should be captured in prospective risk mod-eling.

Other aspects of the ACS NSQIP Surgical Risk Calcu-

lator may contribute to the difficulty of its use. The ACS NSQIP Surgical Risk Calculator (http://riskcalculator.facs.org/) requires an input of a single CPT code for the type of procedure performed. No option exists to enter the multiple CPT codes that are often required for accurate description of some operations. In this study, we only se-lected the first CPT code provided for each patient, though other aspects of the operation (e.g., instrumentation) were excluded. The ACS Surgical Risk Calculator requires the input of 21 preoperative factors into its online interface in addition to the CPT code. The RAT requires only 9 preop-erative and intraoperative factors.

Charlson Comorbidity IndexThe CCI is another well-studied risk assessment tool

that successfully predicted outcomes after a variety of pro-cedures.7,8,16,18,19,24 While it was not designed specifically for the spine surgery population, previous studies have an-alyzed its utility for predicting outcomes in spine surgery patients. One study found the CCI score to be a significant predictor of 30-day complications following surgery for metastatic spine disease.2 Another study of its use on spine surgery patients found the CCI score to be a useful comor-bidity index but not a complete predictor of postoperative complications due to its inadequacy for capturing spine-specific comorbidities or assigning appropriate weights to conditions that may or may not affect outcomes after spine surgery.44 In this study, we found the CCI score to be a poor predictor of complication occurrence after spine surgery with an AUC of 0.555. In our assessment, the vast majority of our patients had low scores on the CCI. This may reflect the inability of the CCI to capture comorbidi-ties that predict complication occurrence. A significant difference between the predictive model of the CCI and that of both RAT and ACS NSQIP Surgical Risk Calcu-lator demonstrates the inadequacy of the CCI compared with the other models in this study.

Other Approaches to Risk Modeling in Spine SurgeryOthers have explored risk modeling in spine surgery.

Lee et al.26 developed a predictive algorithm (available to the public through SpineSage.com) based on known risk factors. This model was found to be effective, with an AUC of 0.76 for any medical complications and 0.81 for major medical complications. The authors used this model to generate an absolute risk percentage, which was then used to counsel patients regarding potential risks of surgery.26 This represents an advantage of their model, as the ability to provide an absolute percent likelihood of ad-verse events after spine surgery maybe easier for patients to comprehend. While the generation of an absolute risk percentage is similar to the RAT, their model is limited by its basis on a much smaller patient size of 1476 drawn from only 2 academic institutions. In comparison, RAT is more robust, developed from records of 279,145 spine surgery patients nationwide.

Another new classification system, known as Spine Ad-Verse Events Severity (SAVES), was also developed and tested at a single institution to determine its reliability as a tool for providers to identify complications after spine surgery.33 Further studies have validated its use in captur-


A. Veeravagu et al.


ing more adverse events as well as more people with ad-verse events after spine surgery.20,40 The SAVES system is primarily a grading tool for adverse events after they occur and, while useful for possible future refinements of adverse event predictor models, is not meant to be a pre-dictive tool like the RAT for spine surgery patients.

LimitationsThere are significant limitations in the RAT measure.

The RAT measure was developed from an administrative database on which the predictive modeling was based. The development of the statistical model was based on a retro-spective data analysis, and the results depend on the choice of cohort. The large administrative claims database was noted to have an underrepresentation of elderly patients, who represent a large portion of spine surgery patients. One should use caution in generalizing the current results to a different population. The present effort is based upon single-institution prospective assessment; broadening the assessment to other institutions will be required.

Dissemination of the RATWe believe that wider use of the RAT may allow it to

be further refined by other investigators. Similarly, the pre-dictive algorithms used in the RAT calculations are too involved to be routinely used in clinical practice. To pro-vide wider use of the tool, an iOS application incorporat-ing the predictive algorithm has been developed and may be downloaded, free of charge, by interested parties (http://itunes.apple.com/app/ratool/id1087663216).

ConclusionsThis prospective analysis of a predictive tool for as-

sessing the risk of surgical complications illustrates that relative risk adjustment in musculoskeletal procedures is feasible. The predictive tool incorporates patient (demo-graphic, comorbidities), procedure (type of surgery, num-ber of levels, whether fusion and/or instrumentation was performed), and diagnostic (surgical indication) elements into a predictive model.

The investigation validates the algorithms used by the RAT to predict postoperative complications in spine sur-gery patients. Although the RAT and the ACS NSQIP Surgical Risk Calculator have substantial room for im-provement, they both performed significantly better than the CCI. Moreover, the results of this study show that the predictive utility of the RAT is superior to that of the ACS NSQIP Surgical Risk Calculator for this patient popula-tion. The ACS NSQIP Surgical Risk Calculator consistent-ly underestimated complication occurrence in our patient cohort. Patient counseling based upon the operative risk assessment provided by the ACS NSQIP instrument will provide an inaccurately low assessment of operative risks.

These findings may indicate that models particular to a given surgical field may prove more valuable than general, multispecialty tools.

AcknowledgmentsFunding for this work was provided by the Stanford Medical

Scholars Program, the Stanford Society of Physician Scholars, the Walsh Foundation, and the Orthopaedic Research Education Foundation.

References 1. Andersson GB: Epidemiological features of chronic low-back

pain. Lancet 354:581–585, 1999 2. Arrigo RT, Kalanithi P, Cheng I, Alamin T, Carragee EJ,

Mindea SA, et al: Charlson score is a robust predictor of 30-day complications following spinal metastasis surgery. Spine (Phila Pa 1976) 36:E1274–E1280, 2011

3. Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY, et al: Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 217:833–842, 842.e1–842.e3, 2013

4. Cahill KS, Chi JH, Day A, Claus EB: Prevalence, compli-cations, and hospital charges associated with use of bone-morphogenetic proteins in spinal fusion procedures. JAMA 302:58–66, 2009

5. Carragee EJ: The increasing morbidity of elective spinal ste-nosis surgery: is it necessary? JAMA 303:1309–1310, 2010

6. Centers for Medicare and Medicaid Services: Physician Quality Reporting System. Measure codes. CMS.gov (https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment -Instruments/PQRS/MeasuresCodes.html) [Ac-cessed January 27, 2017]

7. Charlson M, Szatrowski TP, Peterson J, Gold J: Validation of a combined comorbidity index. J Clin Epidemiol 47:1245–1251, 1994

8. Charlson ME, Pompei P, Ales KL, MacKenzie CR: A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 40:373–383, 1987

9. Chen SH, Lee CH, Huang KC, Hsieh PH, Tsai SY: Postop-erative wound infection after posterior spinal instrumenta-tion: analysis of long-term treatment outcomes. Eur Spine J 24:561–570, 2015

10. Cole T, Veeravagu A, Jiang B, Ratliff JK: Usage of recom-binant human bone morphogenetic protein in cervical spine procedures: analysis of the MarketScan longitudinal data-base. J Bone Joint Surg Am 96:1409–1416, 2014

11. DeLong ER, DeLong DM, Clarke-Pearson DL: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44:837–845, 1988

12. Deyo RA, Cherkin DC, Loeser JD, Bigos SJ, Ciol MA: Mor-bidity and mortality in association with operations on the lumbar spine. The influence of age, diagnosis, and procedure. J Bone Joint Surg Am 74:536–543, 1992

13. Deyo RA, Ching A, Matsen L, Martin BI, Kreuter W, Jarvik JG, et al: Use of bone morphogenetic proteins in spinal fusion surgery for older adults with lumbar stenosis: trends, compli-cations, repeat surgery, and charges. Spine (Phila Pa 1976) 37:222–230, 2012

14. Deyo RA, Ciol MA, Cherkin DC, Loeser JD, Bigos SJ: Lum-bar spinal fusion. A cohort study of complications, reopera-tions, and resource use in the Medicare population. Spine (Phila Pa 1976) 18:1463–1470, 1993

15. Deyo RA, Mirza SK, Martin BI, Kreuter W, Goodman DC, Jarvik JG: Trends, major medical complications, and charges associated with surgery for lumbar spinal stenosis in older adults. JAMA 303:1259–1265, 2010

16. Di Iorio B, Cillo N, Cirillo M, De Santo NG: Charlson Co-morbidity Index is a predictor of outcomes in incident hemo-dialysis patients and correlates with phase angle and hospital-ization. Int J Artif Organs 27:330–336, 2004

17. Faciszewski T, Winter RB, Lonstein JE, Denis F, Johnson L:




The surgical and medical perioperative complications of an-terior spinal fusion surgery in the thoracic and lumbar spine in adults. A review of 1223 procedures. Spine (Phila Pa 1976) 20:1592–1599, 1995

18. Fried L, Bernardini J, Piraino B: Charlson comorbidity index as a predictor of outcomes in incident peritoneal dialysis pa-tients. Am J Kidney Dis 37:337–342, 2001

19. Froehner M, Koch R, Litz R, Heller A, Oehlschlaeger S, Wirth MP: Comparison of the American Society of Anes-thesiologists Physical Status classification with the Charlson score as predictors of survival after radical prostatectomy. Urology 62:698–701, 2003

20. Glennie RA, Noonan VK, Fallah N, Park SE, Thorogood NP, Cheung A, et al: Reliability of the spine adverse events se-verity system (SAVES) for individuals with traumatic spinal cord injury. Spinal Cord 52:758–763, 2014

21. Gray DT, Deyo RA, Kreuter W, Mirza SK, Heagerty PJ, Comstock BA, et al: Population-based trends in volumes and rates of ambulatory lumbar spine surgery. Spine (Phila Pa 1976) 31:1957–1964, 2006

22. Guyot JP, Cizik A, Bransford R, Bellabarba C, Lee MJ: Risk factors for cardiac complications after spine surgery. Evid Based Spine Care J 1:18–25, 2010

23. Kalanithi PS, Patil CG, Boakye M: National complication rates and disposition after posterior lumbar fusion for ac-quired spondylolisthesis. Spine (Phila Pa 1976) 34:1963–1969, 2009

24. Kastner C, Armitage J, Kimble A, Rawal J, Carter PG, Venn S: The Charlson Comorbidity Score: a superior comorbid-ity assessment tool for the prostate cancer multidisciplinary meeting. Prostate Cancer Prostatic Dis 9:270–274, 2006

25. Lebude B, Yadla S, Albert T, Anderson DG, Harrop JS, Hili-brand A, et al: Defining “complications” in spine surgery: neurosurgery and orthopedic spine surgeons’ survey. J Spi-nal Disord Tech 23:493–500, 2010

26. Lee MJ, Cizik AM, Hamilton D, Chapman JR: Predicting medical complications after spine surgery: a validated model using a prospective surgical registry. Spine J 14:291–299, 2014

27. Lee MJ, Hacquebord J, Varshney A, Cizik AM, Bransford RJ, Bellabarba C, et al: Risk factors for medical complica-tion after lumbar spine surgery: a multivariate analysis of 767 patients. Spine (Phila Pa 1976) 36:1801–1806, 2011

28. Manninen PH, Raman SK, Boyle K, el-Beheiry H: Early postoperative complications following neurosurgical proce-dures. Can J Anaesth 46:7–14, 1999

29. Massie JB, Heller JG, Abitbol JJ, McPherson D, Garfin SR: Postoperative posterior spinal wound infections. Clin Or-thop Relat Res (284):99–108, 1992

30. Memtsoudis SG, Hughes A, Ma Y, Chiu YL, Sama AA, Gi-rardi FP: Increased in-hospital complications after primary posterior versus primary anterior cervical fusion. Clin Or-thop Relat Res 469:649–657, 2011

31. Polatin PB, Kinney RK, Gatchel RJ, Lillo E, Mayer TG: Psy-chiatric illness and chronic low-back pain. The mind and the spine—which goes first? Spine (Phila Pa 1976) 18:66–71, 1993

32. Pradhan BB, Nassar JA, Delamarter RB, Wang JC: Single-level lumbar spine fusion: a comparison of anterior and pos-terior approaches. J Spinal Disord Tech 15:355–361, 2002

33. Rampersaud YR, Neary MA, White K: Spine adverse events severity system: content validation and interobserver reliabil-ity assessment. Spine (Phila Pa 1976) 35:790–795, 2010

34. Ratliff JK, Balise R, Veeravagu A, Cole TS, Cheng I, Olshen RA, et al: Predicting occurrence of spine surgery complica-tions using “big data” modeling of an administrative claims database. J Bone Joint Surg Am 98:824–834, 2016

35. Ratliff JK, Lebude B, Albert T, Anene-Maidoh T, Anderson G, Dagostino P, et al: Complications in spinal surgery: com-

parative survey of spine surgeons and patients who under-went spinal surgery. J Neurosurg Spine 10:578–584, 2009

36. Schoenfeld AJ, Ochoa LM, Bader JO, Belmont PJ Jr: Risk factors for immediate postoperative complications and mor-tality following spine surgery: a study of 3475 patients from the National Surgical Quality Improvement Program. J Bone Joint Surg Am 93:1577–1582, 2011

37. Schoenfeld AJ, Reamer EN, Wynkoop EI, Choi H, Bono CM: Does patient sex affect the rate of mortality and complica-tions after spine surgery? A systematic review. Clin Orthop Relat Res 473:2479–2486, 2015

38. Singh K, Ahmadinia K, Park DK, Nandyala SV, Marquez-Lara A, Patel AA, et al: Complications of spinal fusion with utilization of bone morphogenetic protein: a systematic re-view of the literature. Spine (Phila Pa 1976) 39:91–101, 2014

39. Stolke D, Sollmann WP, Seifert V: Intra- and postoperative complications in lumbar disc surgery. Spine (Phila Pa 1976) 14:56–59, 1989

40. Street JT, Thorogood NP, Cheung A, Noonan VK, Chen J, Fisher CG, et al: Use of the Spine Adverse Events Sever-ity System (SAVES) in patients with traumatic spinal cord injury. A comparison with institutional ICD-10 coding for the identification of acute care adverse events. Spinal Cord 51:472–476, 2013

41. Weinstein JN, Lurie JD, Olson PR, Bronner KK, Fisher ES: United States’ trends and regional variations in lumbar spine surgery: 1992-2003. Spine (Phila Pa 1976) 31:2707–2714, 2006

42. Weinstein MA, McCabe JP, Cammisa FP Jr: Postoperative spinal wound infection: a review of 2,391 consecutive index procedures. J Spinal Disord 13:422–426, 2000

43. Whitmore RG, Stephen J, Stein SC, Campbell PG, Yadla S, Harrop JS, et al: Patient comorbidities and complications after spinal surgery: a societal-based cost analysis. Spine (Phila Pa 1976) 37:1065–1071, 2012

44. Whitmore RG, Stephen JH, Vernick C, Campbell PG, Yadla S, Ghobrial GM, et al: ASA grade and Charlson Comorbidity Index of spinal surgery patients: correlation with complica-tions and societal costs. Spine J 14:31–38, 2014

45. Wise JJ, Fischgrund JS, Herkowitz HN, Montgomery D, Kurz LT: Complication, survival rates, and risk factors of surgery for metastatic disease of the spine. Spine (Phila Pa 1976) 24:1943–1951, 1999

DisclosuresDr. Cheng reports consultant relationships with NuVasive, Globus Medical, and Stryker Spine.

Author ContributionsConception and design: Ratliff, Veeravagu, Li, Swinney, Tian, Azad, Cheng, Hu, Desai, Olshen. Acquisition of data: Ratliff, Veeravagu, Swinney, Moraff, Cheng, Alamin, Hu, Shuer, Desai, Park. Analysis and interpretation of data: Ratliff, Veeravagu, Li, Swinney, Tian, Azad, Anderson, Olshen. Drafting the article: Ratliff, Veeravagu, Li, Swinney, Tian, Moraff, Azad, Cheng, Alamin, Hu, Desai, Park, Olshen. Critically revising the article: Ratliff, Cheng, Olshen. Reviewed submitted version of manu-script: Ratliff, Tian, Shuer. Approved the final version of the manuscript on behalf of all authors: Ratliff. Statistical analysis: Ratliff, Li, Tian, Azad, Anderson, Olshen. Administrative/tech-nical/material support: Anderson. Study supervision: Ratliff, Tian.

CorrespondenceJohn Ratliff, Department of Neurosurgery, Stanford University School of Medicine, 300 Pasteur Dr., Stanford, CA 94305-5327. email: [email protected].


Documents

Predicting complication risk in spine surgery: a ...many spine surgery procedures.6 A previously described Risk Assessment Tool (RAT) was developed for patients undergoing spine surgery