Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance

BIO656--Multilevel Models 1Term 4, 2006

PART 07PART 07

Evaluating Hospital PerformanceEvaluating Hospital Performance


PERFORMANCE MEASURESPERFORMANCE MEASURES

Patient outcomes • Mortality, morbidity, satisfaction with care

– 30-day mortality among heart attack patients (Normand et al JAMA 1996, JASA 1997)

Process • Medication & test administration, costs

– Laboratory costs for diabetic patients– Number of physician visits

• Hofer et al JAMA, 1999– Palmer et al. (1996)


DATA STRUCTUREDATA STRUCTURE

Multi-level• Patients nested in physicians, hospitals, HMOs, ...

• Providers clustered by health care systems, market areas, geographic areas

• Covariates at different levels of aggregation: – patient, physician, hospital, ...

Variation in variability• Statistical stability varies over physicians, hospitals, ..


MLMs are EffectiveMLMs are Effective

Correlation at many levels• Hospital practices may induce a strong correlation

among patient outcomes within hospitals even after accounting for patient characteristics

Structuring estimation• Stabilizing noisy estimates • Balancing SEs• Estimating ranks and other non-standard summaries


The Cooperative The Cooperative Cardiovascular Project (CCP)Cardiovascular Project (CCP)

• Abstracted medical records for patients discharged from hospitals located in Alabama, Connecticut, Iowa, and Wisconsin (June 1992May 1993)

• 3,269 patients hospitalized in 122 hospitals in four US States for Acute Myocardial Infarction


GOALSGOALS

• Identify “aberrant” hospitals with respect to several performance measures

• Report the statistical uncertainty associated with ranking of the “worst hospitals”

• Investigate if hospital characteristics explain variation in hospital-specific mortality rates


DATADATA

Outcome• Mortality within 30-days of hospital admission

Patient characteristics• Admission severity index constructed on the basis of

34 patient attributes

Hospital characteristics• Urban/Rural• (Non academic)/(versus academic)• Number of beds


Why adjust for case mix?Why adjust for case mix?(patient characteristics)

• Irrespective of quality of care, older/sicker patients with multiple diseases have increased need of health care services and poorer health outcomes

• Without adjustment, physicians/hospitals who treat relatively more of these patients will appear to provide more expensive and lower quality care than those who see relatively younger/healthier patients

If there is inadequate case mix adjustment, evaluations will be unfair• But, need to avoid over adjusting


Case-mix AdjustmentCase-mix Adjustment

Compute hospital-specific, expected mortality by: 1. estimating a patient-level mortality model using all hospitals 2. averaging the model-produced probabilities for all

patients within a hospital

• Hospitals with “higher-than-expected” mortality rates can be flagged as institutions with potential quality problems, but need to account for uncertainty

• Need to be careful, if also adjusting for hospital characteristics– May adjust away the important signal


(as we know, very poor approach)

• Wrong SEs• Test-based


Hospital Profiling of Mortality Rates Hospital Profiling of Mortality Rates Acute Myocardial Infarction Patients Acute Myocardial Infarction Patients

(Normand et al. JAMA 1996, JASA 1997)


Hierarchical logistic regressionHierarchical logistic regression

I: Patient within-provider• Patient-level logistic regression model with random

intercept & slope

II: Between-provider• Hospital-specific random effects are regressed on

hospital-specific characteristics– Explicit regression


Admission severity indexAdmission severity index(Normand et al. 1997 JASA)


0 + 1(sevij – sevbar)

0

1

sevbar


we use bwe use b0i0i + b + b1i1i(...)(...)


Interpretation of parameters is different for the two levels

bb0i0i = = **0000 + N(..), etc. + N(..), etc.


RESULTSRESULTS

• Estimates of regression coefficients under three models:– Random intercept only– Random intercept and random slope– Random intercept, random slope, and hospital

covariates• Hospital performance measures


Normand et al. JASA 1997Normand et al. JASA 1997


30-DAY MORTALITY30-DAY MORTALITY 2.5th and 97.5th percentiles for a

patient of average admission severity

Exchangeable model • Random intercept and slope, no hospital covariates

log(odds): (-1.87,-1.56) probability,scale: (0.13, 0.17)

Covariate (non-exchangeable) model • Random intercept and slope, with hospital covariates• Patient treated in a large, urban academic hospital

log(odds): (-2.15,-1.45)probability scale: (0.10,0.19)


Effect of hospital characteristics Effect of hospital characteristics on baseline log-odds of 30-day mortalityon baseline log-odds of 30-day mortality

• For an average patient, rural hospitals have a higher odds ratio than urban hospitals

– Indicates between-hospital differences in the baseline mortality rates

– Case-mix adjustment may be able to remove some of this difference


Estimates of Stage-II Estimates of Stage-II regression coefficientsregression coefficients

InterceptsIntercepts


Effect of hospital characteristics on Effect of hospital characteristics on association between severity and mortalityassociation between severity and mortality

(slopes)

• The association between severity and mortality is modified by hospital size

• Medium-sized hospitals have smaller severity/mortality associations than large hospitals

– Indicates that the effect of clinical burden (patient severity) on mortality differs across hospitals


Estimates of Stage IIEstimates of Stage IIregression coefficientsregression coefficients

SlopesSlopes


Homework is on front tableHomework is on front table


Observed and risk-adjusted hospital mortality ratesObserved and risk-adjusted hospital mortality ratesUrban Hospitals

Histogram displays (observed – adjusted)


Observed and risk-adjusted hospital mortality ratesObserved and risk-adjusted hospital mortality ratesRural Hospitals

Histogram displays (observed – adjusted)

Substantial adjustment for severitySubstantial adjustment for severity


FINDINGSFINDINGS

• There is substantial adjustment for admission severity

• Generally, urban hospitals are adjusted less than rural

• There is less variability in observed or adjusted estimated rates for urban hospitals than for rural hospitals

Can you explain why?


Normand et al. JASA 1997Normand et al. JASA 1997


Average the probabilitiesAverage the probabilitiesDon’t average the covariatesDon’t average the covariates


k denotes a draw from the posterior


Plug in the average covariatePlug in the average covariate

Keep the hospital variationKeep the hospital variation



Comparing measures Comparing measures of hospital performanceof hospital performance

Three measures of hospital performance• Probability of a large difference between adjusted

and standardized mortality rates

• Probability of excess mortality for the average patient

• Z-score


Hospital Rankings: Normand et al 1997 JASA


Hospital RanksHospital Ranks

• There was moderate disagreement among the criteria for classifying hospitals as “aberrant”

• Nevertheless, hospital 1 is ranked worst

• It is rural, medium sized non-academic with an observed mortality rate of 35%, and adjusted rate of 28%


Adjusting for hospital-level charateristics Adjusting for hospital-level charateristics

Changes the comparison group in “as compared to what?”

• All hospitals (unadjusted at hospital level)

• Hospitals of a similar size, urbanicity, ...

• Percent of physicians who are board certified

• Hospitals with a similar death rate

Variance reduction and goodness of fit should not be the primary considerations

• “As compared to what?” must dominate


DiscussionDiscussion

• Profiling medical providers is multi-faced and data intensive process with substantial implications for health care practice, management, and policy

• Major issues include data quality and availability, choice of performance measures, formulation of statistical models (including adjustments), reporting results

• The ranking approaches and summaries used by Normand and colleagues are very good, but some improvement is possible


Multi-level models address key technical & Multi-level models address key technical & conceptual profiling issues, includingconceptual profiling issues, including

• Adjusting for patient severity

• Accounting for within-provider correlations

• Accounting for differential sample sizes at all levels

• Stabilize estimates

• Structure ranking and other, derived comparisons

Documents

Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance