Outcomes Research 8 sessions of 90 minutes

Assumes basic knowledge of multi-level modeling

4 lecturers with 4 main topics– Bindman: risk adjustment to judge Rx effectiveness– Smith-Bindman: evaluating test performance– Osmond : multi-level modeling of outcomes– Barron: propensity scores as a means to do risk adjustment

Homework 7 homework exercises

Grade based on homework and class participation/ no final

Some require ability to program in Stata

Homework due to Mintu Turakhia before the next class

epi211@mac.com for questions and to submit homework

Professional Conduct TICR has a prepared statement whose principles I support

Collaborating on homework is permitted; using answers from a student in a previous year is not

Submit your own homework using your own words

Be prepared for possibility of describing homework answers in class

Disclaimer Examples draw from cardiology literature

This doesn’t represent a clinical bias so much as it reflects that state of the research

Don’t hesitate to ask if something unclear about the clinical examples

Bring up examples important to you from your own field

Data for Outcomes Research

Andy Bindman MD

Department of Medicine, Epidemiology and Biostatistics

What is Outcomes Research

Studies of the quality of care as judged by patients’ outcomes

IOM domains of quality– Effectiveness– Safety– Timeliness– Equity– Efficiency– Patient-Centered

Donabedian Model of Quality

Structure Process Outcome

Donabedian Model of Quality

Structure Process OutcomeNumber of nurses per hospital bed

Physicians per capita

Donabedian Model of Quality

Structure Process OutcomeBeta blocker following MI

Immunizations

Donabedian Model of Quality

Structure Process OutcomeSurvival

Functional status

Satisfaction

Which is Best to Monitor Quality?

Structure - necessary but not sufficient

Process - many things we do/recommend don’t have proven health benefit

Outcomes - our ultimate responsibility but related to more than just the care we

provide

Predictors of Outcomes

Outcomes = intrinsic patient risk factors

treatment effectiveness

quality of care

random chance

Goals of Risk-Adjustment

Account for intrinsic patient risk factors before making inferences about effectiveness, efficiency, or quality of care

Minimize confounding bias due to nonrandom assignment of patients to different providers or systems of care

How is Risk Adjustment Done

On large datasets Uses measured differences in compared groups Model impact of measured differences between

groups on variables shown, known, or thought to be predictive of outcome so as to isolate effect of predictor variable of interest

When Risk-Adjustment May Be Inappropriate

Processes of care which virtually every patient should receive (e.g., immunizations, discharge instructions)

Adverse outcomes which virtually no patient should experience (e.g., incorrect amputation)

Nearly certain outcomes (e.g., death in a patient with prolonged CPR in the field)

Too few adverse outcomes per provider

When Risk-Adjustment May Be Unnecessary

If inclusion and exclusion criteria can adequately adjust for differences

If assignment of patients is random or quasi-random

When Risk-Adjustment May Be Impossible

If selection bias is an overwhelming problem If outcomes are missing or unknown for a large

proportion of the sample If risk factor data (predictors) are extremely

unreliable, invalid, or incomplete

Data Sources for Risk-Adjustment

Administrative data are collected primarily for a different purpose (billing), but are commonly used for risk-adjustment

Disease registries

Sources of Administrative Data

Federal Government– Medicare– VA

State Government– Medicaid (Medi-Cal)– Hospital Discharge Data

Private Insurance

Dataset Resources

http://www.epibiostat.ucsf.edu/courses/RoadmapK12/PublicDataSetResources/

http://base.google.com/base/search?a_n0=clinical+trials&a_y0=9&hl=en&gl=US

Advantages of Administrative Data

Computerized, inexpensive to obtain and use Uniform definitions Ongoing data monitoring and evaluation Diagnostic coding (ICD-9-CM) guidelines Opportunities for linkage (vital stat, cancer)

Administrative Hospital Discharge Data Admission Date • Race Discharge Date • Sex Type of Admission • Date of Birth Source of Admission • Zip Code Principal Diagnosis • Patient SSN Other Diagnoses • Total Charges Principal Procedure and Date • Expected Source of Payment Other Procedures and Dates Disposition of Patient External Cause of Injury Pre-hospital Care and Resuscitation (DNR)

Disadvantages of Administrative Data

No control over data collection process Missing key information about physiologic and

functional status Quality of diagnostic coding can vary across sites Non capture of out of plan/out of hospital/out of state

events

Linking Administrative Data

Strategy for enhancing number of predictor or outcomes variables

Deterministic linkage dependent on reliable shared identifiers such as social security numbers in both datasets

Probabilistic matching of less specific variables (age, sex, race, date of birth, etc)

Some Routinely Available Data Linkages

California hospital discharge data and vital statistics– Example: 30 day mortality following AMI

SEER -Medicare– Example: utilization patterns for those with breast cancer

National Health Interview Survey-Medical Expenditure Panel Survey– Example: health care costs for those with self-reported chronic

conditions

California Hospital Discharge Data and Medicaid Eligibility Files

Creates a continuous monthly record of an individual’s pattern of Medicaid enrollment

Discharge data captures all hospitalizations regardless of whether in or out of Medicaid

Have found a 3 fold increase in hospitalizations for ambulatory care sensitive conditions for those with interrupted Medicaid coverage

Health Plans/Delivery Systems

Health insurance claims– Inpatient, outpatient, pharmacy, diagnostics, etc

Electronic Medical Records– VA– Kaiser– SF Dept of Public Health (THREDS)

THREDS

~120,000 patients per year seen in DPH clinics/SFGH

Data begin in 1996 and updated daily Includes demographics, insurance status,visit hx,

diagnostic codes, tests ordered and results, pharmacy, link to death registry

http://ctsi.ucsf.edu/bi/threds.php

Disease Registries Attempt to capture all or large sample of the cases of a

specified condition Often include more clinical information than

administrative datasets Many of these can support assessments of survival

beyond acute period May require permission/approved protocol to access all

or some of the data

Example Registries UNOS:national registry of patients with end stage renal

disease

SEER Cancer Registry

Coronary Artery Bypass Graft Surgery: California Office of Statewide Health Planning and Development

Doing Your Own Risk-Adjustment vs. Using an Existing Product

Is an existing product available or affordable? Would an existing product meet my needs?

- Developed on similar patient population

- Applied previously to the same condition or procedure

- Data requirements match availability

- Conceptual framework is plausible and appropriate

- Known validity

Conditions Favoring Use of an Existing Product

Need to study multiple diverse conditions or procedures

Limited analytic resources Need to benchmark performance using an external

norm Need to compare performance with other providers

using the same product Focus on resource utilization, possibly mortality

A Quick Survey of Existing ProductsHospital/General Inpatient

APR-DRGs (3M) Disease Staging (SysteMetrics/MEDSTAT) Patient Management Categories (PRI) RAMI/RACI/RARI (HCIA) Atlas/MedisGroups (MediQual) Cleveland Health Quality Choice Public domain (MMPS, CHOP, CSRS, etc.)

A Quick Survey of Existing ProductsIntensive Care

APACHE MPM SAPS PRISM

A Quick Survey of Existing ProductsOutpatient Care

Resource-Based Relative Value Scale (RBRVS) Ambulatory Patient Groups (APGs) Physician Care Groups (PCGs) Ambulatory Care Groups (ACGs)

How Do Commercial Risk-Adjustment Tools Perform

Better than age/sex to predict health care use/death Better retrospectively (~30-50% of variation) than

prospectively (~10-20% of variation) Lack of agreement among measures More than 20% of in-patients assigned very different

severity scores depending on which tool was used (Iezzoni, Ann Intern Med, 1995)

Co-Morbidity or Severity?

Are patients at risk for an outcome because they have multiple conditions (co-morbidities), a more severe version of a disease (disease stage) or both?

Before adjusting for co-morbidity and or severity consider whether either is a complication of treatment (or non treatment) rather than an independent health characteristic of the patient

Summary Risk adjustment is a multivariate modeling technique

designed to control for patient characteristics so that judgments can be made about the quality of care

Risk adjustment requires large datasets such as administrative datasets or disease registries

Commercial risk adjustment products exist for patients in different health care settings

There are many reasons why one might choose to develop a risk adjustment model - we will talk about how to do this next week!