Upload
logan-williams
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
HSAG Performance Improvement HSAG Performance Improvement Projects Projects Using Data to Develop Interventions Using Data to Develop Interventions
and Statistical Testing to Evaluate Resultsand Statistical Testing to Evaluate Results
Breakout Session #1Breakout Session #1Florida EQR Quarterly MeetingFlorida EQR Quarterly Meeting
June 18, 2008June 18, 2008
Presented by:Donald Grostic, MSAssociate Director, Research and Analysis TeamYolanda Strozier, MBAEQRO Project Manager
Intervention Cycle Framework
Implement
Evaluate
Plan
Steps for Intervention
Identify
Data Mining andCausal/Barrier Analysis
Three Tips
Statistical Testing andLinking Intervention to Outcomes
Data Collection (CMS Protocol Activity VI)
CMSActivityVII,VIII
CMS ActivityVIII
CMSActivityVII, VIII
CMS ActivityVII,VIII, IX, X
What does the intervention cycle have to do with CMS PIP Activities?
Identify Plan Implement Evaluate
Activity 7Assess the
Improvement Strategy
☑ ☑ ☑ ☑Activity 8Review Data Analysis &
Interpretation of Results
☑ ☑ ☑
Activity 9Reported
Improvement is Real?
☑Activity 10
Sustained Improvement?
☑
Identify
The ‘Identify’ Stage
Data Mining
• What is data mining?
Answer:
Data mining is the process of sorting through large amounts of data and picking out relevant information.
Data Mining (continued)
• What is data mining used for?
Answers:
Data mining is the statistical and logical analysis of large sets of data, looking for patterns of care, or service delivery that can aid decision making.
To identify and determine areas of non-compliance that will be analyzed during the causal/barrier analysis.
Data Mining vs. Data Analysis Plan
• How does data mining differ from a data analysis plan?
Answer:
A data analysis plan includes calculating and comparing overall indicator rates between measurement periods using statistical testing.
Data mining will include analysis that goes beyond just calculating and comparing indicator rates between measurements.
Data Mining–Example
PIP topic (clinical):
Follow-up after acute care inpatient hospitalization.
Indicator:
The percentage of members with follow-up within 7 days
after acute care discharge for a mental health diagnosis.
Data Mining Example Step One
• Group the population or sample.
First, group members by county or ZIP code. For our example, the population breaks into three counties:
County A, County B, and County C.
Data Mining Example Step Two
• Calculate compliance and noncompliance for each county.
The percentage compliant and noncompliant by county are presented in the following table.
Question: Which county should you data mine further?
Percentage Compliant
Percentage Non-Compliant
County A 65% 35%
County B 35% 65%
County C 20% 80%
Data Mining Example Step Three• Identify groups where the majority of members are
noncompliant.
Answer: First we need to know how many members of the population are
in each county. Selecting County B will have the greatest effect on the
compliance rate because it has the majority of the population and the second lowest compliance rate.
Percentage Compliant
Percentage Non-Compliant
Number of Members
County A 65% 35% 80
County B 35% 65% 220
County C 20% 80% 20
Data Mining Example Next Steps
• Now that you have identified County B, what should you do next?
Answer: Continue the process of grouping and selecting to
find the group that will have the greatest effect on compliance.
For County B, you may consider grouping the data by PCP or facility next.
Data Mining Caution!Words of caution:
Grouping and selecting can be taken to a point where the groups selected may be too small to make an impact.
Always keep in mind the number of members affected in the selected group relative to the total population.
If there is difficulty identifying noncompliant groups or non-compliance is equally distributed among groups, you may be dealing with a systemwide issue.
Please keep in mind that data mining is a dynamic, iterative process that takes practice.
The more you data mine the better you will become at selecting groups that yield the best effect on rates.
Questions and Answers
What is a Causal/Barrier Analysis?
• A causal/barrier analysis is:– A systematic process for identifying the problem.
– A method for determining what causes the barriers.
– A way to identify what improvement opportunities are available.
• Causal/barrier analysis has also been called:– Root cause analysis
How do I perform a causal/barrier analysis?
Determine why an event or condition occurs.
1. What is the problem? - Define the problem and explain why it’s a
concern.
2. Determine the significance of the problem. - Look at the data and see how the problem
impacts your members and/or health plan.
How do I perform a causal/barrier analysis?(cont.)
3. Identify the causes/barriers. - Conduct analysis of chart review data,
surveys, focus groups. - Brainstorming at quality improvement
committee meetings. - Literature review.
4. Develop/implement interventions based on identified barriers.
Causal/Barrier Methods and Tools
• Methods:– Quality improvement committees– Develop an internal task force– Focus groups– Consensus expert panels
• Tools:– Fishbone– Control chart– Flow chart (process mapping)– Barrier/intervention table
Questions and Answers
The ‘Plan’ Stage
Plan
Identify
A Physical Health Example
Low Well VisitRates
Data Providers
Members Systems
Demographics
Transportation
Compliance
Knowledge
MedicalRecords
Paper EHRs
BillingWell vs. Sick
Outreach
Knowledge
Compliance
Data accuracy
Data completeness
Demographicchanges
What questions could be asked to drill down these causes?What data are needed to identify the most crucial cause?
A Mental Health Example
• Discharge planning– Client– Communication– Transportation– Community involvement
• No follow-up appointment set at time of discharge• Time lag/claim data• Not client focused• Provider access• Culture change• Demographic information
What questions could be asked to drill down these causes?What data are needed to identify the most crucial cause?
Interventions Checklist
Analyze barriers (root causes)Choose and understand target audienceSelect interventions based on cost/benefit Implement interventionsTrack intermediate results (optional)RemeasureModify interventions as needed
Questions and Answers
The ‘Implement’ Stage
Implement
Plan
Identify
The ‘Implement’ Stage
Three tips:
1. Observe and document whether the intervention is implemented as intended
2. Note any lesson(s) learned
3. Document any change(s) that may threaten the results between measurement periods
– Methodology (e.g. definition of indicators, sampling)
– Circumstances (e.g. merger, population, provider)
Questions and Answers
The ‘Evaluate’ Stage: Statistical Testing
Evaluate
Plan
Identify
Statistical Significance Testing
• What is statistical testing and why do we use it?
Answers: Statistical testing is calculating specific test statistics
and associated p values to determine if an observed difference is a true difference and not due to chance alone.
The CMS Protocols require that statistical testing be used to prove that any improvement in rates is real.
Without statistical testing, a PIP would not meet the CMS Protocols.
Statistical Significance Testing
• What type of statistical testing is appropriate for my PIP?
Answer: Fisher’s Exact Test or Chi-square test for rates or proportions. T test for means would be the appropriate statistical testing.
Statistical Significance Testing
• What type of statistical testing is appropriate for this indicator?
Indicator A: The percentage of members with follow-up within
7 days after acute care discharge for a mental health diagnosis.
Answer: Fisher’s Exact Test or Chi-square test for rates or proportions.
Statistical Significance Testing
• What is the difference between Fisher’s Exact Test and a Chi-square test?
Answer: Fisher’s Exact Test will provide the exact p value while the Chi-
square test is an approximation of the p value. As your numerators and denominators increase in size, the
Chi-square test and Fisher’s Exact Test produce the same p value.
If in doubt about which test to use, use Fisher’s Exact Test.
Statistical Significance Testing
• What type of statistical testing is appropriate for this indicator?
Indicator B: The average response from a member satisfaction
survey where answers range from 1=satisfied to 5=dissatisfied.
Answer:
T test for means would be the appropriate statistical testing.
Statistical Significance Testing
• How do I report statistical significance testing results?
Answer: When using a Fisher’s Exact Test, Chi-square test or a t test, report the test used, its associated p value along with each indicator, and its numerator and denominator in tabular form.
Statistical Significance Testing
Time Periods
Measurement Periods Numerator Denominator Rate or Results
Industry Benchmark
Statistical Testing and Significance
CY 2003 Baseline 20 41 48.8% 60% N/A
CY 2004 Remeasurement 1 27 51 52.9% 60% Fisher’s Exact Test
P value = 0.8340
Chi-square test
P value = 0.8517
NOT SIGNIFICANT AT THE 95% CONFIDENCE LEVEL
Indicator A: The percentage of members with follow-up within 7 days after acute care discharge for a mental health diagnosis.
Statistical Significance Testing
Time Periods
Measurement Periods Numerator Denominator Rate or Results
Industry Benchmark
Statistical Testing and Significance
CY 2008 Baseline 253 100 2.53 N/A STD DEV = 1.298
CY 2009 Remeasurement 1 371 113 3.28 N/A STD DEV = 1.561
T-test
P value = 0.0002
SIGNIFICANT AT THE 95% CONFIDENCE LEVEL
Indicator B: The average response from a member satisfaction survey where answers range from 1=satisfied to 5=dissatisfied.
Statistical Significance Testing
• If I use the entire population for my study, do I still have to do statistical significance testing?
Answer: Yes. It is appropriate to do statistical
testing on the entire eligible population.
Reasons for Statistical Significance Testing on Entire Populations
• CMS is interested in performance over time. • The population will continuously change over time. • The members who are studied in one year may or may not
appear in the following years. • A population that is selected at one point in time is a sample
from the true population that contains all members. • The entire eligible population for a measure in one year is a
sample population drawn from a universe of “all years” or “all populations” that could be selected.
• CMS has approved statistical testing on populations.
Questions and Answers
The ‘Evaluate’ Stage: Linking Intervention to Outcomes
Identify
PlanImplement
EvaluateIm
prov
ed?
Yes
No
Revise
Standardize
The ‘Evaluate’ Stage: Linking Intervention to Outcomes
Threats to internal/external validity: Any environmental, organizational, methodological changes between measurement periods?
No Yes
Outcome: Improved • Intervention seems to be effective• Consider standardizing the intervention to subsequent measurement periods
• Cannot ascertain if the improvement really is due to intervention• Investigate the relationship between the change circumstances and the outcomesOutcome: No Change or
Worsens•Intervention does not seem to be effective• Consider revising the intervention to subsequent measurement periods
The ‘Evaluate’ Stage: Linking Intervention to Outcomes
QI Implement
EvaluateIm
prov
ed?
No
ReviseY
es
Standardize Plan
Identify
Questions and Answers