Public Health Performance Public Health Performance MeasurementMeasurement

Moving from Version 1.0 to Moving from Version 1.0 to 2.02.0

AcademyHealth Annual MeetingAcademyHealth Annual MeetingBoston, MABoston, MA

June 26, 2005June 26, 2005

University of KentuckyUniversity of Kentucky Team Members Team Members

F. Douglas Scutchfield, MD, P.I.F. Douglas Scutchfield, MD, P.I. Evelyn A. Knight, PhDEvelyn A. Knight, PhD Ann V. Kelly, MHAAnn V. Kelly, MHA Ilie Puiu Vasilescu, PhD (UVa-Ilie Puiu Vasilescu, PhD (UVa-

Wise)Wise) Michelyn Bhandari, MPHMichelyn Bhandari, MPH

Goals of the National Public Health Goals of the National Public Health Performance Standards ProgramPerformance Standards Program

(NPHPSP)(NPHPSP) Provide performance standards for public health Provide performance standards for public health

systemssystems Improve quality and accountability of public Improve quality and accountability of public

health practicehealth practice Conduct systematic collection and analysis of Conduct systematic collection and analysis of

performance dataperformance data Develop a science-base for public health Develop a science-base for public health

practice improvementpractice improvement

Aims of the Assessment Aims of the Assessment InstrumentsInstruments

Provide a NPHPSP performance Provide a NPHPSP performance report to the public health agency report to the public health agency and the state or local system.and the state or local system.

Provide baseline, feedback and Provide baseline, feedback and guidance to the public health guidance to the public health system for continuous performance system for continuous performance improvement. improvement.

Four Concepts Applied in Four Concepts Applied in NPHPSNPHPS

1. 1. Based on the Based on the 10 10 Essential Essential Public Health Services Public Health Services (EPHS)(EPHS)

2. Focus on the overall public 2. Focus on the overall public health health systemsystem

3. Describe an 3. Describe an optimaloptimal level of level of performanceperformance

4. Support a 4. Support a processprocess of quality of quality improvementimprovement

Goals of UKY’s Instrument Goals of UKY’s Instrument ResearchResearch

(Local and State PHPA)(Local and State PHPA) Decrease the burden on users due to Decrease the burden on users due to

instrument characteristics.instrument characteristics. Assure content supports the goals of Assure content supports the goals of

the performance assessment the performance assessment process. process.

Increase the usability of the results Increase the usability of the results for performance improvement.for performance improvement.

Research PhasesResearch Phases Preliminary interviews-Preliminary interviews--determine areas of -determine areas of

need for instrument improvement—state need for instrument improvement—state and local instruments.and local instruments.

Psychometric analysis-Psychometric analysis--identify items that -identify items that might be changed or eliminated in future might be changed or eliminated in future versions—local instrument only. versions—local instrument only.

Discussion groups-Discussion groups--identify specific -identify specific changes in question inclusion and/or changes in question inclusion and/or wording which would improve instrument wording which would improve instrument usability and usefulness—state and local usability and usefulness—state and local instruments. instruments.

Preliminary Interviews—Preliminary Interviews—ResultsResults

Directions clear; some questions clear, some notDirections clear; some questions clear, some not Understandable by public health--but not partnersUnderstandable by public health--but not partners Too long, instrument fatigue--but details importantToo long, instrument fatigue--but details important Rating scale clear—but doesn’t correspond with Rating scale clear—but doesn’t correspond with

reportreport Rating methodology varies—consensus, vote, etc.Rating methodology varies—consensus, vote, etc. Subjective scores—depend on who is presentSubjective scores—depend on who is present Costly to gather information needed to answer the Costly to gather information needed to answer the

questionsquestions

Indicator—Level 2

ModelStandard

Measure—Level 3(First-tier stem)

Essential Service—Level 1

Instrument FormatInstrument Format

Measure—Level 4

Measure—Level 5

SummaryQuestions--Not analyzed

Measure—Level 4(Second-tier stem)

Numbers of Indicators & Questions(Local Instrument)

EPHS Indicators

Level 3

stem Qs

Level4 Qs

Level5 Qs

Total

1. Monitor Health Status 3 21 51 24 96

2. Diagnose & Investigate 4 19 51 9 79

3. Inform, Educate & Empower

2 7 33 15 55

4. Mobilize Community Partnerships

2 7 38 4 49

5. Develop Policies & Plans 4 12 25 36 73

6. Enforce Laws & Regulations

3 11 26 0 37

7. Link People to Health Services

3 9 45 0 54

8. Assure Competent Workforce

4 16 41 15 72

9. Evaluate 3 13 38 25 76

10. Research 3 11 23 0 34

TOTAL 31 126 371 128 625

Performance Scoring—Levels 3 & 2Performance Scoring—Levels 3 & 2

Level Level 33 stem scores are weighted scores stem scores are weighted scores based on stem and level 4 & 5 sub-questions.based on stem and level 4 & 5 sub-questions.

Stem question (3-digit) Stem question (3-digit)

Sub-questions (4-digit)Sub-questions (4-digit)

Sub-sub-questions (5 digit)Sub-sub-questions (5 digit) Level Level 22 scores (model standards) are average scores (model standards) are average

of Level 3 scores--yields 31 Indicator Scores. of Level 3 scores--yields 31 Indicator Scores.

Performance Scoring—Performance Scoring—Level 1 & OverallLevel 1 & Overall

Indicator Scores are averaged to Indicator Scores are averaged to obtain 10 EPHS scores—Level 1.obtain 10 EPHS scores—Level 1.

10 EPHS scores are averaged to 10 EPHS scores are averaged to obtain one obtain one overalloverall Public Health Public Health Performance Score.Performance Score.

Psychometric Analysis of Psychometric Analysis of Local InstrumentLocal Instrument

Determine Cronbach’sDetermine Cronbach’s internal internal reliabilityreliability

Determine Item-Total Correlations (ITC)Determine Item-Total Correlations (ITC) Test effect on reliability of removing Test effect on reliability of removing

specific questions or tiers of questionsspecific questions or tiers of questions Use results to frame discussion group Use results to frame discussion group

interviewsinterviews

EPHS Overall

Chronbach’s if EPHS Removed

Item-total Correlation Coefficient

1. Monitor Health Status 0.908 0.660

2. Diagnose & Investigate 0.918* 0.448

3. Inform, Educate, Empower 0.906 0.697

4. Mobilize Partnerships 0.909 0.657

5. Develop Policies & Plans 0.897 0.840

6. Enforce Laws & Regulations 0.909 0.645

7. Link People to Health Services

0.9050.703

8. Assure Competent Workforce

0.8960.717

9. Evaluate 0.890 0.797

10 Research 0.896 0.719*Reliability of group to total increases slightly if EPHS #2 (item) is removed. Reliability is slightly higher with only 9 EPHS (0.918), than with all 10 EPHS (0.915).

For Overall Score = 0.915 (very reliable)

Table 1. Internal Reliability of Overall Performance Score Based on 10 EPHS Scores (n=228)

EPHS #Indicators

EPHSChronbach’s

1. Monitor Health Status

30.651

2. Diagnose & Investigate

40.753

3. Inform, Educate, Empower 2 0.832

4. Mobilize Partnerships 2 0.778

5. Develop Policies & Plans 4 0.748

6. Enforce Laws & Regulations

30.651

7. Link People to Hlth Services 3 0.620**

8. Assure Competent Workforce

40.695

9. Evaluate 3 0.847

10 Research 3 0.787

Table 2: Internal Reliability of 10 EPHS Scores based on Indicator Scores (n=228)

Summary of Internal Summary of Internal Reliability Reliability

Overall & EPHSOverall & EPHS Instrument has very high reliability Instrument has very high reliability

overall—measuring the PH constructoverall—measuring the PH construct EPHS-level reliability variesEPHS-level reliability varies

– EPHS 3,9EPHS 3,9 Very reliableVery reliable– EPHS 2,4,5,10EPHS 2,4,5,10 Acceptable reliabilityAcceptable reliability– EPHS 8EPHS 8 Marginal reliabilityMarginal reliability– EPHS 1,6,7EPHS 1,6,7 Not reliableNot reliable

Table 3. Internal Reliability—Indicators based on Level 3 First-Tier Stem Question

Scores

EPHS Ind #1

Ind #2

Ind #3

Ind #4

1. Monitor Health Status** 0.950 0.802 0.593 --

2. Diagnose & Investigate 0.689 0.730 0.739 0.766

3. Inform, Educate, Empower** 0.767 0.687 -- --

4. Mobilize Partnerships 0.805 0.825 -- --

5. Develop Policies & Plans 0.497 0.738 0.818 0.826

6. Enforce Laws & Regulations** 0.568 0.825 0.796 --

7. Link People to Health Services**

/1 0.852 0.801 --

8. Assure Competent Workforce**

0.877 0.708 0.693 0.886

9. Evaluate** 0.919 0.861 0.798 --

10 Research 0.788 0.839 0.790 --

Summary of Internal Summary of Internal Reliability Reliability

of 31 Indicatorsof 31 Indicators 1414 indicators very reliable indicators very reliable >>0.80.8 10 10 indicators of acceptable indicators of acceptable

reliabilityreliability 0.7 0.7 to 0.8to 0.8– 3 3 become very reliable if one become very reliable if one

stem question is removedstem question is removed

66 indicators not reliable indicators not reliable <0.7<0.7

Psychometrics of Current and Psychometrics of Current and Simulated Instrument Versions Simulated Instrument Versions

(V.1-V.4)(V.1-V.4)

Current Local Assessment InstrumentCurrent Local Assessment Instrument– V.1V.1—All questions as scored by CDC—All questions as scored by CDC

EliminateEliminate scores of drill-down details scores of drill-down details– V.2V.2—No level 5 details—use —No level 5 details—use onlyonly level 3&4 level 3&4 – V.3V.3—No level 4&5 details—use —No level 4&5 details—use onlyonly level 3 level 3

stemstem– V.4V.4—No questions with high or ns —No questions with high or ns

correlation correlation to stems at to stems at levels 4&5levels 4&5

Table 4. Comparison of Reliability across the Four Versions (v.1-4) for

10 EPHS

Table 5. Internal Reliability—10 EPHS Scores based on Indicator ScoresV.1 and simulated V.2-V.4

EPHS

V.1

V.2

V.3

V.4

1. Monitor Health Status 0.651 0.645 0.553 0.649

2. Diagnose & Investigate 0.753 0.750 0.732 0.754

3. Inform, Educate, Empower 0.832 0.830 0.699 0.823

4. Mobilize Community Partnerships

0.778 0.769 0.731 0.768

5. Develop Policies & Plans 0.748 0.746 0.723 0.760

6. Enforce Laws & Regulations 0.651 0.651 0.620 0.641

7. Link People to Health Services 0.620 0.620 0.563 0.610

8. Assure Competent Workforce 0.695 0.690 0.621 0.690

9. Evaluate 0.847 0.844 0.828 0.835

10 Research 0.736 0.733 0.682 0.732

Moving from V1.0 to V2.0Moving from V1.0 to V2.0

Instrument constructionInstrument construction Instrument contentInstrument content Instrument contextInstrument context Instrument consensusInstrument consensus

Instrument ConstructionInstrument Construction

ClarityClarity TimeTime ComplexityComplexity SubjectivitySubjectivity

Instrument ContentInstrument Content

Examine unreliable EPHS and Examine unreliable EPHS and subscalessubscales

Determine key contentDetermine key content Fit content into a logic model with Fit content into a logic model with

inputs—outputs--outcomesinputs—outputs--outcomes

Instrument ContextInstrument Context

Large vs small systemsLarge vs small systems Centralized vs decentralizedCentralized vs decentralized Urban vs ruralUrban vs rural

Instrument ConsensusInstrument Consensus

Agreed on definitions of Agreed on definitions of PH practicePH practice

PurposePurpose ProcessProcess StrengthsStrengths WeaknessesWeaknesses

NPHPSP Instrument V2.0 shouldNPHPSP Instrument V2.0 should::

Build on current instrument and Build on current instrument and lessons learned.lessons learned.

Specify how users will validate Specify how users will validate their results.their results.

Recognize and respond to Recognize and respond to agency--system issues. agency--system issues.

Agree on purpose.Agree on purpose. Use for purpose for which designed.Use for purpose for which designed.