Telling the Performance StoryTelling the Performance Storyof the Real Choice Systems of the Real Choice Systems

Change GrantsChange Grants

Melissa HulbertMelissa Hulbert

Division of Advocacy and Special InitiativesDivision of Advocacy and Special Initiatives

Disabled and Elderly Health Programs GroupDisabled and Elderly Health Programs Group

Centers for Medicaid and State OperationsCenters for Medicaid and State Operations

22

Telling the Performance StoryTelling the Performance Storyof the Real Choice Systems of the Real Choice Systems

Change GrantsChange Grants

Alert you to important changes CMS is making in Alert you to important changes CMS is making in the reporting system for FY04 RCSC grants.the reporting system for FY04 RCSC grants.

First in a series of communications between CMS First in a series of communications between CMS and the FY04 grantees. Today: and the FY04 grantees. Today: – Overview of the “why and how of the proposed Overview of the “why and how of the proposed

changes”;changes”;– Next Steps.Next Steps.– Introduction to the Logic Model Framework.Introduction to the Logic Model Framework.

33

Need for ChangeNeed for Change RCSC Grants for Community Living: Investment by RCSC Grants for Community Living: Investment by

Congress, CMS, and state partners in programs that Congress, CMS, and state partners in programs that will improve the lives of people with long-term illness will improve the lives of people with long-term illness and/or disabilities. and/or disabilities.

Think strategically about how to apply pertinent Think strategically about how to apply pertinent lessons from grantee accomplishments and lessons from grantee accomplishments and challenges to the formation of state and national challenges to the formation of state and national agendas for the provision of long-term supports in the agendas for the provision of long-term supports in the future.future.

Demonstrate, in a measurable way, the value of the Demonstrate, in a measurable way, the value of the investments we have made. We must better “tell the investments we have made. We must better “tell the performance story” of the impact of the RCSC grants. performance story” of the impact of the RCSC grants.

44

Need for ChangeNeed for Change Adoption of various “Performance Adoption of various “Performance

Measurement” techniques, i.e. GPRA, Measurement” techniques, i.e. GPRA, PART, etc.PART, etc.

CMS: Logic modeling as a tool for strategic CMS: Logic modeling as a tool for strategic planning and program evaluation. planning and program evaluation. – What policies/programs are making What policies/programs are making

improvements in the lives of individuals?improvements in the lives of individuals?– How outcomes should be measured?How outcomes should be measured?– How these initiatives and their impact should be How these initiatives and their impact should be

described?described?

55

Existing ChallengesExisting Challenges

Grant Diversity:Grant Diversity:– Scope: nature of the expected changeScope: nature of the expected change– Target populationTarget population

Each grantee describes/characterizes the Each grantee describes/characterizes the components of their grant differentlycomponents of their grant differently

Existing reports tend to lack the “so what”; what Existing reports tend to lack the “so what”; what difference is/could the grant making in the difference is/could the grant making in the operation of the system or in lives of beneficiaries. operation of the system or in lives of beneficiaries.

66

What does this mean for you?What does this mean for you? Purpose of Today’s web-cast:Purpose of Today’s web-cast:

– Notify you of upcoming changes to the web-Notify you of upcoming changes to the web-based reporting systembased reporting system

– Introduce grantees to the context, terminology Introduce grantees to the context, terminology and concepts of the refined web-based and concepts of the refined web-based reporting system.reporting system.

– Provide an initial introduction to the logic model Provide an initial introduction to the logic model approach and encourage use of this approach approach and encourage use of this approach in grant project planning and evaluation. in grant project planning and evaluation.

77

What does this mean for you?What does this mean for you?

Refinements to the RCSC Reporting System Refinements to the RCSC Reporting System based upon logic modeling framework.based upon logic modeling framework.

Reordered/replaced existing reporting categories, Reordered/replaced existing reporting categories, such as Goals, Activities, and Accomplishments:such as Goals, Activities, and Accomplishments:– Establish a standard and more informative definition of Establish a standard and more informative definition of

reporting categories;reporting categories;– Provide a better description of each grant’s programs Provide a better description of each grant’s programs

and expected outcomes.and expected outcomes.– Create a sequence of related events – connect the need Create a sequence of related events – connect the need

for project planning with the grant s desired results. for project planning with the grant s desired results. – Create measureable, quantitative measures for each Create measureable, quantitative measures for each

grant and in some cases across grants.grant and in some cases across grants.

88

What does this mean for you?What does this mean for you? New Reporting Framework and Data Fields:New Reporting Framework and Data Fields:

– Goal/ImpactGoal/Impact– Objectives/ActivitiesObjectives/Activities– OutputsOutputs– Outcomes: Intermediate, System, and BeneficiaryOutcomes: Intermediate, System, and Beneficiary

With support from RTI, for each grant: With support from RTI, for each grant: – Incorporated framework into the web-based systemIncorporated framework into the web-based system– Extracted information from grant applications and semi-Extracted information from grant applications and semi-

annual report and developed proposed text for each annual report and developed proposed text for each new data field. new data field.

99

What does this mean for you? What does this mean for you? AssumptionsAssumptions

Each grant will directly achieve, at a minimum:Each grant will directly achieve, at a minimum:– Goal and Impact statements;Goal and Impact statements;– Objectives and Activities; andObjectives and Activities; and– Outputs.Outputs.

Given difference in scope, recognize that Given difference in scope, recognize that “outcomes” for some grantees will be outside the “outcomes” for some grantees will be outside the grant’s “sphere of influence”.grant’s “sphere of influence”.

However, articulation and measure of possible and However, articulation and measure of possible and intended impacts are still important to telling our intended impacts are still important to telling our performance story.performance story.

1010

Next StepsNext Steps September 9September 9thth: “Report Cross-Walk” disseminated to each : “Report Cross-Walk” disseminated to each

granteegrantee– Description of new framework;Description of new framework;– Summary of new data fields and definitions;Summary of new data fields and definitions;– Grant-specific proposed text for the new fields developed by Grant-specific proposed text for the new fields developed by

CMS/RTI.CMS/RTI.– Proposed Text will be automatically uploaded to the web-based Proposed Text will be automatically uploaded to the web-based

report. report.

September 27th: Grantee Access to Web-based reporting September 27th: Grantee Access to Web-based reporting system system

Conference calls to “walk through” new web-based report.Conference calls to “walk through” new web-based report.– September 28September 28thth at 10:00 EST at 10:00 EST – September 29September 29thth at 6:00 EST. at 6:00 EST.

1111

Next StepsNext Steps

Annual Report: Due November 1 Annual Report: Due November 1

RCSC Technical Assistance Provider: RCSC Technical Assistance Provider: – First responders to questions; and First responders to questions; and – A resource to review your grant-specific cross-walk and A resource to review your grant-specific cross-walk and

determine any necessary refinements to grant activities determine any necessary refinements to grant activities and/or proposed cross-walk text.and/or proposed cross-walk text.

CMS Project Officer: Request alternative text for CMS Project Officer: Request alternative text for web-based reporting fields. web-based reporting fields.

Logic ModelingLogic Modeling::A Tool to Guide Program Design & A Tool to Guide Program Design & EvaluationEvaluation

John A. McLaughlinJohn A. McLaughlin

MACGROUPX@AOL.COMMACGROUPX@AOL.COM

1313

My Aim TodayMy Aim Today

Orient you to a new way think about Orient you to a new way think about conceptualizing and telling the performance conceptualizing and telling the performance story of your project.story of your project.

Provide a simple Provide a simple tooltool for creating a for creating a functional picture of functional picture of how your program how your program works to achieve its aimsworks to achieve its aims

Offer some helpful hints for framing a useful Offer some helpful hints for framing a useful evaluation strategy for your program.evaluation strategy for your program.

1414

Orientations for Performance Measurement Orientations for Performance Measurement & Evaluation& Evaluation

PERFORMANCE MEASUREMENTPERFORMANCE MEASUREMENT– Accountability, Accountability, descriptiondescription

What objectives/outcomes have been accomplished at what What objectives/outcomes have been accomplished at what levels?levels?

PROGRAM EVALUATIONPROGRAM EVALUATION

– LearningLearning, Program Improvement, Program Improvement, , DefenseDefense What factors, internally and/or externally influenced my What factors, internally and/or externally influenced my

performance? (performance? (RetrospectiveRetrospective)) What effect will this level of performance have on future What effect will this level of performance have on future

performance if I don’t do something? (performance if I don’t do something? (ProspectiveProspective)) What roles (+/-) did context play in my performance?What roles (+/-) did context play in my performance?

1515

Benefits of Logic Modeling Benefits of Logic Modeling

Communicates the performance story of the Communicates the performance story of the program or project.program or project.

Focuses attention on the most important Focuses attention on the most important connections between actions and results.connections between actions and results.

Builds a common understanding among Builds a common understanding among staff and with stakeholders.staff and with stakeholders.

Helps staff “manage for results” and informs Helps staff “manage for results” and informs program design.program design.

Finds “gaps” in the logic of a program and Finds “gaps” in the logic of a program and works to resolve them.works to resolve them.

1616

Logic Models as RecipesLogic Models as Recipes

Recipes have three essential components:Recipes have three essential components:– A description of the entree to be produced; A description of the entree to be produced; – A list of specific ingredients according to A list of specific ingredients according to

specific measures; specific measures; – and specific steps to put the ingredients and specific steps to put the ingredients

together.together. A good cook follows the recipe – managers A good cook follows the recipe – managers

would do well to create and follow their would do well to create and follow their recipe for success!recipe for success!

1717

7651 3 42

CustomersActivities Outputs1rst Order Outcome

2nd Order Outcome

ImpactResources

Program’s Sphere of Influence

Contextual Influences

Simple Logic Model

HOW WHY

1818

Elements of the Logic ModelElements of the Logic Model Resources/Inputs:Resources/Inputs: Programmatic investments available Programmatic investments available

to support the program.to support the program. Objectives/Activities:Objectives/Activities: Things you do– activities you Things you do– activities you

plan to conduct in your program.plan to conduct in your program. Outputs:Outputs: Product or service delivery/implementation Product or service delivery/implementation

targets you aim to produce.targets you aim to produce. Customer: Customer: User of the products/services. Target User of the products/services. Target

audience the program is designed to reach.audience the program is designed to reach. Outcomes:Outcomes: Changes or benefits resulting from activities Changes or benefits resulting from activities

and outputs.and outputs. Outcome StructureOutcome Structure

– Short-term (Short-term (K, S, AK, S, A) – Changes in learning, ) – Changes in learning, knowledge, attitude, skills, understandingknowledge, attitude, skills, understanding

– Intermediate (Intermediate (BehaviorBehavior) – Changes in behavior, ) – Changes in behavior, practice or decisions practice or decisions

– Long-term (Long-term (ConditionCondition) – Changes in condition ) – Changes in condition External Influences:External Influences: Factors that will influence change Factors that will influence change

in the affected community.in the affected community.

1919

Outputs & OutcomesOutputs & Outcomes

An annual conference disseminates An annual conference disseminates the latest forage research.the latest forage research.

Low-income families are better able Low-income families are better able to manage their resources.to manage their resources.

Program staff teach financial Program staff teach financial management skills to low-income management skills to low-income families. families.

Community volunteers have Community volunteers have knowledge and skill to work knowledge and skill to work effectively with at-risk youth. effectively with at-risk youth.

The camp experience provides The camp experience provides leadership development leadership development opportunities for 4-H youth. opportunities for 4-H youth.

Forage producers in Pasture Forage producers in Pasture County know current research County know current research information and use it to make information and use it to make informed decisions.informed decisions.

The program trains and empowers The program trains and empowers community volunteers.community volunteers.

Campers, aged 12-15 years of age, Campers, aged 12-15 years of age, learn new leadership and learn new leadership and communication skills while at camp.communication skills while at camp.

2020

Outputs & OutcomesOutputs & Outcomes

An annual conference disseminates An annual conference disseminates the latest forage research.the latest forage research.

Low-income families are better able Low-income families are better able to manage their resources.to manage their resources.

Program staff teach financial Program staff teach financial management skills to low-income management skills to low-income families. families.

Community volunteers have Community volunteers have knowledge and skill to work knowledge and skill to work effectively with at-risk youth. effectively with at-risk youth.

The camp experience provides The camp experience provides leadership development leadership development opportunities for 4-H youth. opportunities for 4-H youth.

Forage producers in Pasture Forage producers in Pasture County know current research County know current research information and use it to make information and use it to make informed decisions.informed decisions.

The program trains and empowers The program trains and empowers community volunteers.community volunteers.

Campers, aged 12-15 years of age, Campers, aged 12-15 years of age, learn new leadership and learn new leadership and communication skills while at camp.communication skills while at camp.

OUTPUT OUTCOME

2121

Outputs or Outcomes?Outputs or Outcomes?

Recruiting & Training Recruiting & Training Staff/VolunteersStaff/Volunteers

#/% of clients served#/% of clients served

#/type of participants#/type of participants

Participant SatisfactionParticipant Satisfaction

Surveys conductedSurveys conducted

Curricula DevelopedCurricula Developed

Research GeneratedResearch Generated

2222

VolunteersVolunteers

If the program is addressing a situation of If the program is addressing a situation of low volunteer involvement in community low volunteer involvement in community affairs and the purpose of the program is to affairs and the purpose of the program is to increase volunteering among community increase volunteering among community residents as a part of a larger community residents as a part of a larger community development initiative, then increased development initiative, then increased numbers of residents volunteering in numbers of residents volunteering in community life would be an outcome. The community life would be an outcome. The outcome is expressed as a behavioral outcome is expressed as a behavioral change. change.

2323

Number or type of participants who attend; Number or type of participants who attend;

number of clients served.number of clients served.

If the purpose of the program is to increase If the purpose of the program is to increase use of a service by an underserved group, use of a service by an underserved group, then numbers then numbers using using the service would be an the service would be an outcome. The outcome is not numbers outcome. The outcome is not numbers attending or served; the outcome is attending or served; the outcome is expressed as expressed as use use that indicates behavioral that indicates behavioral change. change.

2424

Participant Satisfaction.Participant Satisfaction.

For our purposes in education and outreach For our purposes in education and outreach programming, client satisfaction may be necessary programming, client satisfaction may be necessary but is not sufficient. A participant may be satisfied but is not sufficient. A participant may be satisfied with various aspects of the program with various aspects of the program (professionalism of staff, location, facility, (professionalism of staff, location, facility, timeliness, responsiveness of service, etc) but this timeliness, responsiveness of service, etc) but this does not mean that the does not mean that the person learned, benefited person learned, benefited or his/her condition improvedor his/her condition improved. .

2525

Training, Research, ProducingTraining, Research, Producing

These are Outputs. They may be essential These are Outputs. They may be essential aspects that are necessary and make it aspects that are necessary and make it possible for a group or community to possible for a group or community to change. But, they do not represent benefits change. But, they do not represent benefits or changes in participants and so are not or changes in participants and so are not outcomesoutcomes. They lead to, result in outcomes, . They lead to, result in outcomes, but in and of themselves, they are outputs.but in and of themselves, they are outputs.

2626

Steps in the Logic Model ProcessSteps in the Logic Model Process

11 Establish a stakeholder work group and collect Establish a stakeholder work group and collect documents.documents.

22 Define the problem and context for the program or Define the problem and context for the program or project.project.

33 Define the elements of the program in a table.Define the elements of the program in a table.44 Verify the logic table with stakeholders.Verify the logic table with stakeholders.55 Develop a diagram and text describing logical Develop a diagram and text describing logical

relationships.relationships.66 Verify the Logic Model with stakeholders.Verify the Logic Model with stakeholders.77 Then use the Logic Model to identify and confirm Then use the Logic Model to identify and confirm

performance measures, and in planning, conducting performance measures, and in planning, conducting and reporting performance measurement and and reporting performance measurement and evaluation. evaluation.

2727

““Z” LogicZ” LogicUnpacking the Program’s LogicUnpacking the Program’s Logic

In real life program’s achieve their strategic results In real life program’s achieve their strategic results through a series of actions similar to a relay race.through a series of actions similar to a relay race.– Action A produces a set of outcomes that become Action A produces a set of outcomes that become

inputs to Action B.inputs to Action B.– Action B produces a set of outcomes that become Action B produces a set of outcomes that become

inputs to Action C.inputs to Action C.– Action C produces a set of outcomes that lead to the Action C produces a set of outcomes that lead to the

final strategic goal of the program.final strategic goal of the program. These actions could be thought of as nested These actions could be thought of as nested

programs within the larger program.programs within the larger program.

2828

Action AResources A Outcomes

Action BResources B Outcomes

Action CResources C Outcomes

Strategic Program Results!

““Z” LogicZ” LogicSupplier-Customer RelationshipSupplier-Customer Relationship

Unpacking supports more focused Performance Measurement and thus more useful evaluation, as well as better understanding and communication about how the “Program” is supposed to work!

2929

3030

3131

3232

3333

3434

3535

3636

3737

Logic Modeling, Performance Logic Modeling, Performance Measurement, and EvaluationMeasurement, and Evaluation

3838

Key Questions Grantees Need to Key Questions Grantees Need to Answer About Their ProgramsAnswer About Their Programs

What am I doing, with whom, to whom/what? (What am I doing, with whom, to whom/what? (efforteffort)) How well am I doing it? (How well am I doing it? (qualityquality))

– Customer FeedbackCustomer Feedback– Peer Review for Technical QualityPeer Review for Technical Quality– User Review for Social ValidityUser Review for Social Validity

Is anybody (anything) better off? (Is anybody (anything) better off? (effecteffect))– Short-termShort-term– Long-termLong-term

What role, if any, did my program play in the results?What role, if any, did my program play in the results? What role, if any, did the context play?What role, if any, did the context play? Were there any unintended outcomes?Were there any unintended outcomes? What will happen if I don’t do something?What will happen if I don’t do something?

Performance Measurement

Program Evaluation

3939

Pro

gram

Log

ic H

iera

rchy

Hierarchy of P

erformance M

easurement D

ata

Program Logic Elements Matching Levels of Performance Information

7. End results

6. Practice and behavior change

5. Knowledge, attitude, and skill changes

4. Reactions

3. Participation

2. Activities

1. Resources

7. Measures of impact on overall problem, ultimate goals, side effects, social and economic consequences

1. Resources expanded; number and types ofstaff involved; time extended

2. Implementation data on what the program actually offers or does

3. The characteristics of program participants and clients; numbers, nature of involvement; background

4. What participants and clients say about the program; satisfaction; interest; strengths; weaknesses

5. Measures of individual and group changes in knowledge, attitude, and skills

6. Measures of adoption of new practices and behavior over time

Performance Measurement Hierarchy

4040

In the end Logic Models:In the end Logic Models:Enable grantees to:Enable grantees to: Develop a more convincing, plausible argument Develop a more convincing, plausible argument

RE how their program is supposed to work to RE how their program is supposed to work to achieve their outcomes and communicate this to achieve their outcomes and communicate this to funding agencies and other stakeholders. funding agencies and other stakeholders.

Focus their Focus their PM/PEPM/PE on the right elements of on the right elements of performance to enable performance to enable program improvementprogram improvement and and the estimation of the estimation of causal relationshipscausal relationships between and between and among elements.among elements.

Be better positioned to Be better positioned to present and defend their present and defend their claims about their program performance.claims about their program performance.