Targets & Improvement Activities State Improvement Planning Carol Massanari, MPRRC Western Regional Resource Center APR Clinic 2010 November 1-3, 2010

Targets & Improvement Activities

State Improvement Planning

Carol Massanari, MPRRC

Western Regional Resource Center APR Clinic 2010 • November 1-3, 2010 • San Francisco,

California

It’s been six years already?

Old age arrives suddenly, as does the snow. One morning, on awakening, one realizes that everything is white.

Western Regional Resource Center APR Clinic 2010

Getting here has not always been obvious.


A New Opportunity


• Using prior data to look at trends over time.

• Reviewing and evaluating what we’ve been doing.

• Shaping the form that is emerging from the chaos.

State Improvement Planning

• SPP/APR as a management tool

• Evaluating to document and report results

Using the SPP/APR as a Management Tool

Jeanna Mullins

Jim Leinen, WRRC


California

Development

• Started with information from the November 2009 WRRC Think Tank on “Managing Improvement”

• Field tested with KY Part B staff

• Gathered input at Spring TA&D Conference and from RRC Program staff

Design

• TA Tool for TA providers at the national, regional, State or local levels

• Guide conversations on SPP process and management structures

• Serve as an informal formative evaluation

• Assist in identifying potential areas for improvement

Attributes of Promising PracticesTopical Areas• Leadership• Staff/Workforce Assignments• Strategic Planning• Technical Assistance and Improvement Activities• Measurement, Analysis and Knowledge Management• Customer Focus/Results


RubricRating of Attributes

•Fully Operational•Under Development•Under Consideration•Not Considered at This Time•Not Applicable


Needs Identification

• Strengths • Potential opportunities for change• Comparison to federal, State, regional or local priorities for targeting improvement• Resources available to support improvement


Using the SPP/APR as a Management Tool Activity:

• Select one of the state systems’ Attributes:

• Leadership, Staff/Work Force Assignments• Strategic Planning• Technical Assistance and Improvement Activities• Measurement• Analysis & Knowledge Management• Customer Focus/Results

• Answer the questions related to your selected Attribute


Activity #1 Team Report Out:

• What attribute did you select and why?

• Share1-2 highlights about what you discovered while applying this tool

• Any comments about the tool?


Guiding Evaluation and Improvement Planning

Logic Models & Evaluation PlansJeanna Mullins

Jim Leinen, WRRC


California

Logic Model Elements Inputs (resources to be used)

Processes (activities and strategies to be used)

Outputs (immediate, countable results produced by activities), and

Outcomes (benefits and changes that are realized)

Logic Model Elements Inputs (resources to be used)

Processes (activities and strategies to be used)

Outputs (immediate, countable results produced by activities), and

Outcomes (benefits and changes that are realized)

A New Resource

EVALUATING SPP/APR IMPROVEMENT ACTIVITIES

Developed by members of the Systems and Improvement Planning Priority Team (NECTAC and RRCP)

December 2009

Evaluating SPP/APR Improvement Activities

Provides:

1. information about evaluation, improvement planning and strategic systems thinking

2. guidance on selecting an appropriate design for evaluating different types of improvement activities

3. additional resources and tools that support SPP/APR implementation and evaluation


Foundation: Systems Thinking• Systems are made up of interrelated,

interconnected components

• Systems change involves changing the capacity, interrelationships, and interdependencies among parts, levels and stakeholders

• Desired changes in one part or level of the system must be accompanied by changes in other parts or levels


Foundation: Logic Models

Inputs Process Outputs Outcomes

• ‘theory of change’ model to show the process by which change is expected to occur

• illustrated through a visual depiction or logic model

• shows how the inputs (resources) and processes (activities) connect to outputs (products or units produced, such as number of staff trained) and outcomes (intended results)

What is a Logic Model?

“A systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to do, and the changes or results you hope to achieve.”

WK Kellogg Foundation (2002)

Why Use a Logic Model?*

• Serves as a clear roadmap for the work

• Provides a framework for planning, implementation and evaluation

• Graphically illustrates key elements of an improvement activity and how they are interrelated

*Source: Evaluation of Improvement Activities: Considerations and Strategies. George Liacopoulos, Ph.D. Mid-South Regional Resource Center’s Systems Improvement Forum, Baltimore, MD. April, 2010.

Benefits of Logic Models*

• Ensures stakeholders understand the purpose of the activities, the resources needed, the activities to be conducted and their capacity to effect change

• Tracks what has worked and what has not worked well so that success can be replicated

• Identifies and prioritizes questions to be asked in an evaluation

*Source: Evaluation of Improvement Activities: Considerations and Strategies. George Liacopoulos, Ph.D. Mid-South Regional Resource Center’s Systems Improvement Forum, Baltimore, MD. April, 2010.

Foundations: SPP as long term plan for systems change• EI/ECSE are systems of complex, interrelated

components with goal of achieving outcomes for children and families

• Changes to EI/ECSE systems require a combination of improvement activities that are interconnected and support changes to infrastructure that work together to achieve the desired results

Selecting and Reviewing a Set of Improvement Activitiese.g. • Was a root cause analysis conducted to drive the activities?• Was a logical linkage established between the root cause(s)

and proposed activity outcomes? • Have specific action steps been developed (e.g., task,

activity, resources, timelines)? • Have personnel been identified who will develop, implement,

monitor and evaluate the improvement activity? • Have methods to collect the outcome data been identified? • Has a data collection timeline been developed?

(And more … see full paper)

Steps for Evaluating Individual Improvement Activities1. What is the goal or purpose?

2. What methods will you use for data collection?

3. What are your timelines for collecting, analyzing and reporting the data?

4. What is your baseline data that you will monitor for improvement?

5. How will the data be analyzed?

6. How will the evaluation results be used?

7. Who will be responsible for the various aspects of the evaluation?

Scenario: Child Outcomes DataSituation: • Considerable amount of missing data on C3/B7

• Local programs are unclear about when the data should be collected and reported, etc.

• Written policies and procedures are unclear, e.g. roles and responsibilities of Part C v. 619

• Not all local staff have been trained

State Improvement Activities to address missing data issue:

1. Collaborate across State Part C and Section 619 agencies to clarify roles and responsibilities

2. Revise written policies and procedures to clarify e.g. roles/responsibilities

3. Provide additional training on the child outcomes data collection and reporting process

Scenario: Child Outcomes Data

Short term goals of the 3 activities:• To clarify roles and responsibilities of Part C and Section 619

school district staff in the collection and reporting

• To improve the written policies and procedures so that local program staff find them clear and comprehensive

• • To increase local provider/program understanding of the child

outcomes data collection and reporting process

Long term goal of these 3 activities:

To increase statewide reporting of child outcomes data (and decrease the missing data in the child outcomes indicator)

Scenario: Child Outcomes Data

Logic Model for Scenario: Child Outcomes Data

Existing policies and procedures

Staff time

Training protocol

Trainer expertise

State data system

Meetings, communication, collaboration between Part C and 619 staff

Work on revising and clarifying policies and procedures

Survey providers on roles and responsibilities

Revise training to include updated policies and procedures

Provide training

Provide follow-up TA

Revised policies and procedures

Revised training protocol and materials

75% of Part C and 619 staff received training

90% of local staff report increased knowledge of roles and responsibilities related to collecting and reporting child outcomes data

Increase statewide reporting of child outcomes data (decreased missing data) e.g. 90% of all children exiting the program in year with at least 6 mo service are included in outcomes data

Improved reliability and validity of APR data for Indicator C3/B7

Inputs Activities Outputs Outcomes

Contact Information


Christina Kasprzak, NECTAC/[email protected]

Jeanna Mullins, [email protected]

mailto:[email protected]

mailto:[email protected]

Key ConsiderationsConsidering it may not be possible to evaluate all

improvement activities, you could develop criteria to determine which activities you would like to evaluate:

Where is the state investing most resources? “More bang for your buck” – What improvement

activities can be clustered and/or influence multiple Indicators?

Improvement activities associated with indicators for which no progress is being evidenced

Indicators for which you have no or little data Extent of implementation (e.g., statewide or more

targeted) Degree of confidence in the improvement activity


Evaluation Plan/Logic Model Team Activity Instructions:

• Select an improvement activity from your SPP

• Explain reasons for selecting this improvement activity

• Draft an evaluation plan or an evaluation logic model for this activity


Activity #2 Team Report Out:

• What indicators or cluster of indicators did you select?

• What improvement activities did you select and why?

• Any questions or comments about the process?


Documents

Targets & Improvement Activities State Improvement Planning Carol Massanari, MPRRC Western Regional Resource Center APR Clinic 2010 November 1-3, 2010