Moving Toward Evidence-Based Practice:

Innovative Ideas for the CSBG Network

NASCSPSeptember 2012

Barbara Mooney, Ed. D. Karen Walker, Ph.D.National ROMA Training and Certification Project

We know we have a groundwork in the CSBG program for Performance Measurement and Performance Management, but how do we build on this foundation to move forward?

Performance? Evidence?

We need strategies for moving the CSBG Network toward greater adoption of evidence-based practices.

Workshop Objectives

Definitions and examples of Evidence-based, evidence-informed

Participants will understand the regional and national factors pushing toward new practice.

Participants will consider how the core activities in IM 49 support universal implementation of high quality practices

Participants will learn about examples of using evidence to evaluate and improve programs

Participants will brainstorm “next steps” for action to move us toward more evidence based practices

What Does it Mean to be “Evidence-based”?

What Do People Mean by Evidence-based?

Multiple terms are in use: Evidence-based practice Evidence-based program Evidence-informed program

…How do we make sense of these terms?

What do we mean by Evidence-based programs and practices?

Evidence-based programs: Interventions that have been found to have

positive effects or impacts on targeted outcomes (Nurse Family Partnership, Parent Management Training)

Evidence-based practices: Intervention strategies or program practices

that have been found to improve outcomes for children and youth (role-modeling, positive reinforcement, avoiding grouping children with problem behaviors together)

Evidence-informed Program

Intervention that has been designed with research evidence in mind. Some of the program’s strategies may be evidence-based practices, they may be informed by the basic research, or both.

Evidence-informed program may also refer to a program that uses performance information to monitor and guide its progress.

Time Out for a brief description of what we mean by Randomized-Controlled Impact and Quasi-Experimental Evaluations

Randomized-Controlled Impact Evaluation Provides most certainty about program

effectiveness, but must be well-designed and well-implemented

Study participants randomly assigned to control or intervention group (think lottery).

Compares the difference in outcomes between the intervention and control group.

Quasi-Experimental Study

Like randomized-controlled impact study (RCT) a quasi-experimental study includes a comparison between people who received the intervention and those who did not.

Unlike an RCT, the comparison is not based on random assignment.

As a result, researchers are less certain that the comparison is a good one.

What Can You Do to Become Evidence-Based or Evidence-Informed?

Adopt evidence-based programs and monitor their implementation to ensure that they are being replicated correctly

Assess “home-grown” programs to determine whether or not they include evidence-based or evidence-informed practices

Set the program on a path toward rigorous evaluation

…These strategies are not mutually exclusive

Collect and Use Data on Performance & Outcome Measures

Conduct Needs

AssessmentIdentify

Population

Becoming an Evidence-Based Program*

Targeting

Conduct a Quasi-Experimental Outcomes Evaluation

Conduct an Implementation Evaluation

Select Intervention, Develop Logic

Model & Identify Indicators

Implement Program Conduct Ongoing

Performance Management

Conduct a Randomized-Controlled Impact Evaluation

Figure adapted from: Performance Management and Evaluation: What’s the Difference by Karen E Walker and Kristin Anderson Moore. Child Trends

Federal, State and Local Efforts and Policies

Widespread move toward evidence-based programs and practice and performance management Federal funding tied to evidence State initiatives and funds for

intermediaries Local efforts include both public and

non-profit efforts Administration’s guidance from OMB to

agencies about encouraging evidence-based efforts.

Evidence-Based Practice – Approaches to prevention or treatments that are validated by some form of documented scientific evidence. These could be findings established through scientific research, such ascontrolled clinical studies or other comparable and rigorous methods.

Evidence-Informed Practice – Approaches that use the best available research and practice knowledge to guide program design and implementation within context. This informed practice allows for innovation and incorporates the lessons learned from the existing research literature.

Definitions from OCS:

14

Use Data to Show Results and Improve Performance

• “Success should be judged by results, and data is a powerful tool to determine results. We can’t ignore facts. We can’t ignore data.”

President Barack Obama July 24, 2009

• “The test of a performance management system is whether it’s actually used….Federal managers and employees at all levels must use performance goals and measures to set priorities, monitor progress, and diagnose problems.”

Chief Performance Officer Jeffrey ZientsOctober 29, 2009

New Guidance:GPRA Modernization Act 2011

Shift emphasis from production of annual reports to collecting and using performance information more frequently throughout the year to set priorities and make decisions that improve results

Establish common Federal Government performance indicators with Quarterly Targets and Quarterly Milestones

The plan to target CSBG resources

“Building on the President's budget request to target CSBG resources to high-performing, innovative agencies, ACF plans to work with Congress and a wide variety of organizational stakeholders to establish a set of core Federal standards that States will be required to use to assess whether an eligible entity is meeting a high standard of service delivery.

These standards of service delivery will include a focus on compliance with basic requirements of the CSBG Act, and will also go beyond basic compliance to focus on high-quality performance across the CSBG Network.

Core performance standards will focus on organizational management issues, as well as strategic process for performance management.” (OCS T/TA overview, 2012)

The CSBG National T/TA Strategy is designed to:

Measure and document performance in accordance with CSBG reporting requirements.

Assure performance measurement and use of performance data at all levels within the CSBG program (local, State, and Federal).

Mitigate risks associated with the use of CSBG grant funds. Promote exemplary practices and innovative programming that

stimulates the creation and sharing of information and knowledge in the CSBG T/TA Network.

Measure and document performance in accordance with CSBG reporting requirements.

Work with States to help ensure CSBG-eligible entity boards know, understand, and meet their fiduciary responsibilities to CSBG (i.e., participation in the development, planning, implementation, and evaluation of CSBG-funded activities and services).

Ensure the management and delivery of CSBG-funded services creates and strengthens sustainable economic resources in communities

ROMA Next Generation and Organizational Standards

Two targeted T/TA Centers of Excellence embody OCS focus for coming years

Organizational Standards

The CSBG T/TA Organizational Standards Center of Excellence (COE) will coordinate the development and dissemination of a core set of standards with input from the State CSBG Lead Agencies as appropriate, regional and national partners, including the CSBG T/TA Regional Performance and Innovation Consortia (RPIC).

The central mission of this effort will be to ensure that all CSBG-eligible entities have the capacity to provide high-quality services to low-income individuals and communities, and are able to meet high-quality organizational performance standards.

The movement toward continued accountability, risk mitigation, and performance plays a key role in developing organizational standards toward improved performance management and the capacity to thrive within the CSBG Network.

ROMA Next Generation

Progress has been made in recent years in the quality and breadth of State and local performance reporting systems through ROMA and the NPI framework… (which) provides a foundation for additional targeting of resources based on outcomes and effectiveness of services.

However, practices of data collection and reporting within the ROMA system vary from State to State; while some States‘ reporting systems reflect the full ROMA cycle, other States only report NPI data.

Such inconsistencies within the ROMA system minimize the capacity of ROMA to be an integrated and comprehensive effort that lends to accurate and reliable performance management and accountability within the CSBG model.

Key goals of the ROMA Next Generation COE will be:

To review and strengthen the current performance measures and performance indicators with input from experienced CSBG organizations, as well as performance management experts from outside the CSBG Network;

To work with State CSBG agencies to communicate consistent standards for performance management throughout the ROMA cycle, including community needs assessment, program planning, and performance measurement;

To assure a coordinated, national strategy for nationwide implementation of an updated performance management system;

To assure that a performance management system can be used, as necessary, for comparison of the effectiveness of local entity use of CSBG resources.

We are keeping the basics:

Six National Goals ROMA Cycle (core activities for

eligible entities) Core activities for state offices Checklist for Monitoring (IM102) Risk Mitigation (IM 112) Key elements of the Logic Model

Six National Goals

1. Low-income people become more self-sufficient. 

2. The conditions in which low-income people live are improved.

 

3. Low-income people own a stake in their community.  

4. Partnerships among supporters and providers of services to low-income people are achieved.

5. Agencies increase their capacity to achieve results.  

6. Low-income people, especially vulnerable populations, achieve their potential by strengthening family and other supportive systems.

24

IM 49 Core Activities for State Offices

25

1. The state has developed, in coordination with eligible entities and the State CAA association, a State-wide vision statement that speaks to the goals and purposes of community action within the State and that supports the 6 national ROMA goals. The state is encouraged to participate in, and contribute to, broader State anti-poverty/community development initiatives with outcome measures and goals compatible with ROMA;

2. The agency has trained all its eligible entities (staff and boards) in outcome-based management, and that 80% of the entities use ROMA concepts to guide needs assessment, agency mission review, activity planning, resource allocations, service delivery, measuring and reporting results;

Core Activities for State Offices (cont)

26

3. 80%of the plans and program reports received from eligible entities in the State describe plans to achieve projected outcomes, and evaluate results based on measurable improvements of condition(s) among clients and/or communities served;

4. The agency submits complete, accurate, and timely annual reports to OCS on the measured performance of the State and the eligible entities in the State as required by Section 678E of Public Law 105-285, the Community Services Block Grant Reauthorization Act of 1998.

ROMA Logic ModelIdentified Problem,

Need, Situation

Service or Activity (Output)

Identify the timeframe.

Identify the # of clients served or the # of units

offered.

Outcome (General statement of results expected)

Outcome Indicator Projected # and % of

clients who will achieve each outcome.

Actual Results

Actual # and % of clients who achieve

each outcome.

Measurement Tool

Data Source, Collection Procedure,

Personnel

Frequency of Data Collection and

Reporting

(1) Planning (2) Intervention (3) Benefit (4) Benefit (5) Benefit (6) Accountability (7) Accountability (8) Accountability Short Term

Short Term

Short Term

Intermediate Term Intermediate Term

Intermediate Term

Long Term

Long Term

Long Term

CommunityFederal, State and Local Policies PopulationCulture (norms and behaviors) Funding

OrganizationMission & Goals Leadership ResourcesPolicies Program Monitoring Staff assessments

ProgramActivitiesStaff roles and skillsTrainingQuality and FidelityParticipationOutcomes

28

ExamplePerformance Management In Action

After-school tutoring program assisting at-risk youth – attempting to improve their math and reading performance.

Collect the following data: Demographics and risk/protective factors Program attendance Amount of service (hours and topics of

tutoring) Report card grades (first / last grading

period) 29

Example (cont.)Performance Management in Action

Finding #1: Halfway through program, program attendance is low among youth from Neighborhood A. Action: Reach out to teachers in schools in

Neighborhood A Action: Talk with youth in Neighborhood A

Finding #2: Math grades increased, but reading grades did not change. Action: Investigate. Finding: Tutors unfamiliar with reading curriculum Action: Train tutors in reading curriculum

30

Executive leadership To set the tone, to implement policies

Staff expertise To identify measures, to collect data, and to analyze

data Staff time

To collect, enter, analyze, share, and discuss data Technology

Forms (paper or electronic), spreadsheet software, management information systems/commercial software

Consider Resources for Performance Management

31

Performance management policy outlining: Staff roles and responsibilities related to

performance management Description of data collection and reporting

methods How often regular meetings should take place

to review and discuss findings Performance appraisal process which

evaluates staff on how well they carry out performance management roles and responsibilities

Create Policies for Performance Management

32

Are we reaching our intended population with the intended services?

Are participants getting the amount of program time we want them to get?

Are participants engaged in the process? Are participants’ outcomes changed? Do our staff carry out activities as intended? Are we providing the training our staff need?

If the answers are no, organizations need to determine why and what needs to change

Example Questions in Performance Management

34

An example about how sharing performance management data on outcomes can increase staff buy-in, inspire youth, and create a learning culture

Video Clip from “Saving Philanthropy”

http://www.savingphilanthropy.org/preview-footage/

Implementation will take resources

Time and attention to develop a plan of action

Staff time and support (travel, meeting costs, provision of materials, etc.) At the state level, for local EEs

Follow up to evaluate actions and assess new needs/resources

T/TA from National ROMA Trainers

The demand for training and technical assistance exceeded the available resources in the ROMA Trainer network to address the needs.

Two specific areas were identified: The certified trainers do not have any resources to bring local CAAs

together as they begin to implement the full range of ROMA activities. Implementation support is critical to working out solutions to local challenges and establishing a consistent use of terms, concepts and processes.

NCRTs do not have the time or the expertise to provide individual on-site technical assistance to local CAAs as they apply the principles to agency systems.

We are working to increase the number of available NCRTs in partnership with the RPICs and State Offices

State CSBG Discretionary Dollars

NASCSP Resources on discretionary funds: Annual Reports and appendix tables Issue Brief:

State Discretionary Spending

Distribution of FY 2011 CSBG Discretionary Funds by Purpose

CSBG National Resource Center & Clearinghouse

Resources online Evidence-based discussions Forums Lists of consultants

Child Trends’ Resources

Using Data in Multi-Agency Collaborations to Ensure Accountability and Improve Performance. http://www.childtrends.org/Files//Child_Trends-2012_02_23_FR_UsingData.pdf

Performance Management and Evaluation: What’s the Difference? http://www.childtrends.org/Files//Child_Trends-2011_01_19_RB_PerformMgmt.pdf

at www.childtrends.org

Leap of Reason: Managing to Outcomes in an Era of Scarcity, available at leapofreason.org

What’s next?

How can you identify and focus resources so our network can move toward evidence based practices?

What specific steps do you need to take to begin to identify what works and how to collect the evidence?

What do you need to consider?

Steps toward improving practice

IMPROVING REPLICATION AND “SCALE-UP”

The history of replication and scale-up efforts indicates the difficulty associated with these efforts.

As President Bill Clinton frequently declared, “You can find virtually every problem in our country solved by somebody somewhere in an astonishingly effective fashion….The challenge for us is to figure out how to replicate that.”

The day before his first inauguration, Clinton told the nation’s governors that his number one disappointment as governor had been that it was so hard to “take something that works to the next level.” He said he “could never figure out a way to make the exception the rule, and that is our enduring problem in America in public life.”

For more discussion:

Barbara MooneyProject Director

National ROMA Training and Certification Project243 E. High St.Waynesburg, PA

724-852-2272barbaramooney@windstream.net

Karen Walker Senior Research Scientist and Senior Program

Area Co-Director

Child Trends 4301 Connecticut Ave NW

Suite 350Washington, DC 20008

202-572-6005kwalker@childtrends.org