Welcome New and Expanding PCCD Providers New Grantee Orientation October 16, 2013

Meet & Greet Activity • Help yourself to refreshments • Please join the table for your program •  Introduce yourselves • Describe your organization • Describe why your organization/community selected the

program including: •  Target Population •  Targeted Risk and Protective Factors •  Hopes for the future (after PCCD funding ends)

Welcome — PCCD’s Investment in Evidence-based Programs Stephanie Bradley, Ph.D. Upcoming EPISCenter Director

EPISCenter’s Three Key Functions • Build general prevention

capacity

• Build program-specific capacity

• Facilitate interaction/ communication between systems

Tran

sla'

onal  

Research

EPISCenter  (Preven'on  

Support  System)

Build  general  preven'on  capacity  among  prac''oners  and  policy  makers  

Technical  Assistance

Con'nuous  Quality  

Improvem

ent

EBP Grantees & Community Coalitions (Prevention Delivery System)

Penn State’s Prevention Research Center

(Prevention Synthesis & Translation System)

Resource Center Steering Committee

(Policy Makers & Funders)

Rhoades, Bumbarger & Moore (in press). The Role of a State-Level Prevention Support System in Promoting High-Quality Implementation and Sustainability of Evidence-based Programs. American Journal of Community Psychology.

Pennsylvania’s EBP Dissemination in 1999…

Pennsylvania’s EBP Dissemination in 2012…

Resource  Center    for  Evidence-­‐based  Preven'on  and    Interven'on  Programs  and  Prac'ces  

Support to Community Prevention Coalitions

Improve  Quality  of    Juvenile  Jus'ce  Programs  

and  Prac'ces  

Support to Evidence-based Programs

Multi-Agency Steering Committee (Justice, Welfare, Education, Health)

A unique partnership between policymakers, researchers, and communities to bring science to bear on issues of public health and public safety. The EPISCenter is a project of the Prevention Research Center, College of Health and Human Development, Penn State University, and is funded by the Pennsylvania Commission on Crime and Delinquency and the Pennsylvania Department of Public Welfare as a component of the Resource Center for Evidence-Based Prevention and Intervention Programs and Practices.

PCCD Grantee Requirements Christina Cosgrove-Rooks, Program Analyst PCCD

Quality Reporting • Grantees are responsible for reporting in Egrants

quarterly. •  Report can be initiated after previous quarter is approved. •  Reports are due to be submitted by the 20th of the month after the

quarter is closed. •  Data collection tools and other documents are submitted as

attachments

Quality Assurance Process • Purpose: To assess site functioning, review your data

collection process, and ability to demonstrate and communicate impact.

Start Date QA Process Date July 1, 2013 July-September 2014

October 1, 2013 October-December 2014

*Funds should have been budgeted in your grant for this process.

Quality Assurance Process •  The (insert name of County/Agency) site is implementing with sufficient

quality. Given the current implementation, it is reasonable to expect future positive outcomes from the program. At this time, the areas identified as needing improvement are limited and as to be expected for the length of implementation time.

•  The (insert name of County/Agency) site needs to make significant changes

to their implementation and is currently not implementing with the level of quality that is expected for the amount of time it has been implementing. For programs in this category, it is PCCD's intention to work with the developer and grantee to articulate a plan of corrective action and timetable to bring the program into compliance with the developer's requirements.

•  The (insert name of County/Agency) site is not implementing the program as designed and the developer does not believe that the site’s implementation quality is correctable within a reasonable amount of time.

Outcomes Report • Purpose-

•  Summarize Data Collected to Measure •  Program Reach •  Implementation Quality •  Impact

Start Date OR Due Date July 1, 2013 April 2015

October 1, 2013 July 2015

Outcomes Report • Quantitative Data

•  Performance measures and survey data

• Qualitative Data •  Feedback from participants, staff, and program partners •  Changes to project plan, project modification requests

Outcomes Report • Section 1: Description of Population Served

• Section 2: Indicators of Program Impact

• Section 3: Indicators of Implementation Quality

• Section 4: Lessons Learned

• Section 5: Program Sustainability

Top Three Reasons to Call a PCCD Analyst

• Project Modification Requests - Program and Fiscal changes to your application

• Quarterly Program/Fiscal Report questions

• Any fiscal questions/concerns should be directed to the fiscal contact listed in Egrants

Grantee Success Story • Melinda Gordon

•  Today Inc. •  Bucks County •  Project Towards No Drug Abuse

EPISCenter Technical Assistance Roger Spaw EPISCenter Prevention Coordinator

Evolution of Technical Assistance •  1998: PCCD invests in prevention programming.

•  Difficult to monitor implementations of a wide variety of programs. •  Grantees experienced mixed outcomes; some programs worked

well, some failed. •  It was hard to know about problems until “after the fact”. •  No formal technical assistance was available.

Evolution of Technical Assistance •  2001: Penn State’s Prevention Research Center begins

providing Technical Assistance to PCCD grantees. •  Grantees get support from the beginning, but programs are varied

and not all of them have high levels of evidence of effectiveness. •  Technical Assistance was mostly reactive. •  No standard approach to quality assurance •  Performance Measures varied greatly from grantee to grantee •  Difficult to know if programs were having desired impact

Evolution of Technical Assistance •  2008: EPISCenter was established to support programs

with a higher level of evidence of effectiveness. •  “Blueprints” programs www.blueprintsprograms.com

• EPISCenter now PROACTIVELY targets Technical Assistance for 11 programs. •  Goal: To advance high quality implementation, impact assessment,

and sustainability of programs in order to maximize the positive impact of prevention science on outcomes for Pennsylvania youth.

Evolution of Technical Assistance

Who Provides Technical Assistance? • Provided by EPISCenter Prevention Coordinators or

“PCs” • Prevention Coordinators:

•  Lee Ann Cook - IYS •  Kris Glunt – LST, TND, OBPP •  Roger Spaw - ART, BBBS •  New Hire - SFP 10-14, PATHS

Why Technical Assistance • Goal: To advance high quality implementation, impact

assessment, and sustainability of programs in order to maximize the positive impact of prevention science on outcomes for Pennsylvania youth.

• Objectives: •  Assess and support program implementation •  Promote program quality and fidelity •  Support the data collection process •  Maximize the ability to demonstrate and communicate impact •  Increase local investment & sustainability

How?: Explaining the Magic

“I think you should be a bit more explicit here in step 2.”

How? Web-based Resources • Web Based Resources: www.EPISCenter.psu.edu

•  Logic Models •  Frequently Asked Questions •  Readiness Guides •  Implementation Manuals •  Impact Measurement Tools & Guidance •  Performance Measure Reporting Tools and Guidance •  Links to research articles, Developer Websites, etc. •  Templates for Outcomes Reporting to PCCD and community

stakeholders •  Training videos

A word about Logic Models... • What happens when you bake a cake without reading the

recipe?

Poor Outcome… • You do not get what you hoped for…

How? Personal Consultations

• On-Site Consultations •  PC visits you in person in your community to review the Web-based

resources and discuss any successes or barriers •  Typically twice a year, but varies depending on program and need •  Initial consultation always includes completion of an

Implementation Plan with action steps for PC and Grantee

• Web-Based Consultations •  Skype, Adobe Connect, and Go-To Meeting are frequently used to

connect with grantees

• Email & Phone Calls •  Both great ways to contact your PC with a question

Implementation Plan

How? Group Training and Networking • Networking Meetings

•  PCs organize regular networking opportunities to allow grantees who are doing the same program to share lessons learned, tools, and resources.

• Trainings •  Often PCs will identify a common training need for several

grantees. They then identify experts who can meet this need and facilitate in-person or webinar training events. These trainings can be geared towards all of the EBPs or just one program specifically.

Demonstrating Program Impact § Solid data collection systems

§  Implementing with a commitment to quality and fidelity

§ Consistently analyzing process and outcomes data

§ Spreadsheet Tools

Grantee Success Story • Craig Adamson & Pam Thompson • Community Service Foundation • Bucks County • Aggression Replacement Therapy

Collaboration is Key Phyllis Law EPISCenter Communities That Care Consultant

Planning & Start-up Lee Ann Cook, MSS EPISCenter Prevention Coordinator

EPISCenter Resources for Start Up

• Readiness Tools • Training Information & Coordination • Materials Purchasing Guidelines • Staff Requirements

Break Out • EPISCenter PC will review specific “start up” resources • Discuss: How is start up going for you?

•  Successes? •  Barriers? •  Next Steps? •  How can EPIS help?

Lunch • On your own •  List of restaurants in your folder • Be back by 1:30 p.m. Please!

Implementation Quality (IQ) Ensuring the effectiveness of your program Sebrina Doyle, M.S. Research Coordinator, EPISCenter

Basis for “Evidence” • Deterrent effect with a strong research design

• Sustained effect

• Multiple site replication

Deviating from Program Design… •  “When communities ‘tweak’ a program to suit their own

preferences or circumstances, they wind up with a different program whose effectiveness is unknown.”

• Blueprints for Healthy Youth Development • www.BlueprintsPrograms.com

?

Logic Model •  Illustrates how and why the program produces changes

in the targeted outcomes •  Provides theoretical backbone

•  Informs quality implementation

•  Guides quality evaluation

Logic Model Example: Strengthening Families Program

Distal Outcomes

Reduced Youth Antisocial Behavior

Improved Academic

Engagement & Performance

Youth Sessions

Parent Sessions

Family Sessions

Program Components

Improved Youth Skills/Attitudes

Improved Parent Skills/Styles

Improved Family Relationships

Proximal Outcomes

Decrease Risk

Factors

Increase Protective

Factors

Targets

Model created by Penn State EPISCenter

Implementation Quality

Reduced Youth Substance Use

Elements of Implementation Quality • Dosage

• Adherence

• Quality of Delivery

• Program Reach

• Participant Responsivenes

Elements of Implementation Quality • Dosage

Elements of Implementation Quality • Adherence

Elements of Implementation Quality • Quality of Delivery

Elements of Implementation Quality • Program Reach (Target Audience)

Elements of Implementation Quality • Participant Responsiveness

Ways to Measure IQ • Observation

•  Implementer Self-Report

• Administrative Documents

• Participant Self-Report

Why Measure? • Measure IQ à Unlock black box

•  Ex: What is happening in the program?

• Measure IQ à Better IQ •  “What gets assessed gets addressed!”

• Better IQ à Better Outcomes

• Show funding agencies what you’ve done

Need Implementation Help? • We’re here to support the best possible implementation in

your communities.

• Contact your Prevention Coordinator for expert assistance

• Consult our website for additional resources •  www.episcenter.psu.edu

Images Credit •  www.freedigitalphotos.net/images/view_photog.php?photogid=1962 (single red apple) •  www.freedigitalphotos.net/images/view_photog.php?photogid=1993 (orange) •  www.freedigitalphotos.net/images/view_photog.php?photogid=1701 (problem chalkboard) •  www.freedigitalphotos.net/images/view_photog.php?photogid=862 (apple slicer) •  www.freedigitalphotos.net/images/view_photog.php?photogid=905 (multiple slices of fruit) •  www.freedigitalphotos.net/images/view_photog.php?photogid=1499 (apple core) •  www.freedigitalphotos.net/images/view_photog.php?photogid=862 (girl holding apple) •  www.freedigitalphotos.net/images/view_photog.php?photogid=879 (eating apple) •  www.freedigitalphotos.net/images/view_photog.php?photogid=1857 (apple measured) •  www.freedigitalphotos.net/images/view_photog.php?photogid=905 (half apples) •  www.freedigitalphotos.net/images/view_photog.php?photogid=3062 (green check) •  www.freedigitalphotos.net/images/view_photog.php?photogid=2280 (red X)

Break Out • EPISCenter PC will review specific “fidelity” resources &

processes • Discuss:

•  Logic Model •  Fidelity Checklists or Measures •  Quality Assurance Process •  Next Steps? •  How can EPIS help?

Data Collection and Reporting Lee Ann Cook

Data Collection • Process Data

•  How many •  How often •  How well

• Outcomes Data •  Baseline Pre Test or Survey •  Follow-up Post Test or Survey

Data Analysis & Reporting • PCCD Excel Spreadsheet Data Analysis & Reporting

Tools •  Generally each is designed to hold one year of data •  Update and submit to PCCD via egrants on a quarterly basis •  Provides summary data on each performance measure so that

providers can make adjustments to implementation as necessary

Data Reporting •  Its never to early to start sharing data

•  Internally •  Collaborative Board Reporting •  School Board

•  How? •  Community Reporting Templates •  Outcome Report Templates

•  Comprehensive Outcome Report Required at the end of Year Two.

Egrants- Christina Cosgrove •  E-grants – Uploading Reports

•  Program •  Fiscal

•  Grant Changes – PMR, Budget Modification

Break Out • EPISCenter PC will review specific resources &

processes for Data Collection and Analysis • Discuss:

•  Pre/Post Tools •  Program Excel Spreadsheet Tool •  Performance Measures •  Next Steps? •  How can EPIS help?

Blueprint for Sustainability Kris Glunt EPISCenter Prevention Coordinator

Blueprint for Sustainability

A Blueprint for Sustainability From Surviving to Thriving -Adapted from Stanislaus County Children and Families and Penn State Cooperative Extension/Walt Whitmer

What is Sustainability?

“…the continuation of community health or quality of life benefits over time.”

-A Blueprint for Sustainability From Surviving to Thriving adapted from Stanislaus County Children and Families

Sustainability — Develop a Plan • When should you start planning?

• Yesterday!!

Eight Elements for Sustainability 1.  Vision 2.  Results Orientation 3.  Strategic Financing Orientation 4.  Broad Based Community Support/Key Stakeholders 5.  Key Champions 6.  Ability to Adapt to Changing Conditions 7.  Strong Internal Systems 8.  Sustainability Plan

Vision & Desired Results • What are we trying to sustain? For how long? At what

level of activity?

• For what purpose – what do we seek to accomplish for children, families and communities in the long run? Focus should be on the health and well-being of the people being served.

• All partners should embrace the vision.

Community Relationships • Build Community Support

•  Identify Key Stakeholders •  Strategic partnerships and coordinated efforts are crucial to achieving

and sustaining desired results in the community.

•  Identify Key Champions •  Identify a group of key champions within and outside of your agency to

serve as advocates in the community, with funders and policymakers, etc.

Identify Funding Sources •  In-kind Resources • Foundations • Government • Corporations • Businesses • United Way • Community-Service Organizations • Faith-based Organizations •  Individuals • Schools/PTOs

Internal Capacity Building • Develop strong internal systems

•  ensure that all necessary fiscal, personnel, evaluation and other internal capabilities are functioning well.

• Ability to adapt to changing conditions •  potential problems can be proactively avoided •  emerging opportunities can be seized

Strategic Financing • Resource Requirements

•  Summarize the resources needed to carry out your vision

• Cost Management •  Strategies to control costs & be as efficient as possible

• Revenue Enhancement •  List ways to diversify and sustain future funding streams

• Long-Range Financial Forecast •  Develop a long-range financial forecast of revenues/expenditures

that incorporates the Resource Requirements, Cost Management Strategies and Revenue Enhancement Strategies

Seeking Funding Strategically •  List selling points: • State the benefits to the participants • State the benefits to the community • Tie your product to the funder’s

objectives and terminology • Highlight how your outcomes match the potential funders’

goals • Develop an incentivized approach strategy • Set timelines

Turn Your Plan into Action • Having a good plan is only the first step

•  Plan should be carried out diligently •  Designate someone with the responsibility of making sure the plan

is reviewed and updated on a regular basis

Sustainability — Develop a Plan • When should you start planning?

• Yesterday!!

• Planning Sustainability: Impact Beyond Grant Funding •  http://episcenter.psu.edu/PSIBGF

Final Break Out • Please fill out photo release and evaluation forms

• Program specific issues related to Pennsylvania Prevention Infrastructure •  Upcoming Networking Opportunities •  Developing In-State Training Capacity

• Cover remaining questions from previous breakouts

THANK YOU! The EPISCenter is a project of the Prevention Research Center, College of Health and Human Development, Penn State University and is funded by the Pennsylvania Commission on Crime and Delinquency and the Department of Public Welfare.

206 Towers Building, University Park, PA 16802 Phone : (814) 863-2568 w Email: EPISCenter@psu.edu

www.EPISCenter.org

/EPISCenterPSU @EPIS_Center