Upload
chears
View
506
Download
6
Tags:
Embed Size (px)
DESCRIPTION
Presentation give to annual COE Policy Seminar-Washington DC March 2011
Citation preview
Annual Seminar with Department of Education
Using Empowerment Evaluation and Data to Strengthen Your Talent Search Program
Maggie Cahalan and Stephaine MillerPell Institute, COEMarch 9 2011
Presentation Topics1. Overview of Empowerment Evaluation Concepts
and Steps with example from COE’s new I-3 project testing new models for Talent Search (Maggie)
2. Using Data for Evaluation Program Improvement and Student Success (Stepahanie)
3. Breakout Session (Speed Dating Process to move through ideas about steps in empowerment evaluation of your program)—(Everyone--35 minutes)
4. Group empowerment evaluation of our session (what you liked and did not like and how we could improve (Everyone--10 minutes)
Personal Journey in Thinking About Evaluation
Contractor Project Director National Evaluation of Student Support Services
National Evaluation of Talent Search
TRIO performance reporting support contracts
Department of Education National Evaluation of Upward Bound study final technical monitor
Came to see need for taking a more participatory approach involved and began Designing Next Generation of GEAR UP
Next Generation of Gear Up Evaluations
Grew out of challenges (problems with meaningful comparison group, validity issues, length of time to obtain results, black box up or down studies not always useful for program improvement, lack of involvement of stakeholders meant work not viewed as very useful to them)
New Approach (18 month design work—sessions with grantees, expert papers, systemic review of past evaluation methods and challenges, issued call for entries open to all grantees to design evaluations of what they viewed as promising practices, funded 44 grantees proposals and provided technical assistance, and funded 12 to implement the project led evaluations of rigorous designs)
Definition
Empowerment evaluation is the use of evaluation concepts, techniques and findings to foster improvement and self determination. It employs both qualitative and quantitative methodologies.
Empowerment evaluation: Knowledge and Tools for Self-Assessment and Accountability Fetterman, Kaftaraian, and Wandersman 1996
Empowerment Evaluation
Aims to increase the probability of program success by:
1 Providing program stakeholders with tools for planning and implementation, and self evaluation of their program, and
2 Mainstreaming evaluation as part of the planning and management of the program organization (Wandersman, et. Al, 2005, p. 25)
Empowerment Evaluation
Widely usedJoint Committee for Education Program Standards have included conceptsAmercian Evaluation Association and AERA workshops Public Health FieldInternational Development workEducation school systems
ContrastsTraditional Evaluation
External
Expert
Dependency
Independent judgment
Developed when data not available to all— elite with resources to collect and skill to analyze
Empowerment Evaluation
Internal
Coach or Critical friend
Self-determination & capacity building
Collaboration
Makes use of Data Revolution—internet, web, real time interactive sharing of knowledge—all publish—face book, blog, twitter
Internal or External Evaluation
The dilemma of whether to use external or internal evaluation is as false as that between qualitative and quantitative methods. The solution is always to use the best of both, not just one or the other (Michael Scriven)
Learn by Doing
Organizations and their staff evaluate their own strategies
May hire an evaluator to work with them
Rather than presenting a report card, empowerment evaluators coach staff through an evaluation of their own strategies by providing them with the knowledge, skills and resources they need to conduct evaluation
10 Key Principals (Wandersman et. Al 2005)
1. Community Ownership—primary responsibility with organization and not outside evaluator
2. Inclusion—involves representation of key stakeholders
3. Democratic participation—highly collaborative—opportunity to voices questions—every stakeholders voice is heard
4. Community knowledge-promotes growth of knowledge in community—stakeholders are considered to be in the best position to understand the issues and generate solutions to problems
5. Evidence Based strategies-promotes use of strategies with high quality evidence—research evidence of effectiveness—evidence strategies contextualized to fit community
10 Key Principals (cont)6. Accountability-provides data that can be used to determine
whether a strategy has achieved its goals—negative results are not punished;rather are use to inform change in a strategy or the selection of a new strategy for the purpose of producing better outcomes
7. Improvement—Helps organizations improve strategies so that they are more likely to achieve stated goals—process and outcome evaluation (Rossi 1999)
8. Organizational learning—fosters a culture of learning—view positive and negative feeedback as valuable information and believe that all strategies can be improved
9. Social justice—increase capacity to reduce disparities that affect marginalized by persecution, discrimination, prejudice and intolerance
10. Capacity building—builds capacity of organizations to conduct their own evaluations, understand results and use them to continuously improve organization
10 Steps (Chinman, Imm & Wandersman, 2004)
1. Conduct needs and resource assessment
2. Develop goals and outcomes sought
3. Selecting evidence based strategies
4. Assessing strategies fit
5. Building capacity
6. Finalizing plan
7. Evaluate process
8. Evaluate outcomes
9. Implementing continuous quality improvement
10.Sustaining efforts
Fetterman (2001)
Define mission
Taking stock-
Planning for future
Example---COE Investing in Innovation (I-3) project
ED wishes to build capacity at all levels to use data and build in internal and external evaluations as way to improve programming on an on-going basis
October COE won I-3—Investment in Innovation--Using Data to Inform College Access—Power Knowledge
Build on and help validate GE/COE college access work in 2 cities (Erie and Louisville)
Enhanced whole school college access services—college coaches in school
Intensive service Learning Communities
COE Building large data warehouse Power Knowledge Tool for use by all
Contribute to innovative adaptive models for established Federal College Access programs such as Talent Search, GEAR UP, Upward Bound
COE (I-3) project Testing and Validating ---Model-
--Increased access to, skill, and use of data by all stakeholders (students, parents, school, college coaches, community) will lead to increased high school and college participation and success among students in high schools
Internal Empowerment Evaluation and
External evaluators both using the DATA TOOL
DICAP--Builds and Enhances on GE-COE Model
Multiple Power Knowledge Learning CommunitiesUsing data among all stakeholders—key
District and school staffCollege access service providers and planners--College CoachesStudentsParentsCommunityInternal and External Evaluators
Goals of UN Education for Sustainable DevelopmentWhat if every person benefited from an education promoting
development that is environmentally sound, socially equitable, culturally sensitive and economically just?
What if learning was about knowledge and also about doing, being, interacting with others and changing the world?
What if formal learning was enjoyable, hands-on and relevant to life outside school while addressing the problems of our world?
What if every person benefited from genuine learning opportunities throughout life, in the workplace, and within the community?
What if education systems prepared learners to enter the workforce as well as handle a crisis, be resilient, become responsible citizens, adapt to change, recognize and solve local problems with global roots, meet other cultures with respect, and create a peaceful and sustainable society?
Partnership—Collaborative-Empowerment Model
All stakeholders full partners—those most impacted –strongest roles
StakeholdersHelp set goals
Identify assets, challenges, describe current baseline situation and data
Take responsibility for identifying and implementing steps (actions) to meet goals
Take responsibility for analysis, reflection on results and become learning community
Transparency among all stakeholders
Partnership—Collaborative-Empowerment Model
All stakeholders full partners—those most impacted –strongest roles
StakeholdersHelp set goals
Identify assets, challenges, describe current baseline situation and data
Take responsibility for identifying and implementing steps (actions) to meet goals
Take responsibility for analysis, reflection on results and become learning community
Transparency among all stakeholders
Smaller Intensive Learning Community
Power knowledge Learning Community Leadership Group
Summer Bridge
On-going school year workProject to participate in building and using the data tool
Project to communicate to whole school—lead to goals
Project to evaluate and assess efforts and to improve the program –internal empowerment evaluation
Theory of Participatory Action/Change
Participatory goal setting
Identify Inputs
(assets, baselines, challenges
and barriers)
Identify Steps/actions
needed to meet goals
Identify indicators and monitor fidelity
of implementation/challenges/cou
nterfactuals
Power Data Tool
Reflection and
Communicate/disseminate
findings
Learning Community Prepares various
reporting's/feedback for
stakeholders
Impact assessment
and recommendat
ions for future
Assessment of intermediate indicators/out
comes
Logic of Evaluation
Goal or purpose
Baseline –where before intervention
Program or intervention
Measurement (including criteria)
Process
Outcomes and impacts
Speed Dating Stations Idea Generating
1. Goal and Strategy setting1. Setting Goals/Mission—what do want to change or improve
—how much?
2. Evidence based strategy for improvement—what concrete steps do you need to take that you think might work? Now? Next year? Next five years?
2. Taking Stock1. Baseline—where are you now—take stock of assets and
barriers?
2. How can you assess or measure if what you are trying is working-----choose an evaluation design and indicators
3. Provide for gathering credible evidence---Data
3. Plan for Future—Learning Organization1. Use data to analyze and reflect on what you find
2. Improve organizational capacity and the particular strategy evaluated
Resources and contact information
Empowerment evaluation resourceshttp://eevaluation.blogspot.com/
www.cdc.gov/violenceprevention/pdf/evaluation_improvement-a.pdf
http://wwwstatic.kern.org/gems/region4/DavidFettermanPresentation.pdf