GRANTMAKERS READING

Roles for Theory in Evaluation

Social Science Theory

Program Theory

Evaluation Theory

Program Theory

A systematic configuration of stakeholders’ prescriptive assumptions (what actions must be taken) and descriptive assumptions (what causal processes are expected to happen) underlying programs, whether explicit or implicit assumptions.

- Chen, p. 136, Evaluation Roots, 2004

• A leadership retreat, where participants (a) learn the tools they need to carry out a community of practice (b) create a one year action plan, and (c) create an evaluation plan for the community of practice results

• Formation of the community of practice itself, where participants (a) conduct inquiry, (b) share ideas, (c) pilot change, and (d) document and evaluate

• Sustain ongoing recursive practice, where

participants (a) create a cycle of inquiry and change, (b) become trainers for the next generation, and (c) institutionalize the work

Short Term Outcomes

ImpactMedium

Term Outcomes

Long Term Outcomes

Theory of Change

Leveraging Mechanisms

Activities

3CSN ToC Work

3CSN’s ToC Narrative

• “If we provide training on networking and we use action research methodologies, community college professionals will transform their environments and identities to create communities of practice that will produce powerful learning and working across campuses. This will lead to greater student success.”

Theory of Change Exercise

• Pair up• Select a poster and write down suggested

theory of change elements• Prepare to share a few• 10 min

LOGIC MODELINGMichael’s stuff

Definition of a Logic Model

The program logic model is defined as a picture of how your organization does its work – the theory and assumptions underlying the program. A program logic model links outcomes (both short- and long-term) with program activities/processes and the theoretical assumptions/principles of the program.

- The W.K.Kellogg Foundation Logic Model Development Guide

Why Create a Logic Model?

Logic models provide a “Road Map” of a program. – Drawing a picture of expected program

achievements and how the achievements will be realized.

– Creating a visual of relationships hypothesized to exist between the program activities and the intended program effects

• Logic Models describe expectations/intentions of a program.

CDC Evaluation Framework

Resources/ Inputs Activities ImpactOutputs

OutcomesS/M/L

Theory of Action

Action & Change

Theory of Change

Program Theory-Driven Evaluation

Resources and Inputs

Activities Outputs Outcomes Impact

Planned Work Intended Results

Assumptions

Common Logic Model Components

Resources/Inputs

Resources needed to achieve

program’s objectives

Activities

What the program does with resources to meet

objectives

Outputs

Direct products of

program activities

Outcomes

Changes that result from the program’s activities

and outputs

External Factors/Context: Description of environment in which program takes place

Assumptions: The underlying assumptions that influence the program’s design, implementation or goals

Intermediate

Longterm

Shortterm

Constructing a Logic Model

• Move from right to left answering the following questions: – What do I want to accomplish with this program? – What changes do I expect to see from this program?– In whom or what will these changes occur?– What do we do to affect change?

• Then revise, refine, and more precisely describe and visually depict the relationships among components

• Connect components with arrows…to show flow• Describe the context in which your program resides &

operates

Logic Model Exercise• Get together in your teams and begin to

identify elements Remember to:

Start with desired outcomes – short list – consensus

Discussion of how outcomes might occur Leads to details of how the program is expected

to affect intermediate outcomes (know as proximal outcomes or mediators)

Draft of program theory/logic model is developed

Logic Model Analyses• What is your ultimate outcome?• What is your theory of change? Your theory of action?• What part of this model do you expect to evolve and on

what schedule? • Are the outputs indicators of activity?• Are the outcomes indicators of change?• Can you identify potential stumbling blocks?• What resources might be added?• Other ideas for improvement?

Step 2: Describe the Program

Logic Model Example HIV Prevention Program for Native Americans

CA Success Network (CA SN) Logic Model

California community colleges must expand their capacity to improve student outcomes in basic skills instruction, English as a Second Language, and related support services. Colleges report that 70-90% of entering students place into one or more basic skills course areas: reading, writing, and/or mathematics. The statewide course completion rate in basic skills is 60.5%, and only 50% of those students persist to and complete the next level course. Historically underrepresented students are over-represented in basic skills courses, and their success rates are frequently lower than the rates for other student groups. Community colleges' efforts to improve outcomes in basic skills cannot be separated from their responsibility to produce equitable graduation, certificate, and transfer rates for historically underrepresented and underprepared students.

Resources -ASCCC-Steering Committee-LACCD Fiscal Agent & Consortium Coordinator-BSI LACCD Project Director-BSI Regional Network Coordinators-Campus BSI Coordinators-Student Advocates-Campus BSI Action Plans-Sustainable information sharing sites

Target Population -Administrators-Faculty-Support Staff-Students

Activities

Collaborative Learning Networks

Coordination with core organizations (Alliance building among ASCCC, LACCD Project, Other Partners)-Guide statewide infrastructure development-Assist in securing funds for acquiring additional resources to support State & Regional infrastructure

Increased Learning, Success &

Persistence of Students

Short-Term Outcomes

Skilled Implementation of

Responsive, Innovative, Integrated Essential

Academic Skills Curriculum

Inputs

Create statewide infrastructure (Permanent learning network among 110 Colleges and a CCC Center)

-Establish statewide coordinating center-Guide the development of and coordinate & support a system of regional networks-Coordinate & support partnerships among professional development efforts-Provide vision & support for the development of innovative professional development activities -Introduce & engage in collaborative evaluation activities that generate knowledge about effective learning to transform practice

Develop a cadre of campus leaders-Conduct Summer Leadership Institute, ASCCC Workshops, Other Leadership Institutes & on-going professional development activities-Develop strategies to retain effective leaders-Guide annual revision of Campus BSI Action Plans-On-going evaluation of the process and impact of professional development activities

Promote models of effective practices-Build capacity of all faculty to teach students who are underprepared for learning in college courses-Develop Web-based, virtual teaching & learning communities, including a virtual resource library-Engage in collaborative evaluation activities

Formally engage key CCC policy

makers(e.g., APG groups,

Local Board of Trustees)

Innovative/ Alternative &

Engaging Methodologies /Pedagogy at

CCCs

Increased Success of Students’

Personal Goals, Completing

AA/AS & Certificate

Programs &Transferring to 4

Yr Colleges

Statewide Adoption of Policies that

Support Innovative/

Alternative & Engaging

Methodologies /Pedagogy at

CCCs

Create regional level infrastructure (networks)-Develop & support long-term on-going regional professional development training & supporting materials for the target population-Develop infrastructure for colleges to learn about & share innovative instructional programs -On-going evaluation of professional & program activities, including benchmark assessment & capacity building

Long-Term Outcomes

Local Policy Adoption

Supporting SN & Other BSI Initiatives

Sample Logic Model Frameworksource: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

PROFESSIONAL LEARNING RUBRIC

Professional Learning Rubric

• Develop Indicators of Change for Your Professional Learning Hub using Wenger’s Community of Practice framework and Impact Definitions through Value Creation Stories

Network Levels

Wenger’s Community and Network Values

Value Indicators/Evidence

Immediate Activities & Interactions

Potential CapitalApplied Practice/Implementation

Realized Performance Improvement

Reframed Redefined Success

Wenger, Trayner, de Laat (2011) Promoting and assessing value creation in communities and networks: a conceptual framework.

Immediate Value

• Activities and interactions (e.g., meetings, trainings, lunches)

• Intentional and unintentional• Knowledge exchange

Potential Value

• Enlightenment• Personal assets• Relationships• Resources• Collective• Transformation

Applied

• Leveraging networks• Applying a new teaching strategy• Forming an inquiry group• Applying for funding• Shifting pedagogy to recognize a different

domain of learning

Realized

• Impact, beyond changes in practice• Outcomes at various levels• Short-term, rather than long-term,

requiring leveraging beyond practice changes

Reframing

• Creating new definitions for success• Reconsidering measurements• New instruments• Can occur at every level of measurement

Rubric Development Exercise

• 15 Minutes• Appoint a spokesperson and a note-taker• Identify at least one entry for each row• Think about the stories presented Monday• Use your personal narrative work• Come back together and share out

Guiding Questions

1. What kinds of stories resonate?

2. What are the narratives that currently exist about community colleges, students, and faculty?

3. What language do we use?

4. Who are the characters?

5. Who is the protagonist? Antagonist?

Providing Insight to 3CSN

• Using the rubric, identify stories• Analyze and condense into categories• Provide stories “representative” or

“illustrative” of categories• Quantify the categories• Use survey tool to provide data

What Evaluation Is & Isn’t

Evaluation is… Evaluation is not…

Done with you Done TO you

Able to provide rich information

Simply program monitoring

Intended to be used Intended to sit on a shelf or to check a box

For the program stakeholders

For the evaluator or only for management

Systematic Haphazard

FUN! Scary (Really it isn’t!– you’ll see )

Evaluation, like anything else, can be done well and produce benefits or it can be done

poorly adding little value, sometimes having negative impact. The following table

contrasts characteristics of effective and ineffective evaluation.