Center for Research & Innovation in STEM

Education & American Society for

Engineering Education

Understanding Project Evaluation

Design, Implementation, and Use

April 13, 2017

B. Jan Middendorf, Ph.D.

Feed the Future Innovation Lab for Collaborative Research

on Sustainable Intensification (SIIL)

Kansas State University

jmiddend@ksu.edu

Evaluation in Practice

Understanding Theory and Roots

Program Evaluation Standards

Theory Driven Model

Asking the Right Questions

Tools to Address the Questions

Communicating Results

positive & negative

Evaluation Theory Tree*

* Evaluation Roots: A Wider Perspective of Theorists’ Views and Influences edited by Marvin C. Alkin (2013)

Program Evaluation Standards*

Utility

Feasibility

Propriety

Accuracy

Evaluation Accountability

* Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011).

The program evaluation standards: A guide for evaluators and evaluation users

(3rd ed.). Thousand Oaks, CA: Sage

Evaluation Options and Types

Patton catalogued 80 different ways of

focusing evaluations

Accountability, accreditation, appreciative

inquiry to utilization-focused evaluation

(Patton, 2008c, pp. 300-305)

Scriven created the evaluation’s

Thesaurus that identifies evaluation

concepts and terminology for clarity in the

field (Scriven, 1991)

General Expectations for Evaluation:

Independent, qualified evaluation professional

Evaluation based on scope of program

Well-articulated Evaluation Plan

Based on a logic model / program theory

Utilizes a robust, appropriate evaluation design

Includes formative and summative approaches

Identifies measures that are tied to expected

outcomes

Demonstrates impact of program

Includes evaluation of intellectual merits and

broader impacts (NSF specific)

Purpose of Logic Models

What a logic model is:

Identifies the theories and assumptions underlying the

program

Links outcomes (short and long-term) with program

activities/processes

Answers the questions:

What are the intended outcomes?

How will we get there? (activities and resources)

How will we know if we have succeeded (impact)?

What we doConduct

courses,

workshops,

meetings

Deliver

services

Develop

products,

curriculum,

resources

Train

Provide

counseling

Assess

Facilitate

Partner

Who we

reach

Participants

Customers

Clients

Students

Agencies

Learners

Decision-

makers

Reactions

Learning

Results

Awareness

Knowledge

Attitudes

Skills

Opinions

Aspirations

Motivations

Ultimate

Impact(s) or

Conditions

Social

Economic

Civic

Environmental

Value-added

Action

Results

Behavior

Practice

Decisions

Policies

Social action

What we

invest

Staff

Volunteers

Time

Money

Research base

Materials

Equipment

Technology

Partners

Outputs

Activities Participation

Outcomes – Impact

Short Term Medium Term Long Term

Logic Model Example

Inputs

External FactorsAssumptions

EVALUATION Formative Summative

Importance of Logic Models

Brings clarity to project goals

Presents a logical path

Summarizes key elements of a project

Communicates the theory of action

Supports the identification and development of

evaluation plans

Designing Project Evaluations

Project Goals: What are the goals of your project? What is the project designed to

accomplish?

Project Strategies: What activities or strategies are used to accomplish or move toward

those goals?

Project Evaluation: How will you know if your project is successful? How will you know if

the intended outcomes are achieved?

What are the intended and unintended outcomes of the project?

How will you know what you should change or improve in your project?

Evaluation Types

FormativeGenerates information for project improvement

Provides a continuous feedback loop

Ensures flexibility and responsiveness to modifications

Summative:Assesses the quality, outcomes, and impact of a fully

implemented project by collecting information about the

related processes, strategies, and activities that led to

the results.

Supports broader impacts and dissemination

Provides evidence for accountability and decision-making

Effective Evaluation Plans

► Develop at the beginning of the project with the

project team

► Essential to involve stakeholders in the

identification of relevant questions and indicators

► Need to determine how you are going to use the

data to be collected

► Focus on questions of critical interest

Continuing the Discussion

What have you learned about evaluation in

the course of your work?

What modifications have you made in your

project implementation from your

evaluation results?

Choosing Evaluation Methods

Ask what methods will best help you

answer your critical evaluation questions

Consider both quantitative and qualitative

methodsExamples: Surveys, social network analysis, interviews, focus

groups, case studies, document review and content analysis,

observations, site visits, checklists, etc.

Triangulate data for more robust findings

where possible

Linking Evaluation Results to Research Read literature that enables one to see connections, move from local

examples to broader questions, and situate evaluation questions

within larger contexts.

Frame questions from the point of view of advancing knowledge

(research) versus asking questions inform and assess project

development, outcomes, and institutional practice (evaluation).

Use a conceptual framework to view issues and questions (which

helps to situate what one is studying in a context beyond a single

case). Highlight key questions, relationships, factors, and key

variables.

Take note of emerging questions and observations that may lead to

broader research. What observations confirm or contradict the related

literature?

Evaluation in a Research Study

Be innovative in considering how to

ensure the project is “on track” and to

learn from the project

Suggestions:

Advisory Board

Critical reviewer or reader of plans and results

Integrate internal and external evaluation

efforts

Linking Evaluation Results to

Broader Impacts and Dissemination

Recognize the responsibility to communicate what

has been learned to relevant audiences.

Consider various ways to share what has been

learned--writing, speaking, editing, presenting, and

using information technology in ways that are

accessible and meaningful to the relevant audiences

and stakeholders.

Consider who would benefit from what has been

learned.

Link judgments and conclusions with evidence.

Challenges in Evaluating

Broadening Participation Projects*

Pipeline as metaphor

Importance of cultural context & individual

circumstances

Limited understanding of broadening

participation literature & strategies

Identifying Evaluators with Expertise

Professionals are critical for quality evaluations

Evaluation not fully integrated with project

*Presented by Melvin E. Hall, Professor at Northern Arizona University

Summary and Take Aways

Understand the complexity and value of

evaluation within your projects

Ensure high quality professionals

Engage professionals at the beginning (proposal

development), middle (formative), and the end

(summative)

Utilize findings (positive or negative) to inform

future projects and the program overall

Evaluation Resources

Questions / Clarifications

►Questions/Clarifications

►For more information contact :

B. Jan Middendorf, jmiddend@ksu.edu

Instrumentation & Other Resources

Alliance for Graduate Education and the Professoriate http://www.nsfagep.org/evaluation-resources/

Student Assessment of Learning Gains (SALG): http://www.salgsite.org/

Undergraduate Research Student Self-Assessment (URSSA) http://www.colorado.edu/eer/research/undergradtools.html

Field-Tested Learning Assessment Guide (FLAG): http://www.flaguide.org/

Online Evaluation Resource Library (OERL) http://oerl.sri.com/home.html

Reference Materials Patton, M.Q. (2008) Utilization-focused evaluation (4th ed.). Thousand

Oaks, CA: Sage.

National Science Foundation

www.nsf.gov/pubs/2002/nsf02057/nsf02057.pdf

www.nsf.gov/pubs/1997/nsf97153/start.htm

http://www.evalu-ate.org/wp-content/uploads/formidable/Doc_2010_NSFHandbook.pdf

W. K. Kellogg Foundation

http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

Western Michigan University Evaluation Center Checklists

https://www.wmich.edu/evaluation/checklists

University of Wisconsin www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html