Program Evaluation & Research In Service-Learning

Service-Learning Mini-InstituteLynn E. Pelco, Ph.D.

Division of Community Engagement

Objectives• Understand the importance of assessing student

learning and evaluating outcomes, including community benefit,

• Understand the difference between program evaluation and research,

• Know where to find tools to aid you with evaluation and research,

• Be aware of outlets for scholarship related to service-learning

What do we really mean when we say we’ll “assess” something?

Assessment vs. MeasurementAssessment is the process of documenting knowledge, skills, attitudes and beliefs. Measurement is the assignment of numbers to objects or events.

Evaluation/Program EvaluationA systematic method for collecting, analyzing, & using information to answer questions about the effectiveness & efficiency of a service-learning course/program.

Research/ScholarshipAn investigation or experimentation aimed at the revision of accepted theories in the light of new facts, or practical application of such new or revised theories.

Qualitative, Quantitative, & Mixed Methods ResearchQualitative research is a method of deep inquiry into the reasons underlying human behavior that typically does not utilize statistical/computational techniques. Quantitative research is an empirical investigation of social phenomena via statistical, mathematical or computational techniques. Mixed methods methodology integrates qualitative and quantitative techniques. Scholarship of Teaching & LearningSoTL is scholarly inquiry into student learning which advances the practice of teaching by making research findings public

In the context of service-learning…

Why do we do evaluation and/or research?Who are our stakeholders?What do we want to know?

How do we measure?When do we measure?

What do we do with the information?

Developing Research Projects Around Theories

Scholarship of Teaching & Learning Theories: Allport’s Contact Hypothesis, Lewin’s Change Theory, Discipline-based theories: What theories exist in the center of the intellectual dialogue within your discipline? What are the names of the top journals in your discipline? Over the past year, what theories appeared on the pages of these journals?

Newly published 2-volume book:Clayton, Bringle & Hatcher (2012) Research on service learning:Conceptual frameworks and assessments.

What might we evaluate?

Cost and Efficiency

Outcomes and Impact

Process and Implementation

Program Design and Theory

Needs

Getting Started:Developing a Logic Model

1. Goals (determines type of assessment)2. Assumptions (theoretical, conceptual)3. Inputs (resources – people, places, things)4. Activities (things that will occur)5. Outputs* (things that will be produced)6. Outcomes* (measureable changes – can be

short-term and/or long-term)

*How will you measure outputs and outcomes?

Examples of Measurement MethodsMethod Cost Labor Advantage Disadvantage

Commercial Tests High Low Benchmarking Not program-specific

In-House Tests Low High Direct and program-specific

No benchmarks

Simulations High High Clear behavioral measure

Development time

Focus Groups Low High Deeper examination Subjective

Observations Low High Behavioral specificity Rubric complexity

Surveys Med Med Targeted questions Indirect measure

Archival Data Low Low Many sources of data Retrospective

Portfolios Low High Multiple work samples Complex scoring

Course-Embedded

Low High Course-specific; individual feedback

Restrictive sampling; feedback time

Logic models (a.k.a., logical framework, theory of change, or program matrix) are graphical depictions of the logical relationships between the resources, activities, outputs and outcomes of a program.

Logic Models

Inputs Activities Outputs Outcomes

Statement of Goals

Each resource involved

Each major activity that will take place

Each “item” to be produced and how it will be measured

Each potential change and how it will be measured

Disseminating Results• Internal stakeholders

– Your own students– Prospective students– VCU faculty and administration

• External stakeholders– Your community partners– Greater Richmond community– The academic community

Presenting and Publishing Results• Many avenues for presenting and publishing

outcomes associated with service-learning• Remember that publication of research involving

human subjects requires approval from VCU’s Institutional Review Board (IRB)

• For more information and resources, visit– http://www.community.vcu.edu/research/disseminating-your-res

earch/publishing-outlets/

– http://www.servicelearning.org/what-you-do/conduct-research

In your groups: 1. Pick a specific service-learning course or experience

that one of you is considering. 2. Decide what aspect you want to assess/evaluate.3. Draft a model to help you plan your strategy.

Build a Logic Model

Inputs Activities Outputs Outcomes

Statement of Goals

Each resource involved

Each major activity that will take place

Each “item” to be produced and how it will be measured

Each potential change and how it will be measured

Questions?