September 2007

Survey Development

Rita O'SullivanEvaluation, Assessment, & Policy

Connections (EvAP)School of Education,

University of North Carolina-Chapel Hill

ritao@unc.edu or 919/843-7878

September 2007

Overview

• Evaluation Theory & Practice

• Elements of an Evaluation Plan: Logic Models

• Evaluation Instruments• Q & A

September 2007

Surveys for a Purpose

• What information is needed?• Who needs the information?• How will it be used?• Who has the information you

need?• What’s the best way to get it?• What resources are available?

September 2007

Evaluation Planning - Logic Models

Goals and Objectives

ActivitiesPerformance Measures/Outcomes

Monitoring Evidence

Outcome Evidence

What do you want to do?

What purposes does the

program serve? What does it

seek to accomplish?

Activities to implement to achieve the

stated goals & objectives

Expected outcomes or

results for each activity. Answers

the question “Why are we implementing this particular activity? What

do we expect the outcome to be?”

Language in this column often reflects change:

increase x or

decrease y.

Evidence of

Activities & Quality

Instruments and forms

used to collect data which will

demonstrate that the activities

have occurred

Evidence of Results

Evidence they

collected to demonstrate

that the specified outcomes have been achieved

September 2007

Guide to Instrument SelectionData Collection Strategy Key Advantages

Knowledge tests: Assess content knowledge

Limited response Can cover large amount of content domain

Open-ended Can assess higher order cognitive skills

Performance Assessment Can assess actual behaviors

Opinion Surveys: Assess opinions

Written Written record of responses

On-Site Usually good response rates

Mail Can contact people from different geographic areas

Email Ease of survey distribution

Web-Based Ease of survey distribution and data summary

Telephone Can contact people from different geographic areas

Group Cost effective; Can assess degree of consensus

Face-to-Face Can probe for more detail; Respondents can ask questions

Attitude & Interest Surveys:

Assess degree of attitude or interest; Easy to score

September 2007

Logic Model –All Stars

GOALS & OBJECTIVES ACTIVITIES

Performance

Measures/Outcomes

MONITORING EVIDENCE

OUTCOME EVIDENCE

Decrease high school students drug use

Introduce All Stars, Sr. curriculum in HS health education classes

Students will report increased awareness of problematic substance abuse

Student survey

Student attendance records

75% of students will report reduced drug use as measured by pre-post survey

90% of students will report increased awareness of substance abuse as measured by pre-post survey

Improve drug education in HS health ed classes

Introduce All Stars, Sr. curriculum in HS health education classes

Increase teacher capacity to conduct effective drug education curriculum

Video analysis of classroom lessons

Lesson plan logs

90% of participating teachers will use All Stars Curriculum

September 2007

Validity and Reliability

o VALIDITY: Is the degree to which the assessment is appropriate to its purpose.

o RELIABILITY: Will this assessment get the same response every time.

o TRIANGULATION: Verifying findings through multiple data sources.

September 2007

Sampling

Need for Explanation/Rich Description

Issue Complexity

Difficulty around Obtaining Information

QUALITATIVE DATA COLLECTION

QUANTITATIVE DATA COLLECTION

LOW

HIGH

Lack of Information about the Topic

September 2007

Types of Sampling• Simple Random• Stratified • Cluster• Systematic• Purposeful• Convenience• Snowball

September 2007

Response Rates• Factors that influence response rates

SaliencyRemindersEase of response

• Increasing sample size will not increase response rates

September 2007

Ensuring Data Quality• Sample

representativeness?• Consistent across

individuals?• Appropriate to purpose?• Collected in a culturally

appropriate manner?

September 2007

Draft Program Student Survey

6.How strongly do you agree or disagree with each of the following?

Strongly

disagree

Disagree

Neither agree or disagree

Agree

Strongly

agree

a. This program was excellent.

O O O O O

b. This program helped me prepare for a career and life-long learning.

O O O O O

c. This program didn’t decreased my interest.

O O O O O

d. Because of this program I have improved my skills.

O O O O O

e. I would recommend this program to others.

O O O O O

September 2007

Sample Focus Group Protocol• SCALE wants to know how the Student to Student program is working and what impact it

is having on your campuses and with your community partners. They have asked us to conduct this focus group to begin exploring these areas. After we talk, we will summarize what you’ve said to capture the essential points of our discussion; we will make sure that individuals cannot be identified from the summary. Before we share the summary with SCALE, we will share it with you to make sure that what we’ve summarized fits with what you intended to say. If no one objects we’d like to record this focus group so that we won’t lose any detail of the discussion.

• Start-Up/Campus Recruitment• What kinds of expectations are realistic and reasonable in terms of initial campus start

up?• What barriers to recruitment of community partner(s) did campuses face? • What barriers to recruitment of tutors did campuses face?

• Tutor Training:• What worked in terms of the initial tutor training SCALE did on your campuses?• How can this initial tutor training be improved?• What kinds of expectations are realistic and reasonable in terms of initial tutor training?• What ongoing training do tutors need?• What ongoing tutor training opportunities are being offered at campus sites?• How, if at all, is the tutoring manual being used during ongoing tutor training?