Incorporating an Evaluation Plan into Program Design:Using Qualitative Data

Connie Baird Thomas, PhDLinda H. Southward, PhD Colleen McKee, MS

Social Science Research CenterMississippi State University

Positioned for Progress Conference April 30, 2014Jackson, MS

Workshop Goals

• Understand the difference between qualitative and quantitative data

• Learn how to capture qualitative data in a rigorous objective way

• Explore best practices related to qualitative data collection

PROJECT ABC – Literacy Intervention Program

Conceptualize—“If we do X, then Y will result”

ACTIVITIES: (X) Pilot program in 10 schools Group tutoring One-on-one tutoring Curriculum Take home materials Parent participation

Trained facilitators

OUTCOME (Y): Proficient reading by fourth grade

What is Qualitative Research?

Data collection method in which the researcher:• Seeks an in-depth, “rich” view of topic• Explores an idea or topic to describe or give it

meaning• Asks participants broad, general questions• Collects detailed views of participants or subject• Analyzes and codes data for description or

themes

Characteristics of Qualitative and Quantitative Research

Qualitative• Goal: describe, discover• Design: flexible, emergent• Sample: small, non-random• Data collection: interviews,

observation• Data interpretation:

subjective

Quantitative• Goal: predict, confirm• Design: structured• Sample: random, large

representative• Data collection: surveys,

questionnaires• Data interpretation:

objective

Qualitative Research …

• Does – Explain “how” and

“what”– Describe “what

happened”– Explore processes over

time– Explore meaning of

event to those involved

• Does not– Prove/disprove

hypothesis– Measure variables– Test theories– Show relationship

between variables– Make generalizable

statements

Methods of Qualitative Research

Methods

• In-depth Interview

• Focus group

• Case Study

• Direct Observation

Data“Words don’t fly individually- they fly in flocks.” Geoffrey Nunberg- University of CA, Berkley School of Information

MethodologyIn-depth Interview

• Advantages• Open structure allows for individual to

express detailed opinions and experiences• Interviewer can easily follow-up and/or

clarify statements • Appropriate for sensitive or controversial

topics or special populations

• Limitations• Requires skilled interviewer• Responses may be biased• Not generalizable• Time/labor intensive

One-on-one conversation between interviewer and respondent regarding a specific topic

Sometimes audio/video recorded

Methodology

Focus Group

Advantages• Group setting often stimulates

additional thoughts and ideas• Gives more insight into why certain

opinions are held• Can better understand processes

(changes that have occurred over time)• Observe nonverbal responses

Limitations• Group/time management • Peer pressure among respondents• May stress resources

Focused small group (usually 6-10 participants) discussion guided by a leader or moderator

Useful for populations with shared experiences

Methodology

Case Study

Advantages• Provides rich, detailed information• Lends to understanding of processes,

interactions and context• Uses different methods of data collection

(interviews, observations, document analysis, etc.)

Limitations• Applicable only to site or event under study

(limited generalizability)• Requires extended time and effort

In-depth study of a single unit- can be a person, organization, program, etc.

Sometimes referred to as “telling a story”

When to Use Case Study*

• Is the program is unique?• Is it premature to establish impact

measures?• Is there unexplained variation in

program impacts?• Will understanding the program

implementation in detail help design future programs?

*E. Balbach. Using Case Studies to do Program Evaluation. (1999) California Department of Health.

MethodologyDirect Observation

Advantages• Helps to understand the social, physical,

political, and economic context in which participants live

• Can provide important information that was previously unknown

• Observe participants in their natural settings• Complements other forms of data collection

(triangulation)

Limitations• Time consuming/resource intensive• Difficult to document everything- may have to

rely on memory• Objectivity is a challenge (observer bias)• Observer effect

Observing the participants in their natural environment and setting- either as becoming a part of the setting or being strictly an observer

Issues to Consider

• Purpose of research (re: evaluation)• Subjective method (reliability and validity)• Researcher bias• Labor intensive• Cost

Words fly in flocks…

Example:

Assessing the Impact of the Mississippi Healthy Student Act: Year Four Report

Questions

Contact us at:mskidscount@ssrc.msstate.e

duwww.kidscount.ssrc.msstate.

eduTel: 662-325-7127

Connie Baird Thomas, PhD.Tel: 601-407-2742Email: connie.baird@ssrc.msstate.edu

Linda H. Southward, PhDTel: 662-325-0851Email: linda.southward@ssrc.msstate.edu

Colleen McKee, MSTel: 662-325-0851Email: colleen.mckee@ssrc.msstate.edu

Thank You