Embed Size (px)
Citation preview
How To Evaluate Programs/Interventions?
Step 1 – Identify independent and dependent variables/concepts
Independent – InterventionDependent – Stated objective of
program
Step 2 – Define these variables/concepts
Step 3 – Develop indicators (i.e., questions) to measure these variables/concepts
Defining Variables/Concepts
Challenges with definitions
Concepts may be multi-dimensional
Concepts can be value laden
Concepts can have both a subjective and objective definition—you must choose one
Only concepts that VARY (are also variables) can be used
Examples of Multi-DimensionalConcepts
Social class◦Social class has an economic, occupational
and education component
Prejudice◦Although we often speak of prejudice as one
term, there are many different types of prejudice (i.e., racial prejudice, social class prejudice, religious prejudice, etc.). These types comprise different components of prejudice.
Examples of Value Laden ConceptsPrejudice – not acceptable to have these types of
attitudes◦ Are you prejudiced?
Child abuse – not socially acceptable to engage in these types of behaviors or admit having been a recipient◦ Have you abused your child today?◦ Have you ever been abused?
Sexual assault – negative stigma is often associated with having been raped◦ Have you ever raped anyone?◦ Have you ever been raped?
Measurement Error Results from Value Laden Concepts
Social desirability◦ There are socially desirable answers to questions
Example – If you asked the question “Have you ever abused your child?” the socially desirable answer would obviously be no
Acquiescence bias◦ Positive vs. negative answers
Example – If people are evaluating a program and they feel connected or grateful to those in charge, they will positively evaluate every item
Leading questions◦ Questions that predispose respondents to answer a
certain way Example - Rape is a heinous crime. Do you agree?
Subjective vs. Objective Definitions of ConceptsSocial class – objective (as defined
by others)◦Family income, level of education,
employment
Social class – subjective (as perceived by the respondents)◦Ask – What is your social class?◦Ask – Would you say you are poor?
When a Concept is a Variable, It can Vary across time
Vary across place
Vary across individuals or groups
Examples of Concepts that VaryAcross time
◦Crime rates vary across age◦ Annual income varies across age/life span
Across place◦Crime rates vary across rural vs. urban areas◦Median incomes vary across rural vs. urban
areas
Across individuals/groups◦Crime rates vary across gender◦ Income varies across gender
Examples of Concepts that Do NOT Vary
Across time◦ Sex – if you are born a male, you will die a male◦ Eye color – generally you have same eye color all your life
Across place◦ Impact of frost on plants – generally if it freezes, then plants
die no matter where you plant them◦ Whether or not babies are born with teeth does not vary
across place◦ Likelihood that children will be born speaking Chinese
Across Individuals/Groups◦ Impact of restful sleep – generally you are healthier if you
sleep well - no matter who you are◦ Natural hair color doesn’t vary across gender◦ IQ scores don’t vary across gender
You should (1) take a concept, (2) define it as a variable, and (3) develop an indicator to measure it
Concept Variable Indicator
Income Annual earnings What was your total income?
Race Self identification of Race
What race are you?
Poverty
Crime Number of Crimes Committed up to age 55
How many felonies have you committed in your lifetime?How many misdemeanors have you committed in your life time?
Variables can be measured at different levels—These include:
LEVEL
Definition Variable Example
Nominal
Measured in categories. There is no rank order to the category
Gender Measured as male or female
Ordinal There is a rank order, but the distance between ranks is not known.
Placement of vegetables grown at county fair.
Squash may be judged to be a grand prize winner, first place, second place, etc.
Interval There is a rank order, and distance between levels is known.
Income Level Ask respondents to indicate their income category: 0-19,999; 20,000-39,999 etc.
Ratio There is a rank order, and the distance between levels is known. There is an absolute zero.
Age Measured as number of years old. Ranges from 0 to about 107.
You should always measure variables at the most precise level possible
LEVEL Definition Variable
Example
Nominal No Rank OrderCategorical
Gender What is your gender(1) Male (2) Female
Ordinal Rank Order Distance not Known
Satisfaction with Services
I am satisfied with your services. (1) SD (2) D (3) NAND (4) A (5) SA
Interval Rank Order Distance has Meaning
Education Level
How many years of formal education do you have?(1) 0 – 4(2) 5 – 9(3) 10 – 14
Ratio Rank OrderDistance has No MeaningAbsolute Zero
Family Size How many children do you have living in your household?
The same variable can be measured at different levels—For example…
LEVEL Definition Variable Example
Nominal No Rank OrderCategorical
AGE Are you a “baby boomer”? 1. Yes2. No
Ordinal Categories are rank ordered, but the distance between levels is not equal and not known.
Age How old are you?1. Young Adult2. Middle aged3. Elder
Interval Rank Order Distance has Meaning
AGE How old are you?1. 20-292. 30- 393. 40- 49 etc.
Ratio Rank OrderDistance has No MeaningAbsolute Zero
AGE How old were you at your last birthday?_____ years old.
Always measure variables at the highest level possible. For example, whenever possible measure age at the ratio level LEVEL Definition Variable Example
Nominal No Rank OrderCategorical
AGE Are you a “baby boomer”? 1. Yes2. No
Ordinal Categories are rank ordered, but the distance between levels is not equal and not known.
Age How old are you?1. Young Adult2. Middle aged3. Elder
Interval Rank Order Distance has Meaning
AGE How old are you?1. 20-292. 30- 393. 40- 49 etc.
Ratio Rank OrderDistance has No MeaningAbsolute Zero
AGE How old were you at your last birthday?_____ years old.
Now you tryConcept Variable Indicator
Crop Yield # of bales of hay per acre
Educational level # of high school graduates in group
OR the # of years of education completed for participants.
Satisfaction with 4-HWorkshop
Level of satisfaction withtheir experience.
Leadership skills Subjective evaluation of own leadership skills*Leadership skills is a multi-dimensional concept, and implies a
combination of various skills. It would therefore be wise to measure it using multiple indicators.
Age – more detailed example Categorical/Nominal
◦ Are you a baby boomer? (1) yes (2) no Ordinal
◦ Which of the following would describe you? Child Teenagers Young adult Adult
Interval◦ How old are you?
10-14 15-19 20-24
Ratio◦ How old were you at your last birthday? ___ years
Always measure variable at highest level possible
Income – more detailed example Categorical/Nominal
◦ Are you a poor? (1) yes (2) no
Ordinal◦ Rank you income relative to other college students
Much below average Average Below average Somewhat above average Somewhat below average Above average
Much above average
Interval◦ What was your annual income for 2008?
Less than 10,000 20,000 to 29,999 40,000 to 49,999 10,000 to 19,9999 30,000 to 39,999 50,000+
Ratio◦ How much money did you make last year? _____ dollars ______
cents
Always measure variable at highest level possible
Now you tryConcept Variable Indicator
Crop Yield # of bales of hay per acre
Educational level # of high school graduates in group
OR the # of years of education completed for participants.
Satisfaction with 4-HWorkshop
Level of satisfaction withtheir experience.
Leadership skills Subjective evaluation of own leadership skills
*Leadership skills is a multi-dimensional concept, and implies a combination of various skills. It would therefore be wise to
measure it using multiple indicators.
Possible AnswersConcept Variable Indicator
Crop Yield # of bales of hay per acre Ask farmers –How many acres of hay per acre did you harvest this year?
Educational Level
# of high school graduates in group
Ask high school administrators –How many of the participants graduated from high school?
OR the # of years of education completed for participants.
Ask participants –What is the highest level of education you have completed?
Satisfaction with 4-H Workshop
Level of satisfaction withtheir experience.
Ask respondents –Indicate your degree of satisfaction with workshop
Leadership Skills
Subjective evaluation of own leadership skills
Ask respondents –Please indicate the extent to which you agree with the following statements.1. People listen when I talk2. Others respect my
suggestions3. People think I have good ideas
Questions or comments?Please contact:
Carol AlbrechtAssessment Specialist
USU Extension979-777-2421
