The schedule moving forward…• Today

– Evaluation Research – Remaining time = start on quantitative data analysis

• Thursday– Methodology section due– Quantitative analysis continued (start Exercise 10)

• Next Tuesday – Exercise 10 Due– Maahs returns Exercise #9

• Research proposal (everything is now online) is due on Dec 21st (Monday)

Evaluation Research

• PURPOSE rather than specific method• Much more popular in last 20 years• A form of “applied” research • Results, though designed to impact decision-

making, often have no impact

Types of Evaluation Research

• Needs assessment• Cost-benefit study• Monitoring study • Program evaluation

Doing evaluation (program) research

• Research questions / measurement as Job #1– Political context how do measure – Operationalize “outcome” of interest

• “Response” variable • NEED AGREEMENT: what is the program trying to do?

– Operationalize “processes” • Intermediate objectives

– Context of – ”

Research designs for evaluation

• Experimental– Problems, ethics (placebo)– The “black box” issue– Quasi-experimental designs

• Time Series (multiple time series)• Qualitative evaluations

– Low birth weight study

Evaluation Research ISSUES

• Evaluation research as “MESSY”– Administrative control, context of “real life”

• Importance of looking at “black box”

• Ethics– The intervention itself can raise issues– Control group members– Violating Ethics 101 (Tuskegee) – Evaluation impacts people’s lives

The (non) use of findings

• Purely rational/scientific world vs. the one we inhabit– Nixon panel on porn– Scared Straight – 3 Strikes and You’re Out legislation

• Studies gone wrong…– Fire the evaluator (or dismiss as pointy headed idiot) and

keep the program– Why not trust/use the results?

• Scientist/practitioner gap• “True believers”• Vested interests

Bivariate Analysis

Backed up with a little inferential statistics……yeah baby!

Review

• Descriptive statistics– Purpose?– Types?– These are “univariate” statistics

• Explanatory research– Attempt to demonstrate cause-effect

• Necessary Criteria?

Demonstrating Associations

• Bivariate (2 variables) analysis– There are a number of ways to do this

• The method you choose depends largely on characteristics of the two variables

– Levels of Measurement?

Contingency Tables (cross tabs)

• Some find these very intuitive…others struggle– It is very easy to misinterpret these critters

• Convention: the independent variable is on the top of the table (dictates columns) and the dependent variable is on the side (dictates rows). – What is in the individual “cells?”

• Frequencies (number of cases that fit criteria)• Convert to Percentages: a way to standardize cells and make

relationships more apparent

Example

• A survey of 10,000 U.S. residents• Research question: is one’s political view

related to attitudes towards police?– What is the DV and IV?– In constructing a table, what goes where?

An Example

Attitude TowardsPolice

Political Party Total

Repub Democrat Libertarian Socialist

Favorable 2900 2100 180 30 5210

Unfavorab 1900 1800 160 28 3888

Total 4800 3900 340 58 9098

The Percentages of Interest

Attitude TowardsPolice

Political Party Total

Repub Democrat Libertarian Socialist

Favorable 2900 (60%)

2100(54%)

180(53%)

30(52%)

5210

Unfavorab 1900 1800 160 28 3888

Total 4800 3900 340 58 9098

Inferential Statistics

• Researchers are typically not interested in whether there is a relationship in the sample– Want to know about the population– Why might there be a difference between what

you find in the sample and what actually exists in the pop?

• Even with a probability sample, there is always sampling error

• Without a probability sample = error + bias

Inferential Statistics II

• Cannot assume that the relationship in your sample is true in population– BUT: probability theory allows us to estimate:

“The likelihood of obtaining a particular finding if in the population, there was no relationship” OR

“The likelihood of obtaining a particular finding assuming the null hypothesis”

Test Statistics and Significance Tests

• What does “statistically significant” mean?– Not due to sampling error

• What do you need to do to test for statistical significance?

1. A “test statistic” • An indicator of how different the sample is from “no

relationship”

2. The level of error that is acceptable (.05, .01)3. Degrees of Freedom

The Test Statistic for Contingency Tables

• Chi Square, or χ2 – Calculation

• Observed frequencies • Expected frequencies

– Can get confusing calculating these critters by hand

– Intuitive: how different are the observed cell frequencies from the expected (under null hypothesis) cell frequencies

– Degrees of Freedom:• (# of Rows -1) (# of Columns -1)

Political View (3 Category) * Respondent's Sex Crosstabulation

157 229 386166.9 219.1 386.0

215 312 527227.9 299.1 527.0

252 278 530229.2 300.8 530.0

624 819 1443624.0 819.0 1443.0

CountExpected CountCountExpected CountCountExpected CountCountExpected Count

Liberal

moderate

conservative

Political View(3 Category)

Total

Male FemaleRespondent's Sex

Total

The Chi-Square Sampling Distribution (Assuming Null is True)

Conventional Significance Testing

1. Calculate the test statistic2. Set your “Type II” error level

– The risk of being wrong that you are willing to live with

3. Find the “critical region” within the distribution for your sample statistic

– How far out on the curve does your test statistic have to get before you can reject the null hypothesis

4. Decide whether to reject, or fail to reject the null hypothesis

Significance Testing in SPSS

• Within your crosstabs, select “Chi-Square”– SPSS gives you the χ2 value

• What you would calculate based on the observed and expected cell frequencies

– SPSS also gives you “p”• The exact probability of obtaining this χ2, if indeed the

null hypothesis was correct• In newer versions of SPSS, the p values are listed under

“sig” or “significance” or something similar

Review of Contingency Tables

• Level of Measurement– Both IV and DV are Nominal or Ordinal

(Categorical data)• Constructing a table

– IV on top (columns), DV on bottom (rows)• With this format, select “column percentages”

Review of Inferential Statistics• Purpose of inferential statistics

– Figure out the odds that a finding from sample is due to “chance” or “sampling error”

• Process – Assume null hypothesis is correct

• Calculate the odds of obtaining your particular finding under this assumption

• If the odds are very low, you may be suspicious that the null hypothesis is incorrect

• At some point (research sets level), you say that the odds are so low that you are going to go ahead and reject the null hypothesis

Review of Chi-Square

• A measure of how different observed (from your sample data) cell frequencies are from what would be expected under the null hypothesis (e.g., no relationship)– The Chi-square distribution changes shape with

different degrees of freedom• This is because, by definition, the more cells you have,

the higher χ2 gets– df = (R-1)(C-1)

Interpreting Chi-Square

• Chi-square has no intuitive meaning, it can range from zero to very large– The real interest is the “p value” associated with

the calculated chi-square value– This is the exact probability of obtaining the chi-

square if in the population there was no relationship

• In other words, the exact probability of finding that chi-square, under the null hypothesis