1/27/15  

1  

ALTERNATIVE RESEARCH DESIGNS

“Quasi” Experiments • Like experiments, but do not possess… • Experimental control

•  Participants may experience differences other than in the IV

• Randomization •  Participants usually all assigned to

same condition

• Biggest difference… • Quasi-experiments don’t contain

a control group

“Quasi” Experiments • Sometimes real experiments aren’t feasible

• Examining effects that rely on person-factors •  Cannot randomly assign people to be “extraverted,”

“depressed,” “adopted,” “a pack-a-day smoker,” etc.

•  Any experiment examining person-factors as IVs is not a true experiment • Because such factors cannot be randomly assigned

• Having a control group may be unethical •  E.g., giving ill people a placebo

medication

1/27/15  

2  

Types of Quasi-experiments • Single-case Design

• Unlike case studies… •  Case studies just describe behavior, life events, etc.

•  Single-case designs measure the effect of specific IVs on specific DVs (like an experiment)

•  “Single-case” doesn’t mean examining only one person •  The book discusses this as being synonymous with case-

studies; however, most single-case research examines multiple individuals

•  “Single-case” means participants are analyzed individually, instead of only paying attention to group averages

Single-case Design • Most often involves measuring DV before and after the experimental manipulation • Before manipulation = Baseline period

• After manipulation = Treatment period

• E.g., Mayberg et al., 2005

•  Treated 6 severely depressed patients with “deep brain stimulation”

•  Took multiple measures of depres-sion before and after treatment

Single-case Design • E.g., Mayberg et al., 2005

Dep

ress

ion

Sco

re

1

10

Baseline Treatment

5

Benefitted

No benefit

Individual cases

1/27/15  

3  

Single-case Design • E.g., Mayberg et al., 2005

• Quasi-experiment because…

•  Participants weren’t randomly assigned to conditions

• No control group

• Possible confounds?

•  Placebo effect

•  Just “being treated” lessened symptoms

Possible Confounds • E.g., Mayberg et al., 2005

• Possible confounds? •  Maybe 4 out of 6 patients got better on their own

• Typically 60% of people overcome mental illness without any kind of treatment • However, this usually takes longer than a few weeks

• Also, all patients had severe and previously untreatable depression

•  Baseline measures can affect patients’ depression

• Having to report all of their symptoms may motivate participants to overcome their depression

Dealing with Confounds • One way to rule out some of these confounds…

• Reversal Design – after treatment is shown to have an effect, remove treatment to see if participants return back to baseline •  Known as “ABA Design” (A = Baseline, B = Treatment)

• E.g., Mayberg et al., 2005

•  Took one patient and, without his knowledge, deactivated deep brain stimulation

•  Patient returned to baseline within 2 weeks

A B

A

1/27/15  

4  

Dealing with Confounds • Reversible Designs

• Not always ethical, especially when treatment is beneficial

• So, most treatment programs employ ABAB Design

• E.g., Mayberg et al., 2005

•  After patient returned to baseline, deep brain stimulation was re-started (again without his know-ledge)

•  He quickly returned back to treatment levels

A B

A B

Single-case Designs • Characteristics:

• Examines participants individually •  As opposed to group averages

• Most employ baseline-treatment designs •  ABA or ABAB designs

•  Lack crucial characteristics of true experiments •  Experimental control, random assignment, control group

• Most often employed to test effectiveness of treatments

Types of Quasi-experiments • Single-group Design

• Similar to single-case designs, but group averages are analyzed

• E.g., Effect of speed limits on traffic safety

•  Natural experiment: Because of oil crisis, U.S. government mandated 55mph speed limit on all highways in 1974

•  This was fully reversed in 1995 (return to baseline)

1/27/15  

5  

Single-group Design • E.g., National change in speed limit

• Does raising the national speed limit increase fatalities?

40000

40500

41000

41500

42000

42500

1994 1995 1996

Repeal

Fata

litie

s in

Driv

ing

U.S

.

Single-group Design • E.g., National change in speed limit

• Possible confounds? •  Historical changes

• E.g., Cars got bigger, more cars on the road

•  Changes in population

• E.g., different driving population, more drivers on the road

•  Random fluctuation in number of fatalities

36000 37000 38000 39000 40000 41000 42000 43000 44000 45000 46000

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000

Single-group Design • E.g., National change in speed limit (a wider look)

• Did speed limit change really increase fatalities? Repeal

Fata

litie

s in

Driv

ing

U.S

.

1/27/15  

6  

Possible Confounds • No true control group

• Comparison groups (55mph vs. 65mpg) are one year separated, so many things could have changed

• All single-case/ group studies involve examining the same population across time • Results can be confounded by changes in sample over

time

…versus…

1994 1995

Time as a Confound • Fatalities across time:

34000

36000

38000

40000

42000

44000

46000

48000

1982

19

83

1984

19

85

1986

19

87

1988

19

89

1990

19

91

1992

19

93

1994

19

95

1996

19

97

1998

19

99

2000

20

01

2002

20

03

2004

What happened in late ‘80s / early ‘90s?

Inclusion of air bags in new cars

Time as a Confound • Fatalities across time

34000

36000

38000

40000

42000

44000

46000

48000

1982

19

83

1984

19

85

1986

19

87

1988

19

89

1990

19

91

1992

19

93

1994

19

95

1996

19

97

1998

19

99

2000

20

01

2002

20

03

2004

If speed limit change happened in 1988, researchers may have thought it decreased fatalities

1/27/15  

7  

Time as a Confound • Historical factors – changes in culture across time can affect DV, especially when study takes place over years • E.g., much changed on the roads throughout the 1990s

•  Bigger cars, more traffic/ people, less maintained roads

• Maturation factors – changes in participants across time can affect DV • E.g., different demographic of drivers on the road in

early vs. mid ’90s

Other Confounds • Testing factors – simply being exposed to baseline measures can affect the DV • E.g., exposure to depression scale can motivate

participants to get better

• E.g., study examining effect of SAT-training on SAT scores can be confounded by practice effects

… …

Other Confounds • Demand factors – simply being in a study can affect the DV • E.g., depressed patients may get better simply

because…

• They know they are being treated (placebo effect)

• Doctors believe they will get better (experimenter effect)

• They are being observed (Hawthorne effect)

• Biggest problem: There is no control group to compare them too

1/27/15  

8  

A Common Case… • DARE (Drug Abuse Resistance Education)

• Single-group design tested in various schools in 1980s

• Baseline measure of drug-use was assessed

• DARE was implemented

• Treatment measure of drug-use was assessed

•  Initial studies of DARE showed positive results

• DARE went on to be implemented in 75% of U.S. schools and 42 other countries

• Possible confounds?

Possible Confounds • Historical factors – perhaps drug-use lessened on its own in the 1980s

Possible Confounds • Historical factors – perhaps drug-use lessened on its own in the 1980s

• Maturation factors – perhaps the demographics of students in schools began to change

• Testing factors – perhaps mere exposure to baseline measures affected drug-use

• Demand factors – perhaps knowledge that school was taking part in a drug-program affected student/ teacher/ parent behavior

1/27/15  

9  

Possible Confounds • Another possible confound:

• DARE was implemented in schools that had the worst drug-record in 1980

• Why is this a problem? •  These schools were statistically most likely to improve on

their own in the following year • There is a lot of random fluctuation in school drug-use

from year to year • Schools that are the worst one year will not likely be the

worst again next year (like lightening striking twice) • Known as “regression to the mean”

Regression to the Mean • Major confound in studies where cases/ groups are selected because they are “extreme”

• Most behaviors vary randomly over time •  E.g., Rate of smoking in one individual

# of

Cig

aret

tes

Average # of cigarettes smoked per day Jan. 2010 May 2010 Sept. 2010 Dec. 2010

< 2

> 20

5

10

15 Average 10-cig. per day smoker

Regression to the Mean • Study examines effects of smoking treatment on “heavy smokers” (20 cigarettes per day)

•  If study was conducted in October, this guy could be included

# of

Cig

aret

tes

Average # of cigarettes smoked per day

Jan. 2010 May 2010 Sept. 2010 Dec. 2010

< 2

> 20

5

10

15 Average 10-cig p/day smoker

• However, he will decrease his smoking regardless of treatment

• Causes treat-ment to seem effective

1/27/15  

10  

Regression to the Mean • Not every “heavy smoker” will be a product of chance fluctuations • But if even a few of them are, their

“treatment scores” will bias the sample

• Whenever cases/ groups are selected because they are at the “extreme”… •  It’s likely some are actually average,

but happen to appear extreme at the time of the study

Example in the Classroom • Scores on an exam are a product of…

• Actual knowledge

•  Luck

• Most students have an average amount of actual knowledge •  So, those who score at the extremes likely had some amount

of luck (good or bad) playing into their score

• This luck isn’t likely to strike these same students twice, so their future scores will return to the mean

Example in the Classroom • E.g.,

• Students in PSY1 who had highest score on Exam #1

Exa

m G

rade

s

Exam #1 (Baseline)

Exam #2 Exam #3 Exam #4 < 30%

>90%

40%

50%

60%

80%

70%

1/27/15  

11  

Example in the Classroom • E.g.,

• Students in PSY1 who had highest score on Exam #1

Exa

m G

rade

s

Exam #1 (Baseline)

Exam #2 Exam #3 Exam #4 < 30%

>90%

40%

50%

60%

80%

70% Average regression

toward the mean

Example in the Classroom • E.g.,

• Students in PSY1 who had lowest score on Exam #1

Exa

m G

rade

s

Exam #1 (Baseline)

Exam #2 Exam #3 Exam #4 < 30%

>90%

40%

50%

60%

80%

70%

Example in the Classroom • E.g.,

• Students in PSY1 who had lowest score on Exam #1

Exa

m G

rade

s

Exam #1 (Baseline)

Exam #2 Exam #3 Exam #4 < 30%

>90%

40%

50%

60%

80%

70% Average regression

toward the mean

1/27/15  

12  

Example in the Classroom • Regression to the mean can cause problems

• Can create a false sense of confidence in students who score really high due to luck

• Can create a false sense of dread in students who scores really low due to luck

Another Example • Regression to the mean can cause problems

• Not commonly taken into account in our expectations

•  E.g., people who are highly intelligent owe that partly to their genes and upbringing, but also partly to luck

•  When two such people have a child, they are usually surprised to find their child is of average intelligence

• Same with attractiveness, creativity, drive, etc.

Back to DARE • So, even if there weren’t any historical, maturation, testing, or demand factors confounding the study… • Schools were likely to improve without DARE simply

because of regression to the mean

• Experimental evidence… • Researchers haven’t been able to replicate the positive

effects of DARE in controlled experimental trails

• Evaluations of long-term effects of DARE have shown it has no effect on keeping kids off drugs or out of jail (Rosenbaum & Hanson, 1998; Thombs, 2000)

1/27/15  

13  

Single Case / Group Designs • There are many similar stories of programs being widely implemented because of positive results in quasi experiments • Still preferred method of program evaluation in schools,

social/ welfare programs, and most businesses

• Advantages • Allows testing of effects when a true experiment cannot

be conducted •  E.g., all schools with drug problems desperately wanted to

be in the DARE program

Single Case / Group Designs • There are many similar stories of programs being widely implemented because of positive results in quasi experiments • Still preferred method of program evaluation in schools,

social/ welfare programs, and most businesses

• Disadvantages • Without a control group or random assignment to

conditions, there can be any number of confounds actually driving effects • Historical, maturation, testing, demand factors

• Regression to the mean

Developmental Research • Studying changes in individuals over time

• Similar to single-case/ group designs •  IVs (life events, diseases, family upbringing, etc.) cannot be

assigned by researcher (i.e., no random assignment)

•  No way to make all participants have the same experience except for variation in IV (i.e., no experimental control)

• Types: • Cross-sectional Method – participants of different age

groups are studied simultaneously •  Longitudinal Method – one group of participants are

observed as they age

1/27/15  

14  

Cross-sectional vs. Longitudinal • Advantage of Cross-sectional Designs • Much cheaper

and quicker than conducting a longitudinal study •  Studying effects

of childhood factors on adult behavior can take 20 years!

Development of participants:

Development of researcher:

Cross-sectional vs. Longitudinal • Disadvantage of Cross-sectional Designs

• Can be biased by “cohort effect” • Cohort – group of people born around the same time

• Share a common culture/ history

• Cohorts can greatly differ from one another • How is a cohort born in the ‘60s likely to differ from

one born in the ‘90s? • E.g., cross-sectional study finds that as people

become older, they show significant decreases in ability to learn new skills • Test of skills all conducted on computers

Possible Confounds • All developmental designs suffer from…

• Historical confounds – major events in history may change participants in unknown ways

• E.g., effects of stress on life outcomes may be very different depending on when in history the data was collected

Stressors of the 1940s

Stressors of the 2000s

1/27/15  

15  

Possible Confounds • All developmental designs suffer from…

•  Test confounds – longitudinal studies usually involve participants taking the same tests over and over again

• E.g., study examining changes in IQ over time may be confounded by practice effects

• Many tests cannot be performed on children, so studies must compare childhood tests and adult tests that might not actually be measuring the same thing • Measuring Openness in children: reaction to new toys

• Measuring Openness in adults: reaction to new works of art