Chapter 10

download full file at http://testbankinstant.com

Chapter 10

Quasi-Experimental Designs

Learning objectives

To be able to Explain how quasi-experimental designs differ from weak and strong experimental designs.

Describe the characteristics of each of the quasi-experimental designs. Explain the threats to internal validity and how rival hypotheses are ruled out in each of the quasi-experimental designs.

Multiple choice questions1) In which of the following circumstances would a researcher be most likely to choose a quasi-experimental research design?

a) when there are more than three important confounding variables to control*b) when doing research in an applied or natural setting where control is difficult to

imposec) when controlling experimenter and participant expectation effects would involve an unacceptable level of deceptiond) when it is difficult to recruit enough participants to make standard statistical testing

valid2) The most common reason for the use of quasi-experimental research designs is that

a) the participants are maturing too rapidly.

b) the dependent variable cannot be measured reliably.

c) it is unethical to manipulate the independent variable .

*d) participants cannot be randomly assigned to groups.

3) Quasi-experimental designs lack the controls of true experimental designs.

*a) however, causal inferences can be made to the extent that rival hypotheses can be ruled

outb) as such, causal inferences should never be madec) however, in most cases such deficiencies can be safely ignoredd) thus, such studies should not be performed by ethical researchers4) One essential characteristic of quasi-experimental designs that distinguishes them from true experimental designs is a lack of

a) volunteer participants.

*b) control of extraneous variables.

c) experimenters who can conduct them.

d) an interest in laboratory experimentation.

5) When one does not have the ability to , one may consider using quasi-experimental designs.

a) measure the dependent variable quantitatively

*b) randomly assign participants to groups

c) manipulate the independent variable

d) make multiple measures of the dependent variable

6) You have been asked to evaluate the effectiveness of a new anti-smoking campaign in a local high school. You are able to collect pretest information on smoking rates before the campaign begins. You know that the entire school will be required to participate in the campaign and so random assignment is impossible.You are also aware that a reduction in smoking after the campaign could be caused by any number of other confounding variables.What can you do in this situation to enhance your ability to determine if the campaign was effective in reducing smoking?

a) test the students at the end of the campaign and then again a year later if reductions

in smoking remain at that time then you can be confident that the campaign was

effective

b) conduct extensive postexperimental interviews to determine the cause of any change in

smoking rates

*c) identify a similar school that is not using the campaign and include them in a

nonequivalent comparison group design

d) perform a median split among the smokers in the school and perform a regression

discontinuity study

7) Although quasi-experimental designs do not meet all of the criteria for true experimental designs, they may still

a) be used by experimenters without much training.

b) be performed in a fairly short time frame.

*c) allow one to make statements about causation.

d) control for potential effects of extraneous variables.

8) The non-equivalent comparison group design is a quasi-experimental design in which, for reasons of practicality, we cannot insure that the control and experimental groups are equivalent to each other when the experiment begins. The major interpretational difficulty imposed by this design is

a) knowing whether the two groups are different from each other on the dependent measure once the experiment is complete.

b) deciding how much each group has to gain on the posttest compared to the pretest to be sure that the differences are reliable.

c) determining when we have collected enough data points to make a statement about the experiment's outcome.

*d) being sure that any differences between groups at the end of the experiment result from the independent variable's influence alone, and not related to preexisting differences.

9) What is the main distinction between the nonequivalent comparison group design and the pretest-posttest design?

a) the use of pretesting in the latter

b) the use of statistical data analysis in the latter

*c) the random assignment of participants to groups in the latter

d) more than one independent variable in the latter

10) Which of the following designs is the best in terms of eliminating potential confounds?

a) nonequivalent comparison group

b) posttest-only design with nonequivalent groups

*c) pretest-posttest control-group design

d) interrupted time series design

11) Dr. Johannsen wants to test whether recovering alcoholics perform worse on tests of motor coordination than a group of control participants. Unbeknownst to him, the control group meets at an amusement park three hours before testing and goes on as many rides as possible. They perform worse than the alcoholic group. This is an example of a(n)

a) regression effect.

b) selection-maturation effect.

c) regression artifact.

*d) selection-history effect.

12) The selection-history effect is particularly a problem in which quasi-experimental design?

*a) non-equivalent comparison group

b) time seriesc) interrupted time series

d) regression discontinuity

13) The nonequivalent comparison group design can yield several possible outcomes. In one scenario, the experimental group scores higher than the control group at the start of the experiment, and only the experimental group's scores increase from pretesting to posttesting. This could reflect an influence of the independent variable. However, it could also reflect

a) a local instrumentation effect.

b) a attrition effect in the control group.

c) simply that the experimental group was studied for a longer time period than the control group.

*d) selection-maturation effect.

14) One strategy for dealing with preexisting differences between experimental and control groups in the nonequivalent comparison group design is to match the participants.

a) this procedure is as effective as randomly assigning participants from the startb) this procedure eliminates local history as a confounding factor*c) this procedure, however, may introduce a regression-artifact phenomenon as extreme scores are frequently needed to matchd) this procedure, however, confounds the research even more than the preexisting

differences15) The nonequivalent comparison group design can yield several possible outcomes. A possible outcome of the non-equivalent comparison group design is that the groups differ on the pretest and both groups improve on the posttest, but to different degrees. This could reflect the fact that the independent variable had an effect on the experimental group. It could also reflect

a) an unreliable dependent measure.

b) the fact that the control group experienced less of the independent variable than the experimental group did.

c) a participant-expectation effect.

*d) a selection-maturation effect.

16) The nonequivalent comparison group design can yield several possible outcomes. A possible outcome of the non-equivalent comparison group design is that the control group scores higher than the experimental group on the pretest and posttest but the experimental group improves more on posttesting. This could reflect the fact that the independent variable had an effect on the experimental group. It could also reflect

a) an unreliable dependent measure.

b) the fact that the control group experienced less of the independent variable than the experimental group did.

*c) regression-artifact effect if the experimental group had been selected for their low initial scores.

d) a selection-maturation effect if the experimental group had been selected for their

older age.

17) In the physical education department, Dr. Shanz asks two of his strength training classes to participate in an experiment. One class will follow a traditional program of index finger strength enhancement, while the other class will try a new, experimental procedure. At the beginning of the semester, the class that will use the experimental method starts out with, on average, weaker index fingers than the other class. At the end of the semester, the class using the traditional method shows no significant increase in finger strength, while the other experimental class now has fingers that are, on average, significantly stronger than those of the control class. This is an example of a(n) effect.

a) interaction

*b) crossover

c) intragroup regression

d) selection-maturation

18) The presence of a crossover effect in one's data can usually rule out the influence of which rival hypothesis?

a) history

*b) regression-artifact

c) local history

d) attrition

19) Dr. Kaiser, a sports psychologist, wants to know if he can improve the cardiovascular endurance in elementary school children. He is allowed to study two physical education classes at a local school, one boys class and one girls class. On the pretest, the girls on average have slightly better cardiovascular endurance as measured on the step test. He administers the year-long aerobic dancing program to the girls class. The posttests revealed that the girls cardiovascular endurance improved much more than the boys. Dr. Kaiser must now decide if the program really worked or

a) if there was a selection-history effect.

*b) if there was a selection-maturation effect.

c) if girls always do better in aerobic exercises.

d) if the boys just didnt want to compete with the girls.

20) The most common of the quasi experimental designs is the nonequivalent comparison group design. What is the dependent variable most commonly used in this design?

a) latency measure

*b) difference scores (Pretest-posttest)

c) number or percent correct

d) total correct

21) Which of the following is NOT a bias that commonly exits in nonequivalent comparison group designs?

a) selection bias

b) attrition bias

c) selection-regression bias

*d) experimenter expectancy bias

22) According to Glass, Wilson and Gottman, you need about data points to make a statistical assessment of the data from a time-series design.

a) 25

*b) 50

c) 75

d) 100

23) ___________ is a statistical technique designed to assess the effectiveness of an interrupted time series design.

a) Reliability adjusted analysis of covariance (ANCOVA)

b) Propensity score matching

c) Selection modeling

*d) Autoregressive integrated moving average (ARIMA)

24) Making a causal inference from quasi-experimental evidence requires all but which of the following:

a) the suspected cause must covary with the effect

b) the suspected cause must precede the effect

c) rival hypotheses must be ruled out or be highly implausible.

*d) Bayesian moving average statistical analyses.

25) When looking at interaction effects, it is really good to find a crossover effect because it can rule out various rival hypotheses by its very nature. Which of the following rival hypotheses is not ruled out by the presence of a crossover effect?

a) statistical regression effect

b) maturation effect

*c) selection-history effect

d) none of the above26)In a nonequivalent comparison group study,

*

a) the experimental group gets a pretest, then the experimental treatment, then a posttest.

b) participants are assigned to the control group by means of a matching procedure.

c) to interpret the results, the pretest scores have to be identical for the two groups.

d) random assignment can be used if sample size is large enough.27)In a study comparing two nonequivalent groups, a selection-history confound occurs when

*

a) some event intervenes between pretest and posttest and affects just one of the groups.

b) some event intervenes between pretest and posttest and affects both selected groups

equally.

c) selection influences one of the groups and history influences the other.

d) some historical event causes a failure of the random assignment procedure.28) Which of the following threats to internal validity is most likely to be a problem in the nonequivalent comparison group design?

a) history

b) maturation

*

c) selection bias

d) all of the above are potential problems29) Imagine a hypothetical study using a nonequivalent comparison group designed to assess the effectiveness of an anti-drug, Just Say No, campaign. In this study students at school X receive a semester-long series of Just Say No lectures. Students at school Y across town do not receive the intervention. Unfortunately, during the semester a student at school Y died after an accidental drug overdose. This incident may have influenced drug use at school Y (but not school X) and consequently make it impossible to interpret our research findings. Which of the following threats to internal validity is present in this study?

a) selection-instrumentation

b) selection-maturation

*c) selection-history

d) selection-attrition30) Differential drop out rates in your experimental and control groups (selection-attrition) is most likely to be a problem in which of the following designs:

a) time series

b) regression discontinuity

c) pretest-posttest control group

*d) nonequivalent comparison group 31) The ___________ is to ___________ as nonequivalent comparison group design is to posttest-only design with nonequivalent groups.

a) one group pretest-posttest design; interrupted time series design

*

b) interrupted time series design; one group pretest-posttest design

c) pretest-posttest control group design; interrupted time series design

d) interrupted time series design; regression discontinuity design

32) Sometimes it is difficult if not impossible to find a control group that is sufficiently comparable to an experimental group to allow for a traditional two-group experimental design. In these cases, one may want to use a time-series design

*a) which is a quasi-experimental design involving multiple measures of the dependent variable on a single group, both before and after treatment.

b) which is an experimental design involving multiple measures of the dependent variable on a single group, both before and after treatment.

c) which is an experimental design using multiple dependent measures on the experimental group, then taking measures on control participants over time, as he or she can find them.

d) staggers the time of presentation of the independent variable across participants.

33) Dr. Toris wants to test an incentive program to increase attendance in her classes. For the first 8 weeks of the semester she does not use the program, and records attendance rates each week. Then she institutes the incentive program for the last 8 weeks and continues to record attendance rate each week. She detects an increase in attendance over the last 8 weeks. This is an example of which experimental design?

a) one group pretest-posttest*b) interrupted time series

c) non-equivalent control group

d) multiple baseline

34) Visual inspection of the pre- and post-treatment pattern of dependent measure scores is typical of which quasi-experimental design?

*a) interrupted time series

b) non-equivalent control group

c) before-after

d) factorial

35) Visual inspection of the data in the interrupted time series design is not always sufficient. With enough data points, the patterns of the dependent measures before and after can be analyzed with*a) an autoregressive integrated moving average model.

b) Cohens d.

c) an analysis of covariance.

d) correlation coefficients.

36) Of the major extraneous variables discussed in previous chapters, which is most likely a source of confounding in an interrupted time-series design?

a) participant expectations

*b) history

c) attrition

d) regression

37) In using an interrupted time-series design, how does one determine whether the independent variable had an effect?

*a) by looking for changes in the response pattern from pre- to post-treatment

b) by using the chi-square statistic

c) by comparing pre-treatment mean performance to post-treatment mean performance

d) by comparing the treatment group's data to a single observation of a control group

38) In a(n) design, the experimenter takes multiple measures on the dependent variable both before and after treatment.

a) non-equivalent control group

*b) interrupted time series

c) multivariate

d) before-after

39) What is the "interruption" in an interrupted time series design?

*a) introduction of the independent variable

b) instrumentation failure

c) a period of at least two weeks

d) a brief period when measurements are taken.

40) A small town decides to ban cell phone texting while driving. Before putting the law into effect, officials measure the amount of cell phone use among drivers by placing observers at a busy intersection and recording the percentage of drivers using phones. They do this over 6 consecutive one-week intervals, each week being the middle week of a month (i.e., six months of pretesting). Right after the law is passed, officials repeat the observational study for another six months. Which of the following is true about the design of this study?

*

a) it is an interrupted time series design

b) it is an example of a nonequivalent control group design

c) without a control group, there is no way to evaluate trends with this design

d) because there are multiple observations this would qualify as a regression

discontinuity design41) The National Science Foundation has a policy of, each year, selecting the best young scientists and giving them a 5 year grant to allow them to work on their program of research. These scientists are selected based on the number of publications they have produced in the prior year. All scientists who have published 4 or more articles in the prior years receive these grants. The National Science Foundation wants to find out if this program is effective. The research design that could be used to test the effect of this program is

a) posttest-only design.*b) regression-discontinuity design.

c) time series design.

d) pretest-posttest design.42) Which of the following is NOT one of the requirements of the regression-discontinuity design.

a) group assignment must be based on a cutoff score

*b) the cutoff score should be located at one of the extreme ends of the distribution

c) the experimenter must control the assignment of participants to groups

d) all participants must be from the same population43) If you have conducted a study using the regression-discontinuity design, how would you tell if your treatment had an effect?

a) by looking at the data to see if the regression line of the posttest scores differs from the regression line of pretest scores.

b) by statistically analyzing the data using Analysis of Variance.

c) by comparing the pretest scores with the posttest scores to see if there was a significant change.

*d) by determining if there is a discontinuity in the regression line for the people above

and below the cutoff score.

44) What is the primary threat to internal validity that could confound the results of a regression discontinuity study?

a) maturation

*

b) history

c) attrition

d) testing

45) In a study designed to examine the effects of an intensive reading program, Vaughn et. al (2009) screened 2nd grade students and separated good readers from poor readers. The poor readers were given the intervention treatment. At the end of 26 weeks of intervention significant improvement in the poor readers was noted. What type of experimental design was employed in this study?

*

a) regression discontinuity

b) nonequivalent comparison group

c) interrupted time series

d) median split design

46) Which of the following is NOT a quasi-experimental design?

a) interrupted time series

b) nonequivalent comparison group design

*

c) pretest-posttest control group design

d) regression discontinuity design

Vocabulary

Define the following in psychological terms:

qquasi-experimental design

design components

nnonequivalent comparison group designiincreasing treatment and control groups

selection-maturation effect

selection-history effect

selection-instrumentation effect

selection-attrition effect

selection-regression effect

ffirst increasing treatment effect

ssecond increasing treatment effect

ccrossover effect

iinterrupted time-series design

rregression discontinuity design

assignment measure

Essay questions1) Distinguish between experimental and quasi-experimental research designs. When would a researcher be more likely to use a quasi-experimental design?

2) Discuss the principles used to rule out rival explanation in quasi-experimental designs as identified by Shadish et. al (2002) and presented in you text.

3) Describe a hypothetical example of a study using the nonequivalent comparison group design. Be sure to describe all of the essential features of the design.

4) Explain how the nonequivalent comparison group design controls most of the threats to internal validity (e.g., history, maturation, instrumentation etc) as compared to the posttest only design. Explain why selection bias is still a problem with the nonequivalent comparison group design.

5) Describe the differences between a nonequivalent comparison group design and a one-group pretest-posttest design. For each, give an example of a situation in which one might have to use that design rather than a true experimental design. What is the critical aspect of your example that makes it impossible to use a true experimental design.

6) Discuss several different possible outcomes in a study using a nonexperimental comparison group design. Describe the rival hypotheses that could explain these outcomes.

7) Describe the essential features of an interrupted time series design. Explain why this is a better design than the one-group pretest-posttest design.

8) Draw two graphs of hypothetical outcomes for interrupted time series, one clearly showing an effect of the treatment and the other no effect of the treatment. Explain how your hypothetical data support your assertion of an effect and of no effect.

9) Describe the regression discontinuity design and when you would use it.

10) When using a regression discontinuity design, what kind of outcome or data would have to exist to enable you to infer that the treatment was effective? What potential threats to internal validity exist when using this design?Classroom exercise suggestions1) Quasi-experimental designs are often used to evaluate large scale programs that are designed to benefit groups of individuals or entire communities. Examples include the DARE program to reduce drug abuse or the Head Start program designed to help prepare children of poverty for kindergarten. There is increasing pressure on these types of programs to demonstrate positive outcomes. According to Rossi, Freeman, and Lipsey (1999) there are five different types of program evaluation 1) evaluation of need (are there problems that need to be addressed?); 2) assessment of program theory (does the proposed program impact the identified needs?); 3) program monitoring; 4) outcome evaluation (an assessment of whether or not the goals of the program are being met); 5) efficiency assessment (are the outcomes worthwhile?). Although any of the experimental methodologies may be used to evaluate programs it is likely that quasi designs will be used because random assignment is typically not possible. More information about program evaluation can be found in the sources below.

Campbell, D.T. (1969). Reforms as experiments. American Psychologist, 24, 409-429.

Posavac, E., & Carey, R. (1997).Program evaluation: Methods and case studies (5th

ed.). Englewood Cliffs, NJ: Prentice-Hall, Inc.

Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-

experimental designs for generalized causal inference. Boston: Houghton Mifflin.

As suggested by Shaughnessy, Zechmeister, & Zechmeister (2009) one way to make program evaluations more relevant to students is to relate them to institutional changes at your college or university. Below is a brief summary of an activity suggested by Shaughnessy et. al.Ask students to consider recent changes that have been made on your campus (e.g., changes in graduation requirements or student life issues). What evidence was used to determine that a change needed to be made (e.g., comparison with other schools, consultation with experts)? Did the administration use experiments or quasi-experiments to determine whether or not to make these changes? Why do you think institutions do not make use of experiments or quasi-experiments? Can you think of situations in which an experimental approach could have been used?2) You can extend the text discussion of interrupted time series by discussing several variations on the basic design. These include reversal time-series designs which involves multiple removals and re-introductions of the treatment similar to single subject reversal designs. If similar changes in behavior are seen each time the treatment is introduced then we can be more confident in its effectiveness. Present examples to your students with the caveat that in many instances it may unethical or simply not feasible to employ this type of design. Another variation of the interrupted time series design involves the addition of a (nonequivalent) control group that does not receive the experimental treatment. An example of this time of design is a study conducted by Asbridge et. al (2009) who used a time series design to evaluate the effect of a change in Ontario law that required the suspension of drivers license for anyone driving with a blood alcohol level over the legal limit. Monthly traffic fatalities were monitored before and after the introduction of the new law. Additionally, two other provinces served as controls are were monitored as well. The results of the study indicate that the introduction of the new law resulted in a significant decrease in traffic fatalities. The inclusion of control provinces rules out the influence of simple history effects as an explanation for changes in the traffic fatalities. However it should be noted that these types of designs would not eliminate the possibility of selection-history interactive effects.Asbridge, M., Mann, R., Smart, R., Stoduto, G., Beirness, D., Lamble, R., et al. (2009).

The effects of Ontario's administrative driver's licence suspension law on total

driver fatalities: A multiple time series analysis.Drugs: Education, Prevention &

Policy,16(2), 140-151.A third variation that improves the basic design is called interrupted time series with switching replications. This type of design involves repeating the intervention at a different point in time with a different sample. For example, a study by Palmgreen et. al (2001) assessed the effectiveness of public service announcements (PSA) on reducing marijuana use among high users. An interrupted time series design was used with the PSA introduced at different points in time for two different counties in Kentucky. Similar reductions in marijuana use for the two counties provides strong evidence for the efficacy of the PSAs.Palmgreen, P., Donohew, L., Lorch, E., Hoyle, R., & Stephenson, M. (2001). Television

campaigns and adolescent marijuana use: Tests of sensation seeking

targeting.American Journal of Public Health,91(2), 292-296.3) Describe the massed vs. distributed practice effects to the class. Tell them how these differences could be applied to improve learning in school. Then challenge them to design a field experiment in a real school. Let them come to grips with the idea that random assignment is not always feasible. Then let them design the study as a quasi-experiment.

4) If this material is introduced early enough in the semester, students may wish to collect time series data over the course of several weeks or at least several days. The important point of this exercise is to challenge students to decide how to determine whether pre- and post-treatment data are significant. Although a thorough treatment of time series statistical analyses are most likely beyond the scope of the course, students can at least gain an appreciation of the issues involved. This will be a particularly valuable introduction to the next chapter on single-case research designs, as many of these designs are time-series based.

5) Have students identify a treatment or a program that should be investigated using the regression-discontinuity design. After they have identified this program, have them design the study selecting such variables as the pretest and posttest measures and the cutoff score.

6) Investigate the effect of midterms on number of people in the library. Choose a study area of your campus library and visit it at roughly the same time every day; be sure to use a popular study time. Count the number of people in the study area (be sure to define the area's boundaries clearly so that each observer is taking the same kind of data). Pretreatment days = days before midterms, post-treatment = days during and after midterms. Do midterms have a temporary or long-lasting effect on the number of people studying in the library?download full file at http://testbankinstant.com