Name: ___________________________________ Date: ____________ Period:______

CHAPTER 4: Designing Studies CHAPTER INTRODUCTION:

You can hardly go a day without hearing the results of a statistical study. Chapters 1-3 focus on

descriptive statistical analysis. Since it is difficult to describe data before we have collected it,

we will begin our course learning appropriate ways to produce data. Section 4.1 examines the

issues involved in sampling and surveys. We will start by learning the difference between a

__________________ and a ____________. Then we will study sampling techniques and learn

how to identify potential sources of ________. Our goal is to collect data that is representative

of the population we wish to study. Therefore, it is important that our data collection

techniques do not systematically over or under-represent any segment of the population.

Section 4.2 focuses on designing experiments. We will learn the difference between

_________________ studies and __________________. We will learn a number of different

ways to design experiments so we can establish relationships between _______________.

Finally, in Section 4.3, we’ll look at why establishing causation is so difficult, especially in

observational studies. We’ll also consider some ethical issues involved in planning and

conducting a study, as well as reviewing some cautions about using studies wisely. This chapter

contains a lot of vocabulary and concepts. Be sure to study them, as proper application of terms

is important for strong statistical communication!

4.1: Sampling and Surveys SECTION INTRODUCTION: Often is statistics, our goal is to draw a conclusion about a ________________ based on

information gathered from a _____________. In order for us to make a valid _____________

about the population, we must feel confident that the sample we obtained will be

_________________________ of the group as a whole. There are a number of different ways to

select samples from a population, some better than others. In this section, you will explore

ways in which you can sample badly, how to sample well, and cautions to consider when

sampling. You will be introduced to a number of different sampling methods. Be sure to

familiarize yourself with how to select samples using each of the methods and how to explain

potential advantages and disadvantages of each.

4.1 KEY VOCABULARY AND CONCEPTS: Population, Sample, Sample Survey, Convenience

Sample, Convenience Sample, Bias, Voluntary Response Sample, Simple Random Sample (SRS),

Table of Random Digits, Stratified Random Sample and Strata, Cluster Sample and Clusters,

Inference, Undercoverage Bias, Nonresponse Bias, Response Bias, Wording of Question Bias,

Sampling Error, Non-sampling Error

Page | 2

4.1-CONCEPT 1: Population, Samples, and Inference (Pages 207-210)

The purpose of sampling is to provide us with information about a _____________________

without actually gathering the information from every single element of the whole group. A

_____________ is a part of the population from which we collect information. The information

is then used to _________ something about the population. We can more faith in our inference

if we are confident that the sampling methods used is likely to produce sample that is

_________________________ of the population of interest. Some sampling methods may

introduce a level of ___________ into the situation that could cause us to make an incorrect

inference about the population. It is important that you can clearly identify the population of

interest and describe whether or not the sampling _______________ is unbiased.

Sampling Error Non-Sampling Error

4.1 EXAMPLE 1: Identify the population and sample in each of the following settings.

(a) The student government at a high school surveys 100 of the students at the school to

get their opinions about a change to the school schedule.

Population: Sample:

(b) The quality control manager at a vegetable canning company selects a sample of 10 cans

from the production line every hour to see whether the volume of the given vegetable is

within acceptable limits.

Population: Sample:

4.1 EXAMPLE 2: In May 2010, the Los Angeles City Council voted to ban most travel and business contracts with the state of Arizona to protest Arizona’s new immigration enforcement law. The Los Angeles Times conducted an online poll that asked whether the City Council was right to pass a boycott of Arizona. The results showed that 96% of the 41,068 people in the sample said “No”. What type of sample did the LA Times use in their poll? Explain how this sampling method could lead to bias in the poll results.

Page | 3

4.1 EXAMPLE 3: A farmer brings a juice company several crates of oranges each week. A company inspector looks at 10 oranges from the top of each crate before deciding whether to buy all the oranges. Identify the sampling method used. Then explain how the sampling method could lead to bias.

4.1-CONCEPT 2: How to Sample Well (Pages 211-221)

The key to sampling is to use a method that helps ensure that the sample is as representative of

the population as possible. Because sampling methods like ___________________ response

and ______________ samples can __________________________ favor certain outcomes in a

population, we say they are _______. An unbiased sampling method is one that does not favor

any element of the _____________________. We rely on the use of ______________ to select

unbiased samples. The easiest way to do this is to select a ____________ __________

________ (_____). From the population of interest. One way we could select an SRS is to write

the name of each individual of the population on a piece of paper, put the pieces in a hat, mix

them well. And draw out the necessary number of slips for our sample. We could also select an

SRS by labeling each individual in the population with a number of the _________

____________ (i.e. one-, two-, three-digit number depending on the size of the population) and

then generate random numbers using ___________________ or a __________ _________

_________ until you reach the desired sample size. The labeled individuals that match the

generated numbers, ignoring repeats, make up the ________________.

4.1 EXAMPLE 4: An alphabetized list of student names is found below. Use the random digit table provided to choose an SRS of 5 individuals. Clearly indicate how you are using the table to select the SRS.

Abney Brock Greenberg Osters Tyson Anderson Bush Knott Preble Willow Bearden Chu Lacey Roberts Yankay Bready Costello Martin Smith Buckley Derksen McDonald Sterken

19223 95034 05756 28713 96409 12531 42544 82853 73676 47150 99400 11327 27754 40548 82421 76290

Page | 4

LABEL:

TABLE:

STOPING RULE:

IDENTIFY:

While Simple Random Samples give each group member of n individuals in the population an

__________ ___________ of being selected, they are not always the easiest to obtain. Several

other sampling methods are available that can be used instead of an SRS. If the population

consists of several ____________ of individuals that are likely to produce similar responses

within groups, but systematically different responses between groups, we should consider

taking a _______________ ___________ ______________. To do this, select an _______ from

each ____________ (group) to obtain the sample. Another method, ______________

______________, divides the population into smaller groups (clusters) that mirror the overall

population and selects an SRS of those clusters. You should be able to describe how to select a

sample using each of these methods as you may be asked to do on the AP exam!

4.1 EXAMPLE 5: The manager of a beach-front hotel wants to survey guests in the hotel to

estimate overall customer satisfaction. The hotel has two towers, and older one to the south

and a newer on to the north. Each tower has 10 floors of standard rooms (40 rooms per floor)

and 2 floors of suites (20 suites per floor). Half of the rooms in each tower face the beach, while

the other half of the rooms face the street. This means that there are 880 total rooms, (2

towers)(10 floors)(40 rooms)+(2 towers)(2 floors)(20 suites) = 880 rooms.

(a) Explain how to select a simple random sample of 88 rooms.

Page | 5

(b) Explain how to select a stratified random sample of 88 rooms.

(c) Explain how to select a cluster sample of 2 floors. Then explain why selecting 2 of the 24

different floors would not be a good way to obtain a sample.

Page | 6

4.1-CONCEPT 3: What Can Go Wrong? (Pages 221-224)

When designing a sample survey, random sampling helps avoid _______. But there are other

kinds of mistakes in the sampling process that can lead to inaccurate information about the

population. For instance, if the sampling method is designed in a way that ____________

______ certain segments of the population, the sample survey suffers from

______________________. Bias can also result from ________________________ errors.

When selected individuals cannot be contacted or refuse to participate, the survey suffers from

__________________ ________. Finally, if a question is worded in a way that favors certain

responses, the survey suffers from _____________________ ________________ bias. Be sure

to design sample surveys to avoid these issues and make sure you can identify them in existing

studies.

4.1 EXAMPLE 6: Identify what was wrong with the following surveys and how the issue could have affected the results of the survey.

(a) In 1988, Shere Hite published a book entitled Women and Love: A Cultural Revolution in Progress, in which she claimed that 70% of women who have been married five years or more are having extramarital affairs. Hite’s survey was sent to 100,000 women, and her conclusions were based on the 4,500 women who responded to her survey.

(b) A leading research firm is contracted to survey American adults on their preference of candidates for the upcoming election. They use a computer program to randomly contact individuals through their landlines between the hours of 10:00am and 5:00pm.

Page | 7

(c) A survey paid for by the makers of disposable diapers found that 84% of the sample opposed banning disposable diapers. Here is the actual question they used:

“It is estimated that disposable diapers account for less than 2% of the trash in today’s landfills. In contrast, beverage containers, third-class mail and yard wastes are estimated to account for about 21% of the trash in landfills. Given this, in your opinion, would it be fair to ban disposable diapers?”

(d)

4.1 MULTIPLE CHOICE EXERCISES:

(1) A large suburban school wants to access student attitude towards their mathematics

text book. The administration randomly selects 15 math classes and gives the survey to

every student in the class. This is an example of a

(A) multistage sample (B) stratified sample (C) cluster sample (D) simple random sample (E) convenience sample

(2) To determine employee satisfaction at a ;arge company, the management selects an SRS

of 200 workers from the marketing department and a separate SRS of 50 workers from

the sales department. This kind of sample is called a

(A) multistage sample (B) stratified sample (C) cluster sample (D) simple random sample (E) convenience sample

Page | 8

4.2: Experiments SECTION INTRODUCTION: Sample surveys allow us to gather information about the population without actually doing

anything to that population. Other ____________________ ____________ watch animals in

the wild or interactions between customers and sales people. Such observational studies

provide a snapshot of the population, but cannot be used to establish any sort of cause-effect

relationship. Observational studies only allow us to describe the population, compare groups,

or examine basic relationships between variables. In this section, we will move beyond

sampling to study the elements of ___________________ _______________. Experiments are

very different in that they allow us to _______________ data in a way that can lead to

conclusions about causation. There are a lot of vocabulary terms in this section. It is easy to

confuse sampling terms with experimental design terms. Make sure you understand each

vocabulary term and concept!

4.2 KEY VOCABULARY AND CONCEPTS: Observational Study, Experiment, Lurking Variable, Confounding, Treatment (factor levels), Explanatory Variables (factors), Experimental Units, Subjects, Random Assignment, Completely Randomized Design, Control Group, Replication, Placebo, Placebo Effect, Single-Blind, Double-Blind, Statistically Significant, Block, Randomized Block Design, Matched Pairs Design

4.2-CONCEPT 1: Observational Studies vs. Experiments (Pages 231—236)

Sample surveys are examples of ___________________________ ________________. Their

goal is to describe the population and examine relationships between variables. Often,

however, we wish to determine whether or not a _____________-___________ relationship

exists between an __________________ and a _________________ variable. Observational

studies cannot be used to establish this relationship because ________________ variables can

result in ________________________ of our results.

4.2 EXAMPLE 1: Does shoe size affect spelling ability?

A recent study was conducted in a suburban school district to answer this question. 30 students

from grades 1 through 8 were randomly selected. Each student was administered a spelling test

and had his or her feet measured. Test scores were plotted against shoe size and a strong,

positive relationship was observed.

(a) Was this an observational study or an experiment? Explain.

(b) What are the explanatory and response variables?

Page | 9

(c) Suggest a possible lurking variable in this setting. Explain carefully how it may confound

the results.

4.2 EXAMPLE 2: Is soy good for you?

The November 2009 issue of Nutrition Action discusses what the current research tells us about the supposed benefits of soy. For a long time, scientists have believed that the soy foods in Asian diets explain the lower rates of breast cancer, prostate cancer, osteoporosis, and heart disease. However, when experiments were conducted, soy either had no effect or a very small effect on the health of the participants. For example, one study randomly assigned 224 elderly women to either soy or placebo, and showed that soy was no more beneficial for preventing osteoporosis than the placebo.

(a) Why was this research an experiment?

(b) Who or what were the experimental units?

(c) What was the explanatory variable?

(d) What was the response variable?

(e) Was this study blind? Double blind?

Page | 10

4.2 EXAMPLE 3: Growing tomatoes

Does adding fertilizer affect the productivity of tomato plants? How about the amount of water given to the plants? To answer these questions, a gardener plants 24 similar tomato plants in identical pots in his greenhouse. He will add fertilizer to the soil to half of the pots. Also, he will water 8 of the plants with 0.5 gallons of water per day, 8 plants with 1 gallon of water per day, and the remaining 8 plants with 1.5 gallons of water per day. At the end of three months, he will record the total weight of tomatoes produced by each plant. Identify the explanatory and response variables and the experimental units, and list all of the treatments.

4.2-CONCEPT 2: How to Experiment Well (Pages 236-243)

If we wish to establish cause and effect, we must conduct an experiment in which treatments

are imposed on experimental units and other potential influences on the response variable are

_________________ as much as possible. The basic idea behind an experiment is that we

obtain __________________ _______, _________ _________________, and ______________

the _____________. When conducted properly, experiments can provide good evidence of

________________. Proper experimental design incorporates three principles.

First, we must ____________ for the influences of lurking variables that might affect the

response. We want to make sure that any changes are due to the treatment alone.

Second, we must ______________ __________ experimental units to treatments. This

helps ensure equivalent groups of units or subjects. Random assignment also helps

ensure that the effects of lurking variables will be felt equally by all groups of the

experiment, so that the only _____________________ difference between the groups is

the treatments themselves.

Page | 11

Third, _________________ is necessary to ensure enough experimental units are in

each group to convince us the differences in the effect of the treatments can be

separated from _________________ ________________________ between the groups.

If a difference between groups is observed that is too large to have occurred by chance

alone, we say it is _________________________ significant. We will learn how to

establish statistical significance in later chapters. For now, focus on how to design a

quality experiment!

In summary, the three principles of experimental design are ___________, ___________

____________________, and _____________________.

4.2 EXAMPLE 4:

Does regularly taking aspirin help protect people against heart attacks? The Physician’s Health Study was a medical experiment that helped answer this question. In fact, the Physician’s Health Study looked at the effects of two drugs: aspirin and beta-carotene. Researchers wondered whether beta-carotene would help prevent some forms of cancer. The subjects in this experiment were 21,996 male physicians. There were two explanatory variables (factors), each having two levels: aspirin (yes or no) and beta-carotene (yes or no). Combinations of these factors form the four treatments shown in Figure 4.6 on page 242 of your textbook. One fourth of the subjects were assigned at random to each of these treatments.

On odd-numbered days, the subjects took either a tablet that contained aspirin or a dummy pill that looked and tasted like the aspirin but had no active ingredient (a placebo). On even-numbered days, they took either a capsule containing beta-carotene or a placebo. There were several response variables-the study looked at heart attacks, several kinds of cancer, and other medical outcomes. After several years, 239 of the placebo group but only 139 of the aspirin group suffered heart attacks. This difference is large enough to give good evidence that taking aspirin does reduce heart attacks. It did not appear, however, that beta-carotene had any effect on preventing cancer.

Explain how each of the three principles of experimental design was used in the Physician’s Health Study.

Page | 12

4.2 EXAMPLE 5: Mrs. Kim teaches statistics to 150 students. She is interested in knowing whether or not listening to classical music while studying results in higher test scores than listening to no music. She wishes to design an experiment to answer this question.

(a) What are the experimental units, explanatory variable, treatments and response variable?

(b) What are the potential lurking variables in this situation? How could they affect the results?

How could we avoid their effects?

(c) Describe a completely randomized design for Mrs. Kim’s experiment.

4.2-CONCEPT 3: Other Types of Experimental Design (Pages 244-251)

Completely randomized experiments are the simplest design that can give good evidence of

cause-effect relationships. In some cases, however, we can add elements to this basic design to

better _____________ for the effects of lurking variables. For example, we have learned that

placebos can be given to control for the placebo effect-the phenomenon that occurs when

subjects respond to getting any treatment, whether it is real or not! We can use

Page | 13

___________________ when there is a concern that knowing who receives what treatment

might affect the results. ______________-__________ experiments ensure that neither the

subjects not the people who interact with the subjects and measure their responses know who

receives which treatment.

When groups of subjects share a common characteristic that might systematically affect their

responses to treatments, we can use ___________________ to control for the effects of this

lurking variable. For example, suppose researchers are conducting an experiment to compare

the effectiveness of a new medicine for treating high blood pressure with the most commonly

prescribed drug. If they believe that older and younger subjects may respond differently to such

medications, the researchers can separate the subjects into blocks of older and younger people

and then randomly assign treatments within each block. This __________________

__________ ______________ helps isolate the variation in response due to age, which makes it

easier for researchers to find evidence of a treatment effect.

If we are only comparing two treatments, we can sometimes conduct a _______________

__________ design by creating blocks of two similar individuals and then randomly assigning

each subject to a treatment. Another type of matched pairs design involves assigning two

treatments to each subject in a random order.

In each type of experimental design, our goal is to ensure ____________, ___________

__________________, and ____________________. If these principles are addressed in our

design, we can establish good evidence of causal relationships. Make sure you can describe

how you would conduct an experiment using each of these designs as you may be asked to do

so on the AP exam!

4.2 EXAMPLE 6: A cell phone company is considering two different keyboard designs (A and B) for its new cell phone line. Researchers would like to conduct an experiment using 100 subjects who are frequent texters and 100 subjects who are not frequent texters. The 200 subjects will be asked to text several different messages in 5 minutes. The response variable will be the number of correctly typed words.

(a) Explain why a randomized blocked design might be preferable to a completely randomized design for this experiment.

Page | 14

(b) Outline and describe a randomized block experiment with 100 frequent texters and 200 novice texters. Describe how you would carry out the random assignment required by your design using technology.

4.2 EXAMPLE 7: Men, Women, and Advertising! Women and men respond differently to advertising. Researchers would like to design an experiment to compare the effectiveness of three advertisements for the same product.

(a) Explain why a randomized blocked design might be preferable to a completely randomized design for this experiment.

Page | 15

(b) Outline and describe a randomized block design using 300 volunteers (180 men and 120 women) as subjects. Describe how you would carry out the random assignment required by your design using a random number table.

4.2 EXAMPLE 8: Are standing pulse rates generally higher than sitting pulse rates? Describe how you would carry out an experiment on a high school class of 38 students utilizing a matched pairs design.

4.2 MULTIPLE CHOICE EXERCISES:

1. Can changing diet reduce high blood pressure? Vegetarian diets and low-salt diets are both promising. Men with high blood pressure are assigned at random to four diets: (1) normal diet with unrestricted salt; (2) vegetarian with unrestricted salt; (3) normal with restricted salt; and (4) vegetarian with restricted salt. This experiment has (A) one factor, the type of diet. (B) two factors, high blood pressure and the type of diet. (C) two factors, normal/vegetarian diet and unrestricted/restricted salt. (D) three factors, men, high blood pressure, and type of diet. (E) four factors, the four diets being tested.

Page | 16

2. In the experiment of the previous exercise, the 240 subjects are labeled 001 to 240. Software

randomly assigns 60 subjects to each of the four diets. This is

(A) a completely randomized design

(B) a randomized block design

(C) a matched pairs design

(D) an observational study

(E) an SRS

4.3: Using Studies Wisely

SECTION INTRODUCTION:

Researchers who conduct statistical studies often want to draw conclusions (make inferences)

that go beyond the data they produce. Here are two examples:

The U.S. Census Bureau carries out a monthly Current Population Survey of about

60,000 households. Their goal is to use data from these randomly selected households

to estimate the percent of unemployed individuals in the population.

Scientists performed an experiment that randomly assigned 21 volunteer subjects to

one of two treatments: sleep deprivation for one night or unrestricted sleep. The

examiners hoped to show that sleep deprivation causes a decrease in performance two

days later.

What type of inference can be made from a particular study? The answer depends on the

______________ of the study. A common mistake on the AP exam is to confuse inference

about the population and inference about cause and effect. While it is important to know how

to design good surveys and experiments, it is just as important to understand how to properly

use their results!

4.3 KEY VOCABULARY AND CONCEPTS: Inference About the Population, Inference About Cause

and Effect, Lack of Realism, Institutional Review Board, Informed Consent, Confidential vs

Anonymous

4.3-CONCEPT 1: Scope of Inference (Pages 261-263)

Well-designed sample surveys allow us to make inferences about the population from which we

sampled. ___________ ______________ is what allows us to generalize our results with

confidence. If the goal is to make an inference about cause and effect, we must use a

____________________ _____________________. Unless the experimental units were

randomly selected from a larger population of interest, we cannot extend our conclusions

beyond the individuals like those who took part in the experiment.

Page | 17

Were individuals randomly assigned to groups? Were individuals randomly selected?

Yes No

Yes

Inference about the population?

Inference about the population?

Inference about cause and effect?

Inference about cause and effect?

No Inference about the population?

Inference about the population?

Inference about cause and effect? Inference about cause and effect?

4.3 EXAMPLE 1: Many students insist that they study better when listening to music. A teacher

doubts this claim and suspects that listening to music actually hurts academic performance.

Here are four possible study designs to address this question at your school. In each case, the

response variable will be the students’ GPA at the end of the semester. What can we conclude

for each design?

(a) Design 1: Get all the students in your AP Statistics class to participate in the study. Ask

them whether or not they study with music on and divide them into two groups based

on their answer to this question.

(b) Design 2: Select a random sample of students from your school to participate in a study.

Then divide them into two groups as in Design 1.

(c) Design 3: Get all the students in your AP Statistics class to participate in a study.

Randomly assign half of the students to listen to music while studying for the entire

semester and the remaining half to abstain from listening to music while studying.

Page | 18

(d) Design 4: Select a random sample of students from your school to participate in a study.

Randomly assign half of the students to listen to music while studying for the entire

semester and the remaining half to abstain from listening to music while studying.

4.3-CONCEPT 2: The Challenges of Establishing Causation (Pages 263-265)

Well-designed experiments can be used to establish causation. However, _________ ____

_____________ in some experiments prevents us from seeing similar results outside the

laboratory setting. In some cases, it is not practical, safe, or ethical to conduct an experiment.

Even with strong evidence from observational studies, it is very difficult to establish a cause and

effect conclusion.

As mentioned before, it isn’t always practical or even ethical to do an experiment. Consider

these important questions:

Does texting while driving increase the risk of having an accident?

Does going to church regularly help people live longer?

Does smoking cause lung cancer?

To answer these cause-and-effect questions, we just need to perform a randomized

comparative experiment! Unfortunately, we can’t randomly assign people to text while driving

or to attend church regularly or to smoke cigarettes. The best data we have about these and

many other cause-and-effect questions come from ____________________________ studies. It

is sometimes possible to build a strong case for ______________ in the absence of

experiments. The evidence that smoking causes lung cancer is about as strong as non-

experimental evidence can be. To establish causation when we can’t do an experiment, we

must meet the following criteria:

1. The association is strong.

2. The association is consistent.

3. Larger values of the explanatory variable are associated with stronger responses.

4. The alleged cause precedes the effect in time.

5. The alleged cause is plausible.

Page | 19

4.3 EXAMPLE 2: Doctors had long observed that most lung cancer patients were smokers.

Comparison of smokers and similar nonsmokers showed a very strong association between

smoking and death from lung cancer. Could the association be due to a lurking variable? Is

there some genetic factor that makes people both more likely to get addicted to nicotine and to

develop lung cancer? If so, then lung cancer and smoking would be strongly associated even if

smoking had no direct effect on the lungs. Or maybe confounding is to blame. It might be that

smokers live unhealthy lives in other ways (diet, alcohol, lack of exercise) and that some other

habit confounded with smoking is a cause of lung cancer. How were these objections

overcome?

1. The association is strong.

2. The association is consistent.

3. Larger values of the explanatory variable are associated with stronger responses.

4. The alleged cause precedes the effect in time.

5. The alleged cause is plausible.

Page | 20

4.3-CONCEPT 3: Data Ethics (Pages 265-267)

Because some sample surveys and experiments have potential to cause harm to the

participants, it is important to consider data ethics when designing a study. Basic data ethics

include having an institutional review board that monitors the wellbeing of the participants.

Medical professionals are taught to follow the basic principle: “First, do no harm.” Next,

individuals must give their informed consent. And lastly, all individual data must be kept

confidential. The law requires that studies carried out by the federal government obey these

principles. But neither the law nor the consensus of experts is completely clear about the

details of their application. Data ethics is not covered on the AP exam. However, it is a very

important part of statistics.

CHAPTER SUMMARY:

This chapter is an important one in your study of statistics. After all, we cannot describe or

analyze data without collecting it first. Since one of the major goals of statistics is to make

____________________ that go beyond the data, it is critical that we produce data in a way

that will allow for such inferences. Biased data production methods can lead to incorrect

inferences. ______________ _______________ allows us to make an inference about the

______________________ as a whole. Well-designed __________________in which we

randomly assign _______________________ and control for lurking variables allow us to make

inferences about ___________ and _______________. We will learn how to perform these

inferences in later chapters. Your goal of this chapter is to be able to describe good sampling

and experimental design techniques and recognize when sampling or experimental design had

been done poorly. There is almost always a question about sampling or experimental design on

the free-response portion of the AP exam. Be sure to familiarize yourself with all of the

vocabulary and concepts from this chapter so you can answer that question with confidence!