Who are the participants?Creating a Quality Sample

47:269: Research Methods I

Dr. Leonard

March 22, 2010

The importance of a research sample

It is rarely possible to study an entire population, or all people the research is focused on, so to be more efficient we draw a sample

We use findings from the sample to infer conclusions about the population

Therefore, the quality of our conclusions about the population depends on how good, or representative our sample is

Population vs. Sample

Selecting an unbiased sample

Ideally, all samples would be without bias, meaning any individual in the population has an equal chance of being in the study

To do this a researcher needs to have access to every single member of the population, which is unlikely

Therefore, most of our samples have built in biases

Picking a sample

Bias in sampling

When a sample does not reflect some of the similarities or differences present in the population

Chance (by accident) OR Selection Bias (due to your sampling methods)

Example: I want to test whether or not a new pedestrian assist device will aid blind individuals’ in navigating busy intersections• What is my target population?• What is my accessible population? • How might I select participants to be in the study?

Gold standard: Random Sampling

Increases representativeness of population using PROBABILITY Characteristics of population are known so likelihood of getting

each type of individual can be estimated Each member of the population has an equal chance of being

selected into the sample Each possible sample of a given size (e.g., n=10) has an equal

chance of being selected from the population (N=100)

We want our samples to reflect all of the similarities and differences found in our target populations

Age Gender Ethnicity Political or Religious Affiliation Others specific variables of interest The range of characteristics or variety in the sample is almost

never as big as in the population but it should be close

Simple Random sampling: Each individual has an equal & independent chance of being

selected (like drawing names out of a hat) Stratified Random Sampling:

Divide population into strata, or sub-groups, before randomly selecting participants and then draw representative percentage from each strata

Systematic Sampling: Line up the population, randomly select a starting point, and take

every nth (lets say 10th) person Cluster Sampling:

Imagine being interested if learning about high-school students’ attitudes towards military service. You are interested in collecting a sample of 500 students.

Instead of randomly sampling to get 500 students, list the 30 schools and randomly select 5 schools.

Then test 100 students from each of those schools.

Types of Random/Probability sampling)

Stratified Random SamplingIf we wanted sample to represent the SES

breakdown of a given population…

Consider need for different recruiting methods

Systematic Sampling

Cluster Sampling

100

100

100

100

100

n = 500

Nonprobability Sampling Methods Characteristics of entire population not known Therefore, probability of selecting certain types of individuals

can not be estimated Sampling is nonrandom, but with an effort to maintain

representativeness and avoid bias.

Types of nonprobability sampling Convenience (e.g., Use the next ten people through the

door) Quota (e.g., need a certain number of individuals with a

specific trait in each group) Volunteer (e.g., undergraduate psychology majors) Purposive (e.g., looking for expert informants) Snowball (e.g., hard time finding participants so ask them

to recruit others they know)

Which kind of research is more likely to use random (probability) sampling? Experimental research (Any scientific study in which the

researcher systematically varies one or more variables, holding all others constant, to see if another variable is affected)

Why? Exerts more control than non-experimental Seeks to rule out extraneous variables & confounds More manipulation of independent variable More likely to use groups of participants

Reminders about experimental research

Any scientific study in which the researcher systematically varies one or more variables, holding all others constant, to see if another variable is affected

Intervention made or treatment given to observe effects (causal -- does X cause changes in Y?)

Independent variable must have two or more levels for comparison Most often accomplished by having experimental group(s) and

control group

True experiment if assignment of participants to groups is random

Random Sampling vs. Random Assignment

Random sampling is concerned with every individual in the population having an equal chance of being in the study

Random assignment is concerned with every participant in the sample having an equal chance of being in an experimental group (rules out bias)

True experimental designs (1) Pretest-posttest randomized control group design

Participants randomly assigned to groups Results in two “equal” or equivalent groups

Experimental and control group Both groups tested, or observed, before treatment (pretest) Treatment given (independent variable being manipulated) Both groups tested, or observed, again (posttest) Any pre-post gains or differences between groups are likely

attributable to a) the treatment or b) random error

R O X OR O O

True experimental designs (2) Beware pretest sensitization Sometimes the pretest can

cause problems if it also affects the groups in addition to the treatment

Posttest only randomized control group design Participants randomly assigned to groups Results in two “equal” or equivalent groups Treatment given (independent variable being manipulated) Both groups tested, or observed, again (posttest) Any posttest differences between groups are more likely attributable

to a) the treatment or b) random error

R X OR O

True experimental designs (3) Best of both designs… Solomon randomized four-group design

Still have participants randomly assigned to groups (may need more participants to get equivalent groups)

Like running two experiments at the same time Can test for pre-posttest gains and control for pretest

sensitization

R O X OR O OR X OR O

Problems in True Experimental Designs

What if the observed change is NOT due to the treatment but due to random error? We may find a design lacks validity (just like measures can lack validity)

Experimental validity can be…

Internal - the degree to which an experiment’s methods are controlled and free from confounds

External - the degree to which findings from research can be generalized to the real world context (other populations, other settings, other times???)

Threats to internal validity of experiment Any factors that lessens the degree of control

Pre- and post-test problems History - events that occur outside of the study that may affect

the outcome Maturation - changes within individual during the study that

may affect the outcome Instrumentation - changes in the measures between pre- and

post-tests; could be capturing different construct Testing - any feature of the test or task that could change the

responses the second time; practice effect Statistical regression to the mean - over time, scores tend to

move toward the average

Threats to internal validity of experiment

Participant problems - any problems related to how the individuals participating in the study may challenge the validity of the findings; most common:

Selection effects - participants in samples should be equivalent to each other except for the independent variable but human beings are not clones

Also, selection into the study should be totally random AND each participant should have an equal chance of being assigned to groups

Morality/Attrition - every single participant may not complete the study, which can be especially problematic if a certain type of participant is more likely to drop out

Threats to external validity of experiment

Was our sample good enough that we can we generalize our findings???

Population APopulation B Population C

Setting A,Time A

Setting B,Time B

Setting C,Time C

Sample