29
Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

Embed Size (px)

Citation preview

Page 1: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

Evaluating the Experiment from the Inside: Internal Validity

Taking a Broader Perspective: The Problem of External Validity

Handling a Nonsignificant Outcome

Page 2: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What is internal validity?

Evaluating the Experiment from the Inside

An experiment is internally valid when the effects on the dependent variable are due to the independent variable.

An internally valid experiment is free of confounding.

Page 3: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What is a manipulation check?

Evaluating the Experiment from the Inside

A manipulation check evaluates how well the experimenter manipulated the experimental situation.

A manipulation check determines whether subjects followed directions and were appropriately affected by our treatments.

Page 4: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What did Orne (1969) mean by a pact of ignorance?

Evaluating the Experiment from the Inside

Subjects expect their data to be discarded if they guess the experimental hypothesis, and don’t volunteer this information to the experimenter.

Experimenters don’t want to test additional subjects and may take subject reports at face value.

Page 5: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

How can researchers overcome this problem?

Evaluating the Experiment from the Inside

Debrief subjects after the experiment and convey that you want to know if they guessed the hypothesis.

Provide incentives for guessing the hypothesis.

Page 6: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

Which mistakes could produce this threat to internal validity?

Evaluating the Experiment from the Inside

Selecting the wrong statistical test Using a t-test to analyze ordinal data.

Improperly using a statistical testCalculating multiple t-tests.

Drawing the wrong conclusions from the test

Reporting p = .07 as a trend.

Page 7: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

Explain external validity.

The Problem of External Validity

An experiment is externally valid when its findings can be extended to other situations and populations.

Page 8: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What two requirements must an externally valid study satisfy?

The Problem of External Validity

1. The experiment must be internally valid.

2. The experimental findings can be replicated.

Page 9: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What does it mean to generalize across subjects? Why is this important?

The Problem of External Validity

The findings can be extended to a larger group than our sample.

Generalizing across subjects is critical to the external validity and usefulness of experimental findings.

Page 10: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

Which problems prevent us from generalizing across subjects?

The Problem of External Validity

The samples used in psychological research are often biased and may not represent the larger population.

The samples may not always represent even college sophomores since we heavily depend on volunteers.

Page 11: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

Explain generalizing from procedures to concepts.

The Problem of External Validity

Experimental variables like anger may have multiple operational definitions.

When we generalize from our experimental results, we move from discussing our specific operational definition of anger to discussing the concept of anger itself.

Page 12: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

Why is this a problem in research?

The Problem of External Validity

It is dangerous to generalize from a single experiment’s operational definition of anger.

We cannot be sure of the reliability or validity of our procedures.

Page 13: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What is research significance?

The Problem of External Validity

A study achieves research significance when its findings clarify or extend knowledge gained from previous studies and raise implications for broader theoretical issues.

Page 14: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

When should we question novel findings?

The Problem of External Validity

We should question novel findings when they contradict prior findings that have been successfully replicated.

The burden of proof is on the experimenter who claims novel findings to explain this discrepancy.

Page 15: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

Why do we want to generalize beyond the laboratory?

The Problem of External Validity

We want to generalize beyond the laboratory to increase the external validity of our findings.

Page 16: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What is the problem in extending laboratory findings?

The Problem of External Validity

Since extraneous variables are uncontrolled in real world setting and operate in complex combinations, they can modify the influenceof our individual variables.

Page 17: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What is the trade-off between laboratory and field experiments?

The Problem of External Validity

The trade-off is between the laboratory’s more precise control of extraneous variables and the field experiment’s greater realism and external validity.

Page 18: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What did Hanson (1980) find about the agreement between laboratory and field studies?

The Problem of External Validity

Hanson (1980) found that more laboratory than field studies reported a positive correlation between reported attitudes and behavior.

Page 19: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

When can we extend results from an experiment to everyday life?

The Problem of External Validity

We can’t confirm external validity until additional studies are completed in field settings.

Researchers can increase and verify the external validity of laboratory findings using aggregation, multivariate designs, nonreactive measurements, field experiments, and naturalistic observation.

Page 20: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What is aggregation? What is its logic?

The Problem of External Validity

Aggregation is the grouping together and averaging of data to increase external validity.

Combining the results of experiments with different subjects and methodologies increases the generality and external validity of our findings.

Page 21: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

How is this logic shared by meta-analysis?

The Problem of External Validity

Meta-analysis uses statistical analysis to combine and quantify data from many comparable experiments to calculate anaverage effect size.

Page 22: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

How does aggregation establish external validity?

The Problem of External Validity

Aggregation establishes external validity by combining the results of experiments performed using different subjects, stimuli and/or situations, trials or occasions, and measures.

Page 23: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What is a multivariate design?

The Problem of External Validity

A multivariate design studies multiple DVs.

For example, a study of repetitive strain places a computer keyboard at different distances from the subject IV) and measures the effect on three different muscle groups (3 DVs).

Page 24: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

What is the advantage of multivariate designs?

The Problem of External Validity

Multivariate designs allow us to study the effect of an independent variable on combinations of dependent variables.

These designs better simulate the complexity of the real world than univariate designs and provide more detailed information.

Page 25: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

How are multivariate experiments analyzed?

The Problem of External Validity

We analyze multivariate experiments with a multivariate analysis of variance (MANOVA).

Page 26: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

How should researchers handle a nonsignificant outcome?

Handling a Nonsignificant Outcome

Accept the outcome, don’t reframe your result as “almost significant.”

Examine the experimental procedures for design flaws.

If the design appears sound, decide whether the hypothesis was reasonable.

Page 27: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

How should we handle the possibility of faulty procedures?

Handling a Nonsignificant Outcome

Check for possible causes of a nonsignificant outcome like:

1.confounding

2.extraneous variables that increase within-subjects variability

3.weak manipulation of the IV

Page 28: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

How should we handle the possibility of faulty procedures?

Handling a Nonsignificant Outcome

4. inconsistent or flawed procedures

5. ceiling and floor effects

6. insufficient power

Page 29: Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

How should we handle the possibility of a faulty hypothesis?

Handling a Nonsignificant Outcome

If previous studies supported the hypothesis and ours did not, look for differences in experimental design or sample.

If there was no previous support and our design and execution were good, we may have to revise or discard our hypothesis.