The Basics of The Basics of ExperimentationExperimentationCh7 – Reliability and ValidityCh7 – Reliability and Validity

ReliabilityReliability

ReliabilityReliability – consistency and – consistency and dependability. Should yield similar results dependability. Should yield similar results across experiments.across experiments. Interrater reliabilityInterrater reliability – different observers – different observers

score the same behavior.score the same behavior. Test-retest reliabilityTest-retest reliability – measure behavior – measure behavior

twice with the same test.twice with the same test. Interitem reliabilityInteritem reliability – different parts of a test – different parts of a test

measuring the same variable are consistent.measuring the same variable are consistent.

ReliabilityReliability

Interitem reliability – internal consistencyInteritem reliability – internal consistencye.g., in a multiple item questionnaire that e.g., in a multiple item questionnaire that

measures a single construct variable, the measures a single construct variable, the internal consistency is evaluated among the internal consistency is evaluated among the set of items using statistical tests.set of items using statistical tests.

1.1. Split-half reliabilitySplit-half reliability – split the test into – split the test into two halves and compute the correlation two halves and compute the correlation (coefficient of reliability) between halves. (coefficient of reliability) between halves.

2.2. Cronbach’s Cronbach’s αα – correlation of each test – correlation of each test item with every other itemitem with every other item

ValidityValidity

Validity – did you measure what you Validity – did you measure what you intended to measure?intended to measure? Face validityFace validity – procedure is self-evident; – procedure is self-evident;

works with nonconstruct variables that can be works with nonconstruct variables that can be directly manipulated and measured. (e.g., directly manipulated and measured. (e.g., measuring “pupil size” with a ruler). Least measuring “pupil size” with a ruler). Least stringent type of validity.stringent type of validity.

Content validityContent validity – does the content of the – does the content of the measure fairly reflect the content of the measure fairly reflect the content of the variable we are trying to measure.variable we are trying to measure.

Evaluating Operational DefinitionsEvaluating Operational Definitions

Content validityContent validity Does the content of our measure reflect the Does the content of our measure reflect the

content of the qualities of the variable we content of the qualities of the variable we want to measure.want to measure.

Content validity also means that a test does Content validity also means that a test does not measure other qualities not measure other qualities

Evaluating Operational DefinitionsEvaluating Operational Definitions

Predictive validity Predictive validity – do measures of a – do measures of a dependent variable predict actual behavior dependent variable predict actual behavior or performance? E.g. a questionnaire may or performance? E.g. a questionnaire may measure a person’s desire to affiliate but measure a person’s desire to affiliate but will they actually seek out others when will they actually seek out others when given the opportunity.given the opportunity.

Evaluating Operational DefinitionsEvaluating Operational Definitions

Concurrent validity Concurrent validity – is like predictive – is like predictive validity in that it compares scores on a validity in that it compares scores on a measure with an outside criterion, but it is measure with an outside criterion, but it is comparative rather than predictive; you comparative rather than predictive; you compare scores with another known compare scores with another known standard for the variable being studied.standard for the variable being studied.

Elevated Plus MazeElevated Plus Maze

Used as a measure of anxiety in rodents. Anxious animals spend more Used as a measure of anxiety in rodents. Anxious animals spend more time in the enclosed arms and less time on the open arms.time in the enclosed arms and less time on the open arms.

Zero MazeZero Maze

Open Field TestOpen Field Test

Thigmotaxis (time near the wall of an open field) is another measure of Thigmotaxis (time near the wall of an open field) is another measure of anxiety. If concurrent validity is high then the scores on the plus maze anxiety. If concurrent validity is high then the scores on the plus maze should be highly correlated with thigmotaxis scores in the open field.should be highly correlated with thigmotaxis scores in the open field.

Evaluating Operational DefinitionsEvaluating Operational Definitions Construct validityConstruct validity – deals with the – deals with the

transition from theory to research transition from theory to research application. Start with a general idea of the application. Start with a general idea of the qualities of the construct, then convert the qualities of the construct, then convert the idea into an empirical test.idea into an empirical test.

Have I successfully created a measure of Have I successfully created a measure of the construct of interest?the construct of interest?

Example, rats have a natural fear of Example, rats have a natural fear of predation associated with increased anxiety predation associated with increased anxiety in open areas - design test that measures in open areas - design test that measures the tendency to avoid open areas.the tendency to avoid open areas.

Evaluating Operational DefinitionsEvaluating Operational Definitions

Construct validity Construct validity – tests of construct – tests of construct validity are statistical and theoretical. validity are statistical and theoretical. Does the data make sense in the context Does the data make sense in the context of the overall theoretical framework?of the overall theoretical framework?

Intelligence – maze bright versus maze Intelligence – maze bright versus maze dull rats.dull rats.

Internal validityInternal validity

The degree to which a causal relationship The degree to which a causal relationship can be established between the can be established between the antecedent condition and behavior.antecedent condition and behavior.

Three concepts tied to the problem of Three concepts tied to the problem of internal validity:internal validity: Extraneous variablesExtraneous variables ConfoundingConfounding Threats to internal validityThreats to internal validity

Extraneous variablesExtraneous variables

factors that are not the main focus of the factors that are not the main focus of the experiment. Other variables besides the IV experiment. Other variables besides the IV and DV may change throughout the and DV may change throughout the experiment: experiment:

Differences among subjectsDifferences among subjects Time of day of testingTime of day of testing Order of testingOrder of testing Inconsistent treatment Inconsistent treatment Experimenter’s fatigueExperimenter’s fatigue Equipment failuresEquipment failures

ConfoundingConfounding

when the value of an extraneous variable when the value of an extraneous variable changes systematically across different changes systematically across different conditions of the experiment.conditions of the experiment.

Changes we see in the DV can be Changes we see in the DV can be explained equally well by the IV or the explained equally well by the IV or the extraneous variable.extraneous variable.

This is the Threatdown!This is the Threatdown!

Threats to internal validityThreats to internal validity

HistoryHistory (one group tested together) (one group tested together) MaturationMaturation (boredom or fatigue) (boredom or fatigue) TestingTesting (previous administration of a test) (previous administration of a test) Instrumentation (feature of the measuring instrument Instrumentation (feature of the measuring instrument

changes)changes) Statistical regressionStatistical regression (subjects assigned to conditions (subjects assigned to conditions

based on extreme scores)based on extreme scores) SelectionSelection (no random assignment) (no random assignment) Subject mortalitySubject mortality (subjects drop out) (subjects drop out) Selection interactionsSelection interactions (if subjects were not randomly (if subjects were not randomly

assigned to groups a selection threat could interact with assigned to groups a selection threat could interact with any of the others threats that may have affected one any of the others threats that may have affected one group but not others)group but not others)