Some (Simplified) Steps for Creating a Personality Questionnaire

• Generate an item pool• Administer the items to a sample of people• Assess the uni-dimensionality of the item pool• Assess the reliability of the measure• Assess the validity of the measure

Generate an Item Pool

• Sample from the “universe” of possible items

• Rational/theoretical approach• Observation (clinical, narrative, interviews, descriptions

of others)

Administration

• Administer the questionnaire to a large sample of individuals.

• Can use full item set• Can randomly sample items from full set

Dimensionality

• We want to ensure that there is only one major latent variable that the items have in common.

– The statistical tools that we use tend to assume uni-dimensionality.

– Multi-dimensional constructs are treated separately.

• Principal components analysis is sometimes used to determine whether one major variable underlies the item responses.

Choose items to factor

Select “Scree plot” option in “Extraction” options

Choose “Varimax” under “rotation” options

Total Variance Explained

2.589 51.777 51.777 2.589 51.777 51.777

.926 18.524 70.301

.653 13.066 83.366

.522 10.437 93.803

.310 6.197 100.000

Component1

2

3

4

5

Total % of Variance Cumulative % Total % of Variance Cumulative %

Initial Eigenvalues Extraction Sums of Squared Loadings

Extraction Method: Principal Component Analysis.

Reliability

• Reliability: the extent to which measurements are free of random errors.

• Random error: nonsystematic mistakes in measurement– misreading a questionnaire item– observer looks away when coding behavior– nonsystematic misinterpretations of a behavior

Reliability

• What are the implications of random measurement errors for the quality of our measurements?

Reliability

• O = T + E

O = a measured score (e.g., performance on an exam)

T = true score (e.g., the value we want)

E = random error• The error becomes a part of what we’re measuring• Once we’ve taken a measurement, we have an equation with two unknowns. We can’t separate the relative contribution of T and E.

10 = T + E

Reliability: Do random errors accumulate?

• Question: If we sum or average multiple observations, will random errors accumulate?

Reliability: Do random errors accumulate?

• Answer: No. If E is truly random, we are just as likely to overestimate T as we are to underestimate T.

Reliability: Do random errors accumulate?

Note: The average of the seven O’s is equal to T

O = T + EObs. 1 10 10 0Obs. 2 9 10 -1Obs. 3 10 10 0Obs. 4 11 10 +1Obs. 5 8 10 -2Obs. 6 10 10 0Obs. 7 12 10 +2

Average 10 10 0

Reliability: Implications

• These demonstrations suggest that one important way to help eliminate the influence of random errors of measurement is to use multiple measurements.– operationally define latent variables with multiple

items or measures– use more than one observer when quantifying

behaviors

Reliability: Estimating reliability

• Question: How can we estimate the reliability of our measurements?

• Answer: Two common ways:

(a) test-retest reliability

(b) internal consistency reliability

Reliability: Estimating reliability

• Test-retest reliability: Reliability assessed by measuring something at least twice at different time points.

• The logic is as follows: If the errors of measurement are truly random, then the same errors are unlikely to be made more than once. Thus, to the degree that two measurements of the same thing agree, it is unlikely that those measurements contain a large proportion of random error.

Reliability: Estimating reliability

• When we conducted the subliminal recorded experiment, we had an opportunity to assess the test-retest reliability of a 5-item measure of self-esteem.

• The test-retest correlation was approximately .86, suggesting that there was very little random error present in our measurements.

• The test-retest correlation is an estimate of the proportion of true score variance present in a set of measurements.

• About 74% of the variation is “true score” variation; the remaining 26% is error.

Less error(off by 1 point)

More error(off by 2 points)

Time1

Time2

Time1

Time2

Person A 1 2 1 3Person B 7 6 7 5Person C 2 3 2 4Person D 6 5 6 4Person E 3 4 3 5Person F 5 4 5 3Person G 4 5 4 6

Reliability: Estimating reliability

• Internal consistency: Reliability assessed by measuring something at least twice within the same broad slice of time.

Split-half: based on an arbitrary split (e.g, comparing odd and even, first half and second half)

Cronbach’s alpha (): based on the average of all possible split-halves

Alpha reliability

• Click on “scale” in the “analyze” menu

• Choose “reliability analysis”

• Important: When computing alpha in SPSS, you must make sure all the items are keyed in the same direction. Notice our use of the reverse keyed items in this example.

Reliability: Final notes

• An important implication: As you increase the number of indicators, the amount of random error in the averaged measurement decreases.

• An important note: Common indices of reliability are based on correlations and range from 0 to 1; higher numbers indicate better reliability (i.e., less random error).