Scale Construction

7/28/2019 Scale Construction

1/111

Scale Construction

et o s, tat st cs, an o e s

Fridtjof Nussbeck, University of Zurich


2/111

Outline

psychological assessment

criteria for valid assessment

scale development

ISSAS 2009 - scale construction 2

p ases o sca e eve opment statistics

MTMM models

outlook

2ISSAS 2009 - scale construction


3/111

Psychological Measurement

systematic measurement of a persons behavior

different strategies to assess target persons

inferences (and clinical judgments) drawn


rom t e resu ts



4/111

I. Assessment Methods direct observation

psychophysiological measurement questionnaires


-emotional states / mood / well being

single-shot studies

repeated measures

diary methods

ambulatory assessment



5/111

Measurement and ValidityAll decisions (assessments) in psychology should be basedon the best information available. Information is best when

it is objective, reliable, valid, and specific to a givenproblem.

Reliable Measures:


measurement error is small Cronbachs alphaValid Measures:no impact on measurement scores than those one wants to

measure Fit into nomological net

Burns & Haynes (2006); Messick (1995); Courvoisier, Nussbeck, Eid, Geiser, & Cole (2008)

55


6/111

II. Scale Construction

not just:

looking for some words that tap the construct


Standards forEducational and Psychological Testing

(American Psychological Association)



7/111

When to Construct a New Scale? no scale exists measuring a specific construct

existing scales do not represent the constructadequately:

lack of reliabilit


lack of validity

outdated (old words; meaning of words changed;attitudes changed)

trait measure vs. state measure

insensitive for changes



8/111

Phases of test / scale construction

ISSAS 2009 - scale construction 88ISSAS 2009 - scale construction


9/111

Construct Definitionscope of the scale (level of abstraction)

what is to be measured?

broad or narrow construct?


we - e ng vs. spec c emo on

definition of the construct in literature?

adopt a definition vs. work out an own definition

clearly describe what is meant by your construct(and what is not meant)



10/111

Construct Definitionaspects of construct

continuum vs. categories

frequency / intensity of experiencing emotions


,

scores on items can be combined using (weighted)

means

emotional reactions (baroque, rational, positive

reappraising)

response patterns can be analyzed yielding classes ofreaction styles

1010


11/111

Construct Definition unidimensional vs. multidimensional

frequency of experiencing specific positive emotions

love, affection, intimacy, security

joy, happiness, cheerfulness, contentment

fre uenc of ex eriencin ositive affect


Love, affection, joy, happiness

facets of a construct

positive affect may comprise

love Happiness

pride



12/111

Recommendation write out a brief, formal description of the

construct relate it to other constructs


dimensionality / facetsmay also help to avoid known problems with

respect to unclear instructions, problematic

response formats, etc.



13/111


14/111

Design Scalethe initial item pool

is broader than ones own theoretical view about theconstruct

includes multiple items for each potential facet /dimension


includes also items that will finally proof to be distinct ortangential to the construct

search for aspects in literature (scientific, but also fiction,dictionnaries, etc.)

ask friends and family

item generation is a creative act



15/111

Design Scalebasic principles of item writing

simple, straightforward, and appropriatelanguage


a equa e o rea ng eveno trendy expressions / colloquialisms

Ex.:

Do you feel happy? never sometimes always



16/111


one aspect at a time

I feel happy and beloved


I o not insu t peop e ecause it is mora y wrong individuals must differ on items

constants are useless

I am the happiest individual alive



17/111


avoid frequencies in item wordingSometimes, I am happy


terms

I worry about neuroticism

avoid negatives to reverse meaning of an item,but include negative aspectsI am not happy I am sad



18/111

Multidimensional Mood Questionnaire


tems o t e easant-Unpleasant dimension

Steyer, Schwenkmezger, Notz, & Eid (1994)


19/111

Semantic Opposites?

vs.



20/111

Semantic Opposites?



21/111

Design Scaleresponse format

depends on introductionHow often do you feel .

frequency format (never, always)


Do you agree to. Agreement (not at all, very much so)

How much does correspond to you

similarity (not at all like me, very much like me)

How do you judge

evaluation (terrible, excellent)



22/111

Design Scaleresponse format

avoid middle category to increase variability

not always desirable

avoid too many categories, respondents cannot


eren a e a equa e y ana og sca e

avoid check-lists response bias

avoid forced choice formats

indicates relative strength of alternatives and cannot becompared across individuals (no normative interindividual

information)



23/111

Design Scale standard metrical analysis techniques work well with

five or more categories However, labels from 1 to 5 do not guarantee a

metric scale!



24/111

Pilot Test small sample

critique scaleambiguous or confusing items


mismatc o items an sca e



25/111

Administration and Item Analysislarge set of items (all potential items)

considerable sample size (about 100)

correlation with criteria (nontest-criteria)


internal consistency (interitem structure) item response theory (focused on latent trait)



26/111

Validate and NormValidity:

does the scale measure, what it intends tomeasure?

content validit


convergent validity

criterion validity

discriminant validity

Norms:

what are the properties of the distribution of scores

for a given population?26ISSAS 2009 - scale construction


27/111

StatisticsBefore you start:

items recoded?save recoded items using a new name


look at distributions!

ISSAS 2009 - scale construction 27Wut

543210

Frequency

300

200

100

0

Wut

Mean =1.86

Std. Dev. =0.71

N =481

1

2

3

4

Groll

AAAAAA


28/111

Statisticsfirst simple analysis:

item correlations with criteriondichotomous with interval / dichotomous


interval with interval Pearson

dichotomous / ordinal with ordinalWilcoxon



29/111

Internal Structureinteritem correlations

factor analysis for metrical outcomesstandard factor analysis:


exp ora ory ac or ana ys s

confirmatory factor analysis

factor analysis for ordinal data

special estimators (WLS / WLSMV) required



30/111

Interitem Correlations



31/111

Exploratory Factor Analysishow many underlying latent variables exist in

the current set of items?

ISSAS 2009 - scale construction 31ISSAS 2009 - scale construction 31

i i i i

or

j 2Y j j j = + +

i 1Y i i i = + + for some items i

for some (other) items j


32/111

Exploratory Factor Analysisdata driven approach to reduce complexity

model finding

depends strongly on input (items)


you get out, what you put in


33/111

Criteria for Factor AnalysisHow many factors are needed?

scree plot eigenvalues

parallel analysis


explained variance (goodness of fit coefficients)

number of factors does not depend on extraction

methodfor ease of interpretation choose (varimax) rotation


34/111

Scree Plot



35/111

Eigenvalues & Explained Variance



36/111

Factor Loadings



37/111

Exploratory Factor Analysisempirical structure should match the

theoretically expected structure1st case:


items oa on one speci ic actor su imensions factors can be correlated)



38/111

(Sub-) Dimensions



39/111

Exploratory Factor Analysisempirical structure should match the

theoretically expected structure2nd case:


Items oa on one common actor an on onespecific factor (facets)



40/111

Facets



41/111

Confirmatory Factor Analysistheory driven

Model testing

structure is known beforehand


test if assumed factor structure explainsobserved variance-covariance matrix

structural equation modeling software

needed (AMOS, Mplus, LISREL)

Bollen (1989)



42/111

simple example

LOVE

affection

love


JOY

security

joy

fortune

happin.

Conent.

C it i f C fi t F t


43/111

Criteria for Confirmatory Factor

Analysisgoodness of fit:

comparison of the expected and the observedcovariance matrix

?Cov Y Y Cov Y Y =


test

RMSEA

comparative fit indices (TLI: Tucker-Lewis Index;

CFI: Comparative Fit Index)

Schermelleh-Engel, Mller, & Moosbrugger (2003)

1 5 1 5 ( , ) ( , )Cov Y Y Cov LOVE JOY =


44/111

Internal Consistencyestimation:

Cronbachs alpha (should be > .70)

2 2

21

T T

T

k

k

=


association of one item with scale

(corrected) item-total correlation (discrimination)

broad constructsmedium rit

narrow construct higher rit



45/111

Reliability of itemsCTT (Classical Test Theory):

iY i = + +


reliability:2

ii

Var( )Rel(Y )

Var( )iY

=


46/111

Validityitem pool build in such a way that it is content valid

reliability proofedcriterion validity

concurrent: correlation with another measure that shall be


predictedpredictive: prediction (e.g., via regression) of a later event

(test-score)

construct validity


47/111

Validityconstruct validity

Campbell & Fiske (1959): Multitrait-Multimethod(MTMM) Matrix

conver ent validit


high associations between different measures of the sameconstruct


trait-method-unit (TMU) more than one method is needed

MTMM analysis


48/111

MTMM analysis

foundations convergent validity

high associations between different measures of the sameconstruct



Lower associations between measures of differentconstructs

trait-method-unit (TMU)

scores of a measure depend on underlying trait (construct)but also on method

more than one method is needed

disentangle influences due to trait and method


49/111

Methods?

different raters

self-report, friend, acquaintance teacher, pupils

different tests


BDI, Hamilton Scale different kinds of data

cortisol, heart rate, stress rating

productivity rate, ratings of preformance, number ofproduction errors

(measurement occasions)

sub-scales / items


50/111

Multitrait-Multimethod Matrix

heuristic inspection (Campbell & Fiske, 1959)


Monomethod blocks:reliabilities on diagonalHeterotrait-Monomethod triangle



51/111

Multitrait-Multimethod Matrix

heuristic inspection (Campbell & Fiske, 1959)


heteromethod-blocksconvergent validity coefficients (bold)heterotrait-heteromethod-triangles


52/111

convergent and discriminant validity

convergent validitycorrelations on validity significantly larger than 0?

discriminant validitycorrelations besides the validity diagonal smaller than on

validity diagonal?


monotrait-heteromethod-correlations (MHC) should behigher than than heterotrait correlations of the samevariables

correlations of different traits should be similar acrossmono- and heteromethod blocks (pattern of associations

between traits should be similar across methods)method effects

degree to which monomethod correlations are higher thanheteromethod correlations


53/111

Critique on C&F approach

correlations depend on reliabilities

What is a high correlations? What are different patterns of correlations


w at i erences can e consi ere ig

no statistical model to explain genesis of data

no testable consequences

trait- and method effects are not disentangled


54/111

Modern MTMM approaches

confirmatory factor analysis (CFA)

CFA-MTMM / SEM-MTMM Models

Separation of true and error components of scores

specification of measurement models


a ow or mo e es ng

some allow for a separation of trait- and methodcomponents

Quantification of trait- and method effects possible

factor analysis


55/111

y

Y1 = 1 + 1 + 1Y

2 = 2 + 2 + 2

Y1 Y2

1 2

ISSAS 2009 - scale construction 55 55

1 2

observed variables (Y1, Y2)

: latent variable (common factor)

1, 2: residualvariablen [measurement error,

uniqueness, unique factors]

loadings i; intercepts

i

factor analysis


56/111

y

Y1 = 1 + 1 + 1Y

2 = 2 + 2 +

2

Y1 Y2

1 2


Important indices:

- communality

- reliability

2 ( )( ) ( )

ii

i

VarRel Y Var Y

=

1 2

2 factor model


57/111

F1:Extraversion

Var(F1) F2:Repair

Var(F2)

Cov(F1, F2)

2 factor model


E1 E2

1 2

Var(1) Var(2)

R1 R2

3 4

32

Var(3) Var(4)

21 4211

rules of calculations for variances and


58/111

covariances

E1 = 11F1 + 1 E2 = 21F1 + 2

Var(aX+ bY) = a2 Var(X) + b2 Var(Y) + 2ab Cov(X,Y)


Cov(a1X1 + b1Y1, a2X2 + b2Y2) =

a1a2 Cov(X1,X2) + a1b2 Cov(X1, Y2) + b1a2 Cov(Y1,X2) + b1b2 Cov(Y1, Y2)

X, Y: variables a, b : constants

2 factor model


59/111

Var(E1)

Cov(E2, E1) Var(E2)

Cov(R1, E1) Cov(R1, E2) Var(R1)

Cov(R

2,E

1)Cov

(R

2,E

2)Cov

(R

2,R

1) Var(R2)EXTRA1 EXTRA2 REPAIR1 REPAIR2

EXTRA1 0.707

sample-

covariances


R2

E1

Cov(F1, F2) R2 E2 Cov(F1, F2) R2 R1 Var(F2) R22 Var(F2) + Var(4)

E1

2

Var(F1) + Var(

1)E2

E1

Var(F1) E22 Var(F1) + Var(2)

R1

E1

Cov(F1, F2) R1 E2 Cov(F1, F2) R12 Var(F2) + Var(3)

EXTRA2 0.434 0.584

REPAIR1 0.073 0.073 0.359

REPAIR2 0.113 0.114 0.256 0.320

S

Si l I di t MTMM M d l


60/111

Single Indicator MTMM Models

Correlated Traits Model

Correlated Traits-Correlated Uniqueness Model Correlated Traits-Uncorrelated Methods Model


Correlated Traits Model


61/111



62/111


2

( ) ( ) ( )

( )

jk jk jk

jk jk k jk

jk k jk

Y E

Var Y Var T Var E

Var T E

= +

= +

= +

Implications


63/111

Implications

systematic variance in observed variables only

caused by traits

correlations between latent variables =discriminantvalidity


reliability = consistency (convergent validity) residuals consist of measurement error and method

effects (specific to TMU)

Correlated Traits-Correlated


64/111

Uniqueness Modell



65/111


( ) ( ) ( )

( )

jk jk jk

jk jk k jk

jk k jk

Y E

Var Y Var T Var E

Var T E

= +

= +

= +

Implications


66/111

Implications

systematic variance in observed variables onlycaused by traits

correlations between latent variables =discriminantvalidity

=


residuals consist of measurement error and methodeffects (specific to TMU)

residuals belonging to one method may correlate(method effects generalize)

no separation of effects due to method andmeasurement error

Correlated Traits-Uncorrelated


67/111

Methods Modell



68/111


2 2

( ) ( ) ( ) ( )

( ) ( )

jk jk jk

jk Tjk k Mjk j jk

Tjk k Mjk j jk

Y E

Var Y Var T Var M Var E

Var T Var M E

= +

= + +

= + +


69/111

consistency :

2

2 2( )( )

( ) ( )

Tjk k

jk

Tjk k Mjk j

Var TCON YVar T Var M

=+


Method-specificity:2

2 2

( )( )

( ) ( )

Mjk j

jk

Tjk k Mjk j

Var MMS Y

Var T Var M

=

+

Implications


70/111

Implications

systematic variance in observed variables caused bytraits and method

each and every method deviates from the trait (nobest method)


corre at ons etween atent var a es = scr m nant

validity reliability = consistency (convergent validity) +

method-specificity

residuals consist of measurement error method effects generalize across traits

Correlated Traits-Correlated Methods


71/111

Minus 1 Modell



72/111


2

( ) ( ) ( )

( )

jk jk jk

jk Tjk k jk

Tjk k jk

Y E

Var Y Var T Var E

Var T E

= +

= +

= + 2 2

( ) ( ) ( ) ( )

( ) ( )

jk jk Mjk j jk

jk Tjk k Mjk j jk

Tjk k Mjk j jk

Y M E

Var Y Var T Var M Var E

Var T Var M E

= + +

= + +

= + +


73/111

consistency :

2

2 2( )( )

( ) ( )

Tjk k

jk

Tjk k Mjk j

Var TCON YVar T Var M

=+


Method-specificity:2

2 2

( )( )

( ) ( )

Mjk j

jk

Tjk k Mjk j

Var MMS Y

Var T Var M

=

+

Implications


74/111

p

one method is gold standard (reference method)

contrast of other methods against reference

trait factor is true-score of reference method (doesnot chan e if methods are added to the model


comprises trait and method effects of referencemethod)

method effects are residuals in a latent regression

variance components of trait and method effectsestimable (consistency and method specificity)

Multiple Indicator MTMM-Models


75/111

p

basis:

TM model choice of model depends on method structure


TM-Model


76/111

Latent variables represent

TMUseparation of systematic and


latent variables consist of trait

and method influences

basis for further analysis

Choose among MTMM Models


77/111

g

method structure

interchangeable raters / methods? students in courses

friends


employees (at same level in hierarchy) structurally different raters / methods?

self-, friend, and acquaintances

student and teacher ratings und Lehrerrating group leader and subordinates

Interchangeable Raters


78/111

g

ANOVA:

Mean is expected value in factor leveldeviations from mean are independent


mean

Rating A1

Rating A2

Rating B2

Rating B1

Met. A

Met. B

1

E111

CTUM


79/111

M12

Y112

M211

1

Y212

T111

1

Y1211

T211

Y111

Y211

M11

M211

M21

11

T211Trait 1

Rater 1

Rater 2

Rater 1


M22

T121

Y222

M32Y132

T131

Y232

1Y131

Y231

221

Y122

M231

M221

M31

M231

M221

1

1

Trait 3

Trait 2

Rater 2

Rater 1

Rater 2

Structurally Different Raters


80/111

rater differ in the quality of their ratings

standard? referenz?

standard is best approximation of true score (Platonic) true-score is trait


deviations from this true-score are method-effects

Trait(Trait+Method

of Standard)

Rating A1

Rating A2

Rating B2

Rating B1Met. B


81/111

TraitRating A1


of Standard)Rating A2

Rating B2

Rating B1Met. B

CTC(M-1)Y111


82/111

M112

T111

Y112

Y211

Y212

Y113

Y221

Y213

Y121

T121

M113

T111


Y222

Y122

Y223

Y123

Y232

Y132

Y231

Y131

Y233

Y133

T131

M122

M123

M132

M133

T121

T131

Multiple Indicator MTMM Models


83/111

separation of measurement error fromsystematic influences

model choice based on method structure


determined method factors are trait specific (one method

may overestimate DIF but underestimate DDF)

method factors can be related to externalvariables

CTUM and CTC(M-1) model


84/111


CTUM model


85/111


CTC(M-1) model


86/111


Research (common) practice


87/111

content validity is generally assured

many authors report Cronbachs , (corrected)item-total correlations


exp oratory actor ana ysis are requent y use

confirmatory factor analysis is less frequentlyused

MTMM models are found in larger researchprograms


Norms


88/111

Scales of measurement for most constructs isarbitrary

Meaning of scores can only be determined in relationto some frame of reference


(across and within individuals) as frame large sampleRepresentative sample (multiple samples)

descriptive statistics of scale scores

Norms for subpopulations (male and female; students andnon-students samples; ethnical backgrounds; differentcultures sharing the same language)

Outlook


89/111

Computer programs

Dichotomous / ordinal data Cross-cultural research


Longitudinal data

Multilevel MTMM models


Computer Programs


90/111

Simple statistics, Cronbachs , item total

correlations, exploratory factor analysis:

SPSS

R


S-Plus

Confirmatory Factor Analysis (SEM)

Mplus

LISREL

AMOS

EQS

Dichotomous / Ordinal Data


91/111

Graded Response Model (Samejima, 1969) The observed variable Yijk is linked to the item-specific probit

variable by the following probability function:

( ) 2

21

|

ijk isjk x

P Y s e dx

= =


The parameter isjk is a difficulty parameter for each responsecategory bound s (s > 0) for item imeasuring traitjwithmethod k, and represents the probability distribution of the

standard normal distribution.

2s

0.8

1

greatProbabilityto chooseat least category s

( )|ijk ijk P Y s

( )1|ijk ijk P Y


92/111

( ) ( )( ) 2

21

|2

ijk isjk x

ijk ijk ijk isjk P Y s e dx

= =

0

0.2

0.4

0.6

-6 -5 -4 -3 -2 -1 0 1 2 3

unpleasant pleasant

ijk

( )

( )4 |ijk ijk P Y

( )3 |ijk ijk P Y ( )2 |

ijk ijk P Y


0

0.2

0.4

0.6

0.8

1

-5 -4 -3 -2 -1 0 1 2 3

0 -not at all

1 2 3

very much so -4

response probability

pleasantunpleasant

good( )|ijk ijk P Y s =

ijk

( ) ( ) ( )| | 1|ijk ijk ijk ijk ijk ijk P Y s P Y s P Y s = = +

( ) ( ) ( )3 | 3 | 4 |ijk ijk ijk ijk ijk ijk P Y P Y P Y = =

Factor Analysis for Ordered Categorical Data


93/111

latent continuous variable, causes observed responses

*

ijkY

Assumption:


( )*

1

*

( 1)

*

1

1, for

, for

0, for

ijkijk ijk c ijk

ijk sijk ijk s ijk

ijk ijk

c Y

Y s Y

Y

+

Documents

Scale Construction